Compare commits

...

111 Commits

Author SHA1 Message Date
Maxim Vafin fb35278525
Change opt-125m-gptq model (#24901)
# Conflicts:
#	tests/model_hub_tests/torch_tests/test_hf_transformers.py

### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-06-10 11:08:09 +02:00
Xuejun Zhai d9ef8441b1
Port 24606 to release 202303 (#24796) 2024-06-03 09:35:49 +04:00
Yuan Hu e9f8604430
port PR 24562 to release/2023/3 (#24670)
### Details:
 - *port PR #24562 to release 2023.3*
 - *add support for InferAPI1 and api_version*

### Tickets:
-
*[issue-24509](https://github.com/openvinotoolkit/openvino/issues/24509)*

---------

Signed-off-by: HU Yuan2 <yuan2.hu@intel.com>
Co-authored-by: Wanglei Shen <wanglei.shen@intel.com>
2024-05-28 07:37:02 +08:00
Xuejun Zhai 78b55038b5
Port PR 24018 to release/2023/3 (#24371)
### Details:
 - Port PR2418 to release 2023.3
 - *...*

### Tickets:
 - *CVS-132745*

Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
2024-05-07 15:11:35 +04:00
Ilya Lavrenov d3cb701018
Added check for conflicting DEB / RPM versions (#24275)
### Details:
 - Added check for conflicting DEB / RPM versions
2024-04-28 01:19:34 +04:00
Ilya Lavrenov 6b3f564fe8
Port openvino-telemetry support in Debian package to 2023.3 (#24257)
### Ported PRs:
- https://github.com/openvinotoolkit/openvino/pull/23917
- https://github.com/openvinotoolkit/openvino/pull/24060
- https://github.com/openvinotoolkit/openvino/pull/23816

---------

Co-authored-by: Mikhail Ryzhov <mikhail.ryzhov@intel.com>
Co-authored-by: Przemyslaw Wysocki <przemyslaw.wysocki@intel.com>
2024-04-28 00:55:35 +04:00
Xiping Yan f32c023c91
[CPU]Fix regression issue. (#23904)
### Details:
 - *Add callback fun param for selectPreferPrimitiveDescriptor*
 - *Update Snippet node selectPreferPrimitiveDescriptor algorithm.*
 - *Add test*

### Tickets:
 - *137307*

---------

Signed-off-by: xipingya <xiping.yan@intel.com>
2024-04-26 10:21:25 +02:00
Maciej Smyk 53ca00bc81
[DOCS] release policy for 2023.3 (#24185)
Added Release Policy article to docs. Updated with changes from the
https://github.com/openvinotoolkit/openvino/pull/24145
2024-04-23 18:19:37 +02:00
Maciej Smyk 0d6fadd2da
[DOCS] Fixing Tensor Names examples in Model Representation for 2023.3 (#24173)
Port from https://github.com/openvinotoolkit/openvino/pull/23482

* Fixed examples of Tensor Names,
* Added an example of ov::element::Type for both Python and C++
2024-04-22 17:20:08 +02:00
Ilya Lavrenov 28c1585d65
Port several PRs from master to 2023.3 LTS (#23461)
### Ported PRs:
- https://github.com/openvinotoolkit/openvino/pull/22928
- https://github.com/openvinotoolkit/openvino/pull/22963
- https://github.com/openvinotoolkit/openvino/pull/23345
- https://github.com/openvinotoolkit/openvino/pull/23460

---------

Co-authored-by: Andrey Babushkin <andrey.babushkin@intel.com>
2024-04-16 18:23:32 +04:00
Maksim Kutakov 15b0f99132
[CPU] fix gen_pattern::AttrAny::equal_to (#23689)
### Details:
Backport https://github.com/openvinotoolkit/openvino/pull/21890 from
master

### Tickets:
 - CVS-133621

Co-authored-by: Tingqian Li <tingqian.li@intel.com>
2024-04-16 12:35:46 +02:00
Maksim Kutakov e6fbca158a
[CPU] Fix binary PReLU spare register search (#23910)
### Details:
Fix the algorithm of assigning an auxiliary SIMD register in the binary
PReLU post op JIT injector.

OneDNN fork PR: https://github.com/openvinotoolkit/oneDNN/pull/240

### Tickets:
 - CVS-137492
2024-04-16 12:35:35 +02:00
Karol Blaszczak 89e0a839d0
[DOCS] old fix for yolo article 23.3 (#23933) 2024-04-10 12:13:49 +02:00
Tatiana Savina feaeca5191
[DOCS] Port Add MS Windows version for NPU to 23.3 (#23872)
### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-04-04 14:53:13 +02:00
Steve Yoo ddef44e96d
[GPU] Skip concat_input_order (#23787)
### Details:
- *Skip concat_input_order for an element which lies outside of the
buffer range*

### Tickets:
 - *135790*
2024-04-02 20:56:35 -07:00
Sungeun Kim c4ada32cc8
[GPU] add reorder between mvn and conv. (#23670) (#23765)
- add reorder to avoid inserting padding to mvn before ocl conv.
- mvn does not support padding.

### Tickets:
 - 136385
2024-04-03 10:01:34 +09:00
Sebastian Golebiewski 769fa5764e
[DOCS] Update code examples with imports and save_model for for 23.3 (#23684)
Updating code examples with imports for ``convert_model`` and adding an
example of how to save a converted model. This PR addresses the 120254
ticket.
Porting: https://github.com/openvinotoolkit/openvino/pull/23654
2024-03-26 13:42:30 +01:00
Andrzej Kopytko 687c648120
[DOCS] update banner for 2023.3 (#23656)
### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-03-25 13:44:49 +01:00
Andrzej Kopytko beff6958a9
[Docs] restore banner information (#23623)
### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-03-22 09:36:06 +01:00
Andrzej Kopytko 385782df37
[DOCS] Added off on click for 2023.3 (#23467)
Port for 
https://github.com/openvinotoolkit/openvino/pull/23411
2024-03-14 15:16:15 +01:00
tadamczx 655a72607f
[DOCS] Python API version fix (#23403)
CMakeLists.txt update, to be able to build right version of Python API.
2024-03-13 07:05:06 +01:00
Andrzej Kopytko beb158f6f0
Docs remove canonical tag from 2023.3 (#23375)
### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-03-12 09:43:03 +01:00
Maciej Smyk 0c9ad9ede5
[DOCS] Fix of home tab title for 2023.3 (#23367)
Jira: 133084

* Fixed the home page tab title name.
* Fixed the missing line in banner
2024-03-11 11:51:35 +01:00
Wang, Yang 735a3ba80a
Update port name checking logic to 2023.3 (#23339)
Port from https://github.com/openvinotoolkit/openvino/pull/23081 to
2023.3
### Details:
 - *item1*
 - *...*

### Tickets:
 - 122932
2024-03-08 13:12:19 +04:00
Anastasia Kuporosova c4034018fd
[PyOV] Cherry-picked convert ExecutionMode enum to any (#23296) 2024-03-07 11:02:31 +04:00
Chen Peter 004a6e9cf2
Map configuration to API 2.0 properties (#23161)
### Details:
- *Create table to map some of the Inference Engine configurations to
API 2.0 C++/Python properties*
- *Provide configurations/properties header files for other
configurations/properties*

### Tickets:
 - *CVS-94044*

---------

Signed-off-by: Peter Chen <peter.chen@intel.com>
2024-03-01 11:25:44 +04:00
River Li 0671bf3161
Port PR-23100 for Windows unicode issue (#23120)
### Details:
 - solve windows unicode inconsistent issue
 - add more test case

### Tickets:
 - Closes https://github.com/openvinotoolkit/openvino/issues/23072
 - CVS-128116
2024-02-28 11:08:13 +04:00
Pawel Raasz d02cd0c143
[core] Restore OV exception create function (#23014)
### Details:
- Restore `create` member function in OV exception classes to be
compatible with 2023.2 API.

### Tickets:
 - [CVS-133319](https://jira.devtools.intel.com/browse/CVS-133319)
2024-02-26 13:21:44 +04:00
Karol Blaszczak 7fe8b03cba
[DOCS] benchmark fix table (#23062) 2024-02-23 20:19:17 +01:00
Andrzej Kopytko 381a550f74
DOCS Udpate banchmark files to 2023.3 (#23040)
Port from https://github.com/openvinotoolkit/openvino/pull/23029 to
2023.3
2024-02-23 10:07:37 +01:00
Andrzej Kopytko 51055ef9e1
[DOCS] upgrade chart js to v4 (#23020)
port to https://github.com/openvinotoolkit/openvino/pull/23008
2024-02-22 13:41:53 +01:00
River Li 6758735364
[Core] port PR-22525 and PR-22804 (#22554)
### Details:
 - Port: https://github.com/openvinotoolkit/openvino/pull/22525
 - Port: https://github.com/openvinotoolkit/openvino/pull/22804

### Tickets:
 - CVS-131048

---------

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
Co-authored-by: Vitaliy Urusovskij <vitaliy.urusovskij@intel.com>
2024-02-19 14:24:31 +04:00
Ilya Lavrenov 376428b59a
Ported to 2023.3 from master (#22899)
### Ported from master:
- https://github.com/openvinotoolkit/openvino/pull/22816
- https://github.com/openvinotoolkit/openvino/pull/22847
- https://github.com/openvinotoolkit/openvino/pull/22866

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Maksim Kutakov <maksim.kutakov@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-19 11:18:24 +04:00
Sebastian Golebiewski ee39aaaa84
[DOCS] Updating Interactive Tutorials for 23.3 (#22862)
Updating Interactive Tutorials - adding new notebooks. Porting:
https://github.com/openvinotoolkit/openvino/pull/22815
2024-02-15 15:35:33 +01:00
Ilya Lavrenov c17db5989c
Ported several PRs from master to 2023.3 LTS (#22843)
### Ported:
- https://github.com/openvinotoolkit/openvino/pull/22606
- https://github.com/openvinotoolkit/openvino/pull/22734
- https://github.com/openvinotoolkit/openvino/pull/22776
- https://github.com/openvinotoolkit/openvino/pull/22820
- https://github.com/openvinotoolkit/openvino/pull/22827

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Andrei Kashchikhin <andrey.kashchikhin@intel.com>
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-15 11:30:04 +04:00
Vishniakov Nikolai b03c36fee3
[OV JS] Port of Fix npm dependencies (#22830)
Port of:
https://github.com/openvinotoolkit/openvino/pull/22810

To fix this issue:
https://github.com/openvinotoolkit/openvino/security/dependabot/514
2024-02-14 17:39:18 +01:00
Sebastian Golebiewski 4bf8a469dc
[DOCS] Removing the note about GPU support for static models only (#22803)
Removing the note about GPU support for static models only, as GPU now
supports both type of shapes. Porting:
https://github.com/openvinotoolkit/openvino/pull/22802
2024-02-13 08:20:45 +01:00
Karol Blaszczak bed27aa64d
[DOCS] tiny fix for ovms linking (#22755) 2024-02-09 12:29:57 +01:00
Anastasia Kuporosova 493539be12
remove flake8-commas requirement (#22611)
### Details:
 - cherry-picked https://github.com/openvinotoolkit/openvino/pull/22546

### Tickets:
 - *ticket-id*
2024-02-09 12:10:03 +04:00
Karol Blaszczak 1886fc6d31
[DOCS] review for 22356 and 22410 (#22466)
relates to:
https://github.com/openvinotoolkit/openvino/pull/22410
https://github.com/openvinotoolkit/openvino/pull/22356

---------

Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
2024-02-08 18:56:53 +01:00
Pavel Esir 77a0b916ba
Keep compressed constants produced by FW #22095 (#22096)
### Details:
 - keep compressed constants produced by FW
 - cherry pick of #22095

### Tickets:
 - CVS-119139
2024-02-08 07:16:44 +04:00
Maciej Smyk f33953bb0c
[DOCS] Fix for xml comments + ScatterUpdate Inputs for 23.3 (#22624)
Port from https://github.com/openvinotoolkit/openvino/pull/22623

- Fixed xml comments throughout the docs.
- Fixed ScatterUpdate Inputs paragraph.
2024-02-06 14:06:53 +01:00
River Li af87a9c5e4
[CAPI] Port PR-22579 (#22596)
https://github.com/openvinotoolkit/openvino/pull/22579
2024-02-04 06:15:46 +01:00
Andrzej Kopytko 6683952c0a
[DOCS] Benchmark Latency as default - port (#22622)
change from master
2024-02-02 12:24:59 +01:00
Alexander Suvorov 28e80c22cf
[DOCS] Update selector tool 23.3 (#22605) 2024-02-01 18:57:34 +01:00
Ilya Lavrenov 7a0f7481cc
Ported several PRs from master to 2023.3 (#22528)
* Bump actions/cache from 3 to 4 (#22224)

Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Disable proxy for GPU plugin (#22235)

* Disable proxy for GPU

* Fix

* Ported conda-forge patch to work with latest protobuf (#22277)

* Bump sccache version (#22282)

* [GHA] Set SCCACHE_IGNORE_SERVER_IO_ERROR (#22346)

* [GHA] Change default port for sccache server to avoid collisions (#22364)

* Change Windows CC workflow but based on last successful WinCC commit (#22469)

* changed linux cc build timeout (#22561)

* Bump mozilla-actions/sccache-action from 0.0.3 to 0.0.4 (#22552)

Bumps [mozilla-actions/sccache-action](https://github.com/mozilla-actions/sccache-action) from 0.0.3 to 0.0.4.
- [Release notes](https://github.com/mozilla-actions/sccache-action/releases)
- [Commits](https://github.com/mozilla-actions/sccache-action/compare/v0.0.3...v0.0.4)

---
updated-dependencies:
- dependency-name: mozilla-actions/sccache-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [CI] [GHA] Download `ninja` from GitHub instead of installing through `choco` in Windows workflows (#22570)

* download ninja from github

* add ninja to path

* add note

* add retries

* [CI] [GHA] Add retries for APT, update deadsnakes action (#22515)

* update setup_python version, add retries for apt in Linux workflows

* account for sudo

* use sudo via sh

* bump to v5 (#22505)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Oleg Pipikin <oleg.pipikin@intel.com>
Co-authored-by: Andrey Babushkin <andrey.babushkin@intel.com>
Co-authored-by: Alina Kladieva <alina.kladieva@intel.com>
Co-authored-by: Mikhail Ryzhov <mikhail.ryzhov@intel.com>
Co-authored-by: Andrei Kashchikhin <andrey.kashchikhin@intel.com>
2024-02-01 17:37:27 +04:00
Sebastian Golebiewski 91cde333cb
[DOCS] Port docs patches for 23.3 (#22590)
* Porting 21679 - fixing typo

* Porting 22571 - specify number of jobs in build
2024-02-01 14:02:35 +04:00
Maciej Smyk 4cf6f57fb5
comment fix (#22557) 2024-01-31 14:02:50 +01:00
Maciej Smyk ffb5270986
[DOCS] ref fix + benchmark fix + ov:state_network fix (#22529) 2024-01-30 10:38:47 +01:00
Zlobin Vladimir ec4f36b015
Fix grammar (#22457) 2024-01-30 01:21:31 +04:00
Sebastian Golebiewski d9faf192a6
Removing NPU plugin specific properties (#22512) 2024-01-29 15:17:04 +01:00
Karol Blaszczak f52de9434b
[DOCS] conformance tables update (#22459) 2024-01-29 13:45:44 +01:00
Sebastian Golebiewski 9cd1afbe96
Updating reference to Inference Modes (#22502) 2024-01-29 13:18:03 +01:00
Sebastian Golebiewski 935533429f
Fix Ubuntu version in NPU article (#22499) 2024-01-29 13:01:17 +01:00
Sebastian Golebiewski af64de8add
[DOCS] Updating interactive tutorials for 23.3 (#22495)
* Updating Interactive Tutorials

* Updating Tutorials
2024-01-29 11:03:24 +01:00
Ivan Tikhonov 39fc555364
Stateful Models Documentation Update (#22410)
* Squashed commits: Updating Stateful Models Documentation

* Apply suggestions from code review

Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>

* update illustrations

* Fix links; change illustrations format to svg; apply review comments

* add missing files

* Fix missprints; apply review comments; fix links

* fix missprints

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

* fix illustrations

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

* Fix the remaining links, add a correct illustration for makestateful transformation, refactoring

---------

Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
2024-01-29 13:35:05 +04:00
Karol Blaszczak 48d5d4b79c
[DOCS] excell link restructure (#22463) 2024-01-26 16:55:27 +01:00
Tatiana Savina 8ea3f717cb
[DOCS] Fix nightly link (#22431)
* fix nightly link

* change link
2024-01-25 17:06:24 +01:00
Karol Blaszczak b1c8402eaa
[DOCS] fix link (#22404) 2024-01-25 14:03:19 +01:00
Sebastian Golebiewski 44d661a0ea
[DOCS] Fixing reference to PyTorch conversion for 23.3 (#22393)
* Fixing reference to PyTorch conversion
* Fixing formatting in Release Notes
* Removing Openvino Supported Models article
2024-01-24 16:19:19 +01:00
Sergey Lyalin ad379d7dd5
Updated Optimize and Deploy Generative AI Models documentation article (#22356)
* Updated Optimize and Deploy Generative AI Models article with stateful models information and new examples/resources

* Apply suggestions from code review

Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>

* Apply suggestions from code review

Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>

---------

Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
2024-01-24 18:20:11 +04:00
Maciej Smyk c4d1b11ee8
[DOCS] NPU table fix for 23.3 2024-01-24 14:44:06 +01:00
Karol Blaszczak 9645660166
[DOCS] NPU further tweaks (#22385) 2024-01-24 13:31:59 +01:00
Maciej Smyk 2d6b7e2b80
Update NPU.rst (#22377) 2024-01-24 11:21:27 +01:00
Andrzej Kopytko 0390d68ddb
[DOCS] quick fix graph.js to 23.3 2024-01-24 11:17:38 +01:00
Karol Blaszczak 489cc9304d
[DOCS] final benchmark data + conf version (#22375)
* [DOCS] final benchmark data
* conf.py update 23.3
2024-01-24 11:09:17 +01:00
Karol Blaszczak 900d7f8048
[DOCS] NPU update (#22373) 2024-01-24 09:44:14 +01:00
Karol Blaszczak 0245affbb4
[DOCS] releasenotes 23.3 update (#22362) 2024-01-24 08:34:59 +01:00
Sebastian Golebiewski 7721fa7603
[DOCS] Porting Benchmark Changes to 23.3 (#22357)
* change benchmarks to 23.3

* Updating benchmarks - porting 22353

* Porting 22353

---------

Co-authored-by: Kopytko, AndrzejX <andrzejx.kopytko@intel.com>
2024-01-23 17:52:59 +01:00
Tatiana Savina 1207ec4277
[DOCS] Docs review (#22340)
* add changes

* remove article
2024-01-23 14:32:31 +01:00
Andrzej Kopytko d9d4c9dacd
Main changes without valid data (#22065) (#22099) 2024-01-23 14:22:04 +01:00
Andrzej Kopytko a9d502b315
[Docs] benchmarks precisions port 2023 3 (#22319)
* [DOCS] update

* added rst update
2024-01-23 13:35:55 +01:00
Maciej Smyk 57a9c41a9a
[DOCS] Removing broken links from Model Representation article for 23.3 (#22344)
* Update model_representation.rst

* Update model_representation.rst
2024-01-23 13:00:12 +01:00
Tatiana Savina dd0a833d68
update ov version (#22342) 2024-01-23 12:02:15 +01:00
Vishniakov Nikolai d5c91c758c
Fix wstring join_paths (#22271) (#22305)
* Fix wstring join_paths

* Fix codestyle
2024-01-23 03:10:12 +04:00
Mikhail Ryzhov ecbb1387ab
increased timeout (#22306) 2024-01-22 22:04:40 +04:00
Andrzej Kopytko 45b501bb6f
[Docs] graph.js bf16 prec port to 2023.3 (#22317)
* Main changes without valid data

* fix
2024-01-22 16:08:27 +01:00
Alexander Kozlov aeda8aae2e
[DOCS] Updated weight compression documentation (#22063)
authored by alexander.kozlov@intel.com
2024-01-22 12:52:52 +01:00
Tatiana Savina b58d9cede7
add save model info (#22267) (#22269) 2024-01-19 14:49:17 +01:00
Ilya Lavrenov d19307ba13
Ported changes from master (#22193)
* [CI] [GHA] Introduce JS API as a part of the existing workflows (#21898)

* add js api to linux

* try inside the ov repo

* use rel path

* use a separate job for js api

* correct command formatting

* add missing var

* use spacing

* mv js building

* add node installing

* add to windows

* check pwsh and cmd running npm

* add smart CI conditions; disable for win

* use node version as env var

* extract js job into a separate workflow, add to other *nix

* fix input name

* Activate js bindings tests for arm64

* upload ov js package

* correct formatting

* add missing syntax

---------

Co-authored-by: Vishniakov Nikolai <nikolai.vishniakov@intel.com>

* Cmake Python build option flags should be added to the command in step #3 not step #4. I fixed the typo (#21993)

* [CI] [GHA] [JS API] Remove explicit default values settings in Linux ARM64 `cmake` (#22019)

* rm explicit default values settings

* Activate mac arm64 js api check

* Specify test run

---------

Co-authored-by: Vishniakov Nikolai <nikolai.vishniakov@intel.com>

* [OV JS] Activate validation for mac x86 (#22035)

* Extend validation for mac x86

* Remove extra params

* fixed broken doc links (#22088)

Co-authored-by: Przemyslaw Wysocki <przemyslaw.wysocki@intel.com>

* [GHA] Update MO deps (#22130)

* [GHA] Update MO deps

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>

* Update .github/components.yml

---------

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>

* Avoid DOWNLOAD_EXTRACT_TIMESTAMP warning (#22135)

* Avoid DOWNLOAD_EXTRACT_TIMESTAMP warning

* Change applying policy condition

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>

---------

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>

* Fixed API validator search (#22136)

* [OV JS] Conditional enabling of JS API (#22139)

* Disable js api building for vcpkg

* Disable JS API by default

* Add disable JS API conditions in features.cmake

* Update cmake/features.cmake

* Update src/bindings/js/CMakeLists.txt

---------

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>

* Fixed GHSA-h5c8-rqwp-cp95 (#22159)

* [PyOV][SAMPLES] Fix bugbear issue B038 (#22183)

* Fixed compilation on GHA CI

* Decrease number of workers for ONNX Model tests to prevent OOM kills (#22243)

* Decrease number of workers for ONNX Model tests to prevent OOM kills

* Try to use "-n auto" also

---------

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
Co-authored-by: Andrei Kashchikhin <andrey.kashchikhin@intel.com>
Co-authored-by: Vishniakov Nikolai <nikolai.vishniakov@intel.com>
Co-authored-by: fredrickomondi <omondifredrick@gmail.com>
Co-authored-by: Santhosh Mamidisetti <92091342+SANTHOSH-MAMIDISETTI@users.noreply.github.com>
Co-authored-by: Przemyslaw Wysocki <przemyslaw.wysocki@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
Co-authored-by: Andrey Babushkin <andrey.babushkin@intel.com>
2024-01-19 16:46:19 +04:00
Sebastian Golebiewski 187e262c92
[DOCS] Updating templates for Binder and Google Colab buttons for 23.3 (#22238)
* Update for Binder and Google Colab buttons

* change image source
2024-01-19 07:55:20 +01:00
Alexander Suvorov aeff428378
[DOCS] Update 2023.3.0 Selector Tool (#22218)
* Update selector tool
2024-01-17 19:48:54 +01:00
Sergey Lyalin 3dd41ff8c3
String Tensors Basic Documentation (#22097)
* Basic documentation of how to use string tensors

* Fix typo: str_data -> bytes_data

* More clarification about UTF-8 and represenation capabilities

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/string_tensors.rst

Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Applied code review suggestion about 'floating-point'

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/string_tensors.rst

* Fixed a snippet tab rendering and items formatting in Additional Reasources section

---------

Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
2024-01-17 19:02:16 +04:00
Sebastian Golebiewski 10cae14374
Port 21661 Merge Samples Articles Language Versions (#22187) 2024-01-17 13:27:12 +04:00
Sebastian Golebiewski af4b99e7cf
Port 22006 Updating options for torch.compile (#22188) 2024-01-16 16:51:20 +01:00
Xuejun Zhai 7b782956ab
[CAPI][TEST] fix cancel api test with infer finished before cancel call (#22175)
* [CAPI][TEST] fix cancel api test with infer finished before cancel call

Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>

* fix error

Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>

---------

Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
2024-01-16 13:17:13 +00:00
Karol Blaszczak 1ef753a590
[DOCS] torchvision and optimization section (#22142)
port: https://github.com/openvinotoolkit/openvino/pull/22043
2024-01-15 08:20:18 +01:00
Karol Blaszczak e64ec3ff5a
[DOCS] fix npu and install 23.3 (#22125) 2024-01-12 17:51:51 +01:00
Sebastian Golebiewski 258dde9f52
Fix images for notebooks (#22131) 2024-01-12 14:18:51 +01:00
River Li 7c37e0916c
[CAPI] avoid random timeout issue in test cases (#22086) 2024-01-11 14:34:14 +04:00
Ilya Lavrenov b981887d6f
Compatibility with newer OpenCL versions (#22051) 2024-01-10 07:31:17 +00:00
Maciej Smyk 47c49e7e84
[DOCS] Hyperlink 23.3 update for 23.3 (#21909)
* link update

* link update

* link update

* Update dev_api_references.rst

* Update dev_api_references.rst

* Update apidoc.py

* notebooks links

* notebook link update

* filename update

* link update

* link update api
2024-01-08 14:37:37 +01:00
Luo Cheng ceeafaf64f
[CPU] Support group beam search (#21983)
* support group beam search

* support dyn batch without set_state

* apply review comments

* strides may be incorrect when batch changed

---------

Co-authored-by: Yu Xu <yu.xu@intel.com>
2024-01-08 16:42:05 +04:00
River Li 49231c0196
[CAPI] fixed ov_infer_request_wait_for return value issue (#22014) 2024-01-08 16:41:21 +04:00
Ilya Lavrenov 38ee887911
Ported changes from master to 2023.3 (#22012)
* [JS OV] Specify JS API GitHub config (#21785)

* Extend CODEOWNERS by js api responsibles

* Extend smart ci by js component

* Fix codeowners

* Fix component definition

* Update components.yml

* Update components.yml

---------

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>

* [GHA] Fix smart ci for docs changes in linux arm64 (#21798)

Update linux_arm64.yml

---------

Co-authored-by: Vishniakov Nikolai <nikolai.vishniakov@intel.com>
Co-authored-by: Alina Kladieva <alina.kladieva@intel.com>
2024-01-08 16:33:58 +04:00
Maciej Smyk 429b89f0e4
Debian9 update to Ubuntu 18.04 (#21935) 2024-01-08 15:39:37 +04:00
Vitaliy Urusovskij 1df3008870
Fix `INFER_NOT_STARTED` infer status code (#21939)
* Fix `INFER_NOT_STARTED` infer status code

* Update `returnDeviceBusyOnSetBlobAfterAsyncInfer` test
to check `INFER_NOT_STARTED` status only

* Update `wait` tests with `INFER_NOT_STARTED` check
2024-01-08 15:35:20 +04:00
Maxim Vafin 7835368489
[DOCS] Update supported ops by PT FE (#21961) 2024-01-08 11:11:16 +01:00
Maxim Vafin cc8fadf187
[DOCS] Describe dict support (#21923)
* [DOCS] Describe dict support

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

---------

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
2024-01-08 11:11:05 +01:00
Pavel Durandin a741d5a8ba
Fix minus axis case for kv cache primitive buffer fusing (#21958)
Co-authored-by: Taylor Lee <taylor.lee@intel.com>
2024-01-05 09:22:42 +09:00
Alina Kladieva ca31765d1a
Bump OMZ submodule (#21943)
unlimit pyparsing to fix CVS-128637
2024-01-04 09:51:29 +01:00
Roman Lyamin 112a16a056
[GPU] Small fix for dynamic_shape_gather_opts pass (#21807) (#21882) 2023-12-28 14:48:38 +04:00
Roman Lyamin 7d948bf596
[GPU] Add set_arguments() call in case of dynamic dependencies of static inst (#21664) (#21884)
Co-authored-by: Sergey Shlyapnikov <sergey.shlyapnikov@intel.com>
2023-12-28 14:48:13 +04:00
Maciej Smyk 4e15cc0b7f
Port from 21815 (#21839) 2023-12-27 16:01:33 +01:00
Wilson Seok e1bd2a7b55
Revert "[GPU] Fix accuracy problems brought by permute_f_y kernel (#20241)" (#21830) (#21876)
This reverts commit 6719133af2.

Co-authored-by: Pavel Durandin <pavel.durandin@intel.com>
2023-12-27 18:31:41 +04:00
Surya Siddharth Pemmaraju 8a5e6b6738
Fixed a bug in input validation for torch.compile options (#21835)
* Fixed bug in input validation for torch.compile options

* Added default device if device is None

* Addressed PR comments
2023-12-22 14:26:08 +04:00
Sebastian Golebiewski 81b5895d24
Move Troubleshooting Reshape article to legacy section (#21837) 2023-12-22 11:15:01 +01:00
Maciej Smyk e37569ffd5
[DOCS] Removal of unused images + fixing image directives for 23.3 (#21818)
* removal of unused images

* removal of duplicate image files

* removal of unused files + fix for image directive

* removal of unused images

* removal of unused images + fix for protecting_model.rst

* overview document graph update
2023-12-21 13:14:37 +01:00
Karol Blaszczak 07fc4d0813
[DOCS] NPU update 23.3 (#21812) 2023-12-21 09:24:52 +00:00
Tatiana Savina 7ef8faecf0
fix note and img (#21797) 2023-12-21 08:47:51 +01:00
Ilya Lavrenov 53960d6018
Switched GHA branches to 2023.3 (#21793) 2023-12-20 19:22:51 +04:00
1226 changed files with 141630 additions and 41611 deletions

2
.github/CODEOWNERS vendored
View File

@ -28,6 +28,7 @@
/src/bindings/python/ @openvinotoolkit/openvino-ie-python-api-maintainers
/src/bindings/c/ @openvinotoolkit/openvino-c-api-maintainers
/src/bindings/js/ @openvinotoolkit/openvino-js-api-maintainers
/src/common/*transformations/ @openvinotoolkit/openvino-ie-transformations-maintainers
/src/core/ @openvinotoolkit/openvino-ngraph-maintainers
@ -35,6 +36,7 @@
/samples/c/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-c-api-maintainers
/samples/cpp/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-maintainers
/samples/python/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-ie-python-api-maintainers
/samples/js/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-js-api-maintainers
/thirdparty/zlib/ @openvinotoolkit/openvino-samples-maintainers
/thirdparty/json/ @openvinotoolkit/openvino-samples-maintainers
/thirdparty/gflags/ @openvinotoolkit/openvino-samples-maintainers

View File

@ -26,22 +26,27 @@ runs:
- if: ${{ runner.os == 'Linux' && inputs.self-hosted-runner == 'true' }}
name: Install 'actions/setup-python@v4' dependencies
shell: bash
run: apt-get update && apt-get install -y ca-certificates software-properties-common
run: apt-get update && apt-get install -y ca-certificates software-properties-common gpg-agent tzdata
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
TZ: "Europe/London" # to prevent tzdata from waiting user input
- if: ${{ runner.os == 'Linux' && runner.arch == 'ARM64' }}
name: Setup sudo and python3
shell: bash
run: apt-get update && apt-get install -y sudo python3 # Needed for the deadsnakes action
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
- if: ${{ runner.os == 'Linux' && runner.arch == 'ARM64' }}
name: Setup Python ${{ inputs.version }}
uses: akashchi/deadsnakes-action@f01521a69eee61eaca3a34440bea3ce838317846
uses: akashchi/deadsnakes-action@92417281055a5878a0450f240a5b95883eb2d7e2
with:
python-version: ${{ inputs.version }}
- if: ${{ runner.os == 'macOS' || runner.os == 'Windows' || (runner.os == 'Linux' && runner.arch != 'ARM64') }}
name: Setup Python ${{ inputs.version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ inputs.version }}
env:

View File

@ -29,6 +29,7 @@ CPU:
revalidate:
- C_API
- Python_API
- JS_API
- samples
- ONNX_RT
- PyTorch_FE
@ -115,6 +116,7 @@ IR_FE:
revalidate:
- C_API
- Python_API
- JS_API
- samples
build:
- CPU
@ -181,6 +183,13 @@ Python_API:
- TFL_FE
- PyTorch_FE
JS_API:
revalidate:
- samples
build:
- CPU
- IR_FE
samples:
build:
- CPU
@ -205,7 +214,8 @@ IE_Tests:
MO:
revalidate:
- POT
- PyTorch_FE
- TF_FE
build:
- Python_API

6
.github/labeler.yml vendored
View File

@ -109,6 +109,9 @@
'category: packaging':
- 'cmake/**/packaging/**/*'
- 'src/bindings/python/wheel/**/*'
- any: ['src/bindings/js/node/CMakeLists.txt',
'src/bindings/js/node/package.json',
'src/bindings/js/node/package-lock.json']
- 'tools/openvino_dev/**/*'
'category: PDPD FE':
@ -124,6 +127,9 @@
'category: Python API':
- 'src/bindings/python/**/*'
'category: JS API':
- 'src/bindings/js/**/*'
'category: samples':
- 'samples/**/*'
- 'thirdparty/zlib/**/*'

View File

@ -53,6 +53,8 @@ jobs:
CMAKE_GENERATOR: 'Ninja'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
OPENVINO_REPO: '/__w/openvino/openvino/openvino'
VCPKG_ROOT: '/__w/openvino/openvino/vcpkg'
BUILD_DIR: '/__w/openvino/openvino/build'
@ -81,13 +83,14 @@ jobs:
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/json
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/gtest
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/gflags
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/open_model_zoo
popd
- name: Clone vcpkg
uses: actions/checkout@v4
with:
repository: 'microsoft/vcpkg'
# Keep in sync with <root>/vcpkg.json <builtin-baseline>
ref: '7ba0ba7334c3346e7eee1e049ba85da193a8d821'
path: 'vcpkg'
fetch-depth: '0'
@ -123,9 +126,9 @@ jobs:
echo "yes" | ./cmdline-tools/bin/sdkmanager --sdk_root=${ANDROID_TOOLS} --install "ndk-bundle" "platform-tools" "platforms;android-${{ env.ANDROID_SDK_VERSION }}"
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
#
# Build

View File

@ -20,7 +20,7 @@ jobs:
lfs: 'true'
- name: Install apt-get dependencies
uses: awalsh128/cache-apt-pkgs-action@v1.3.1
uses: awalsh128/cache-apt-pkgs-action@v1.4.1
with:
packages: graphviz texlive liblua5.2-0 libclang1-9 libclang-cpp9
version: 3.0
@ -54,7 +54,7 @@ jobs:
- name: Cache documentation
id: cache_sphinx_docs
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: build/docs/_build/.doctrees
key: sphinx-docs-cache

View File

@ -30,7 +30,7 @@ jobs:
submodules: 'true'
- name: Install OpenCL
uses: awalsh128/cache-apt-pkgs-action@v1.3.1
uses: awalsh128/cache-apt-pkgs-action@v1.4.1
if: runner.os == 'Linux'
with:
packages: ocl-icd-opencl-dev opencl-headers

View File

@ -36,6 +36,8 @@ jobs:
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
OPENVINO_CONTRIB_REPO: /__w/openvino/openvino/openvino_contrib
@ -62,7 +64,7 @@ jobs:
repository: 'openvinotoolkit/openvino_contrib'
path: ${{ env.OPENVINO_CONTRIB_REPO }}
submodules: 'true'
ref: 'master'
ref: 'releases/2023/3'
#
# Dependencies
@ -75,9 +77,9 @@ jobs:
apt install --assume-yes --no-install-recommends default-jdk
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: ./openvino/.github/actions/setup_python

View File

@ -53,6 +53,8 @@ jobs:
CMAKE_GENERATOR: 'Ninja'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
INSTALL_DIR: /__w/openvino/openvino/openvino_install
@ -85,9 +87,9 @@ jobs:
run: bash ${OPENVINO_REPO}/install_build_dependencies.sh
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: Install python dependencies
run: |

View File

@ -30,6 +30,9 @@ jobs:
PARALLEL_TEST_SCRIPT: ${{ github.workspace }}/install/tests/functional_test_utils/layer_tests_summary/run_parallel.py
PARALLEL_TEST_CACHE: ${{ github.workspace }}/install/tests/test_cache.lst
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@v3
with:
@ -83,7 +86,7 @@ jobs:
run: python3 -m pip install -r ${INSTALL_TEST_DIR}/functional_test_utils/layer_tests_summary/requirements.txt
- name: Restore tests execution time
uses: actions/cache/restore@v3
uses: actions/cache/restore@v4
with:
path: ${{ env.PARALLEL_TEST_CACHE }}
key: ${{ runner.os }}-${{ runner.arch }}-tests-functional-cpu-stamp-${{ github.sha }}
@ -98,10 +101,10 @@ jobs:
fi
python3 ${PARALLEL_TEST_SCRIPT} -e ${INSTALL_TEST_DIR}/ov_cpu_func_tests -c ${PARALLEL_TEST_CACHE} -w ${INSTALL_TEST_DIR} -s suite -rf 0 -- --gtest_print_time=1 --gtest_filter=*smoke*
timeout-minutes: 20
timeout-minutes: 25
- name: Save tests execution time
uses: actions/cache/save@v3
uses: actions/cache/save@v4
if: github.ref_name == 'master'
with:
path: ${{ env.PARALLEL_TEST_CACHE }}

View File

@ -1,4 +1,4 @@
name: Samples
name: C++ Unit Tests
on:
workflow_call:
@ -32,6 +32,9 @@ jobs:
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@v3
with:

View File

@ -26,6 +26,8 @@ jobs:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
DEBIAN_PACKAGES_DIR: ${{ github.workspace }}/packages
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO debian packages
uses: actions/download-artifact@v3
@ -40,7 +42,7 @@ jobs:
- name: Install debian packages & check conflicts
run: |
apt-get update -y
if [[ "${{ runner.arch }}" == "X64" ]]; then
# Install debian packages from previous release
apt-get install --no-install-recommends -y gnupg wget ca-certificates
@ -48,7 +50,8 @@ jobs:
apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
echo "deb https://apt.repos.intel.com/openvino/2023 ubuntu20 main" | tee /etc/apt/sources.list.d/intel-openvino-2023.list
apt-get update -y
apt-get install -y openvino
apt-get install -y openvino-2023.2.0
rm /etc/apt/sources.list.d/intel-openvino-2023.list
fi
# install our local one and make sure the conflicts are resolved
@ -63,21 +66,25 @@ jobs:
run: |
/usr/share/openvino/samples/cpp/build_samples.sh
/usr/share/openvino/samples/c/build_samples.sh
[[ "${{ runner.arch }}" == "X64" ]] && path_by_arch="intel64" || path_by_arch="aarch64"
~/openvino_cpp_samples_build/$path_by_arch/Release/hello_query_device
# check integrity of OpenVINO Python API installation
apt-get install python3-pip -y
python3 -m pip check
python3 /usr/share/openvino/samples/python/hello_query_device/hello_query_device.py
python3 -c 'from openvino import Core; Core().get_property("CPU", "AVAILABLE_DEVICES")'
if [[ "${{ runner.arch }}" == "X64" ]]; then
python3 -c 'from openvino import Core; Core().get_property("GPU", "AVAILABLE_DEVICES")'
fi
python3 -c 'from openvino import Core; Core().get_property("AUTO", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("MULTI", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("HETERO", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("BATCH", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("AUTO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("MULTI", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("HETERO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("BATCH", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino.frontend import FrontEndManager; assert len(FrontEndManager().get_available_front_ends()) == 6'
benchmark_app --help
opt_in_out --help
ovc --help

View File

@ -28,9 +28,15 @@ jobs:
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
ONNX_MODELS_PATH: ${{ github.workspace }}/onnx_test_models
MODELS_SHARE_PATH: "/mount/onnxtestdata"
# instead of using static MODELS_SHARE_PATH
# choose one of the replicas dynamically instead
# depending on GITHUB_RUN_NUMBER variable
NUMBER_OF_REPLICAS: 2
ONNX_MODEL_ZOO_SHA: "d58213534f2a4d1c4b19ba62b3bb5f544353256e"
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@v3
with:
@ -49,6 +55,8 @@ jobs:
echo "OPENVINO_REPO=$GITHUB_WORKSPACE/openvino" >> "$GITHUB_ENV"
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
echo "MODELS_SHARE_PATH=/mount/onnxtestdata$((GITHUB_RUN_NUMBER % NUMBER_OF_REPLICAS))" >> "$GITHUB_ENV"
echo $MODELS_SHARE_PATH
- name: Extract OpenVINO packages
run: |
@ -103,4 +111,4 @@ jobs:
python3 -m pip install pytest-xdist[psutil] pytest-forked
- name: ONNX Models Tests
run: python3 -m pytest --backend="CPU" --model_zoo_dir="${MODELS_SHARE_PATH}" ${INSTALL_TEST_DIR}/onnx/tests/tests_python/test_zoo_models.py -v -n 12 --forked -k 'not _cuda' --model_zoo_xfail
run: python3 -m pytest --backend="CPU" --model_zoo_dir="${MODELS_SHARE_PATH}" ${INSTALL_TEST_DIR}/onnx/tests/tests_python/test_zoo_models.py -v -n auto --forked -k 'not _cuda' --model_zoo_xfail

View File

@ -33,11 +33,16 @@ jobs:
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
SCCACHE_AZURE_KEY_PREFIX: ${{ inputs.sccache-azure-key-prefix }}
ONNX_RUNTIME_REPO: ${{ github.workspace }}/onnxruntime
ONNX_RUNTIME_UTILS: ${{ github.workspace }}/install/onnxruntime
ONNX_RUNTIME_BUILD_DIR: ${{ github.workspace }}/onnxruntime/build
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@v3
with:
@ -95,9 +100,9 @@ jobs:
run: bash ${OPENVINO_REPO}/install_build_dependencies.sh
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: Build Lin ONNX Runtime
run: |

61
.github/workflows/job_openvino_js.yml vendored Normal file
View File

@ -0,0 +1,61 @@
name: OpenVINO JS API
on:
workflow_call:
inputs:
runner:
description: 'Machine on which the tests would run'
type: string
required: true
container:
description: 'JSON to be converted to the value of the "container" configuration for the job'
type: string
required: false
default: '{"image": null}'
jobs:
JS_API:
name: OpenVINO JS API
timeout-minutes: 10
runs-on: ${{ inputs.runner }}
container: ${{ fromJSON(inputs.container) }}
defaults:
run:
shell: bash
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
OPENVINO_JS_DIR: ${{ github.workspace }}/openvino/src/bindings/js
OPENVINO_JS_LIBS_DIR: ${{ github.workspace }}/openvino/src/bindings/js/node/bin
NODE_VERSION: 18
steps:
- name: Fetch OpenVINO JS sources
uses: actions/checkout@v4
with:
sparse-checkout: |
src/bindings/js
path: 'openvino'
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
run: |
echo "OPENVINO_JS_DIR=$GITHUB_WORKSPACE/openvino/src/bindings/js" >> "$GITHUB_ENV"
echo "OPENVINO_JS_LIBS_DIR=$GITHUB_WORKSPACE/openvino/src/bindings/js/node/bin" >> "$GITHUB_ENV"
- name: Download OpenVINO JS package
uses: actions/download-artifact@v3
with:
name: openvino_js_package
path: ${{ env.OPENVINO_JS_LIBS_DIR }}
- name: Setup Node ${{ env.NODE_VERSION }}
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Configure OpenVINO JS API
working-directory: ${{ env.OPENVINO_JS_DIR }}/node
run: npm i
- name: Test OpenVINO JS API
working-directory: ${{ env.OPENVINO_JS_DIR }}/node
run: npm run test

View File

@ -37,6 +37,9 @@ jobs:
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
LAYER_TESTS_INSTALL_DIR: ${{ github.workspace }}/install/tests/layer_tests
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@v3
with:
@ -68,7 +71,7 @@ jobs:
- name: Install OpenVINO dependencies (Linux)
if: runner.os == 'Linux'
run: $INSTALL_DIR/install_dependencies/install_openvino_dependencies.sh -c=core -c=dev -y
run: $INSTALL_DIR/install_dependencies/install_openvino_dependencies.sh -c=core -c=dev -y -c=gpu
- name: Fetch setup_python action
uses: actions/checkout@v4

View File

@ -33,11 +33,18 @@ jobs:
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
MODEL_HUB_TESTS_INSTALL_DIR: ${{ github.workspace }}/install/tests/model_hub_tests
steps:
- name: Check sudo
if: ${{ runner.os == 'Linux' }}
run: if [ "$(id -u)" -eq 0 ]; then apt update && apt --assume-yes install sudo; fi
- name: Set apt retries
run: |
if [ "$(id -u)" -eq 0 ]; then
echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
else
sudo sh -c "echo 'Acquire::Retries \"10\";' >> /etc/apt/apt.conf.d/80-retries"
fi
- name: Download OpenVINO package
uses: actions/download-artifact@v3
with:

View File

@ -31,6 +31,9 @@ jobs:
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
BUILD_DIR: ${{ github.workspace }}/build
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@v3
with:

View File

@ -33,11 +33,18 @@ jobs:
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
MODEL_HUB_TESTS_INSTALL_DIR: ${{ github.workspace }}/install/tests/model_hub_tests
steps:
- name: Check sudo
if: ${{ runner.os == 'Linux' }}
run: if [ "$(id -u)" -eq 0 ]; then apt update && apt --assume-yes install sudo; fi
- name: Set apt retries
run: |
if [ "$(id -u)" -eq 0 ]; then
echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
else
sudo sh -c "echo 'Acquire::Retries \"10\";' >> /etc/apt/apt.conf.d/80-retries"
fi
- name: Download OpenVINO package
uses: actions/download-artifact@v3
with:

View File

@ -61,10 +61,13 @@ jobs:
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
OPENVINO_CONTRIB_REPO: /__w/openvino/openvino/openvino_contrib
INSTALL_DIR: /__w/openvino/openvino/openvino_install
INSTALL_DIR_JS: /__w/openvino/openvino/openvino_install/js
INSTALL_TEST_DIR: /__w/openvino/openvino/tests_install
DEVELOPER_PACKAGE_DIR: /__w/openvino/openvino/developer_package_install
BUILD_DIR: /__w/openvino/openvino/openvino_build
@ -73,6 +76,9 @@ jobs:
if: "!needs.smart_ci.outputs.skip_workflow"
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Install git
run: |
apt-get update
@ -90,7 +96,7 @@ jobs:
repository: 'openvinotoolkit/openvino_contrib'
path: ${{ env.OPENVINO_CONTRIB_REPO }}
submodules: 'true'
ref: 'master'
ref: 'releases/2023/3'
#
# Print system info
@ -110,9 +116,9 @@ jobs:
apt install --assume-yes --no-install-recommends default-jdk
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: ./openvino/.github/actions/setup_python
@ -222,6 +228,15 @@ jobs:
-B ${BUILD_DIR}
cmake --build ${BUILD_DIR} --parallel --config ${{ env.CMAKE_BUILD_TYPE }}
- name: CMake configure, build and install - OpenVINO JS API
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
run:
cmake -DCPACK_GENERATOR=NPM -DENABLE_SYSTEM_TBB=OFF -UTBB* -S ${OPENVINO_REPO} -B ${BUILD_DIR}
cmake --build ${BUILD_DIR} --parallel
cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR_JS} -P ${BUILD_DIR}/cmake_install.cmake
#
# Upload build artifacts
#
@ -234,6 +249,14 @@ jobs:
path: ${{ env.BUILD_DIR }}/openvino_package.tar.gz
if-no-files-found: 'error'
- name: Upload openvino js package
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
uses: actions/upload-artifact@v3
with:
name: openvino_js_package
path: ${{ env.INSTALL_DIR_JS }}
if-no-files-found: 'error'
- name: Upload openvino developer package
if: ${{ always() }}
uses: actions/upload-artifact@v3
@ -275,6 +298,15 @@ jobs:
image: 'openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04'
affected-components: ${{ needs.smart_ci.outputs.affected_components }}
JS_API:
name: OpenVINO JS API
needs: [ Build, Smart_CI ]
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
uses: ./.github/workflows/job_openvino_js.yml
with:
runner: 'aks-linux-4-cores-16gb'
container: '{"image": "openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04"}'
Conformance:
needs: [ Build, Smart_CI ]
timeout-minutes: ${{ matrix.TEST_TYPE == 'API' && 5 || 15 }}
@ -473,6 +505,8 @@ jobs:
CMAKE_CUDA_COMPILER_LAUNCHER: sccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
INSTALL_DIR: /__w/openvino/openvino/install
OPENVINO_DEVELOPER_PACKAGE: /__w/openvino/openvino/install/developer_package
OPENVINO_REPO: /__w/openvino/openvino/openvino
@ -483,6 +517,9 @@ jobs:
if: fromJSON(needs.smart_ci.outputs.affected_components).NVIDIA
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Fetch install_build_dependencies.sh
uses: actions/checkout@v4
with:
@ -521,7 +558,7 @@ jobs:
with:
repository: 'openvinotoolkit/openvino_contrib'
path: ${{ env.OPENVINO_CONTRIB_REPO }}
ref: 'master'
ref: 'releases/2023/3'
#
# Dependencies
@ -533,9 +570,9 @@ jobs:
apt -y --no-install-recommends install software-properties-common curl
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: Install CUDA
run: |

View File

@ -21,6 +21,7 @@ jobs:
runs-on: ubuntu-latest
outputs:
affected_components: "${{ steps.smart_ci.outputs.affected_components }}"
skip_workflow: "${{ steps.smart_ci.outputs.skip_workflow }}"
steps:
- name: checkout action
uses: actions/checkout@v4
@ -36,6 +37,8 @@ jobs:
commit_sha: ${{ github.sha }}
component_pattern: "category: (.*)"
repo_token: ${{ secrets.GITHUB_TOKEN }}
skip_when_only_listed_labels_set: 'docs'
skip_when_only_listed_files_changed: '*.md,*.rst,*.png,*.jpg,*.svg'
- name: Show affected components
run: |
@ -60,18 +63,24 @@ jobs:
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
OPENVINO_CONTRIB_REPO: /__w/openvino/openvino/openvino_contrib
INSTALL_DIR: /__w/openvino/openvino/openvino_install
INSTALL_DIR_JS: /__w/openvino/openvino/openvino_install/js
INSTALL_TEST_DIR: /__w/openvino/openvino/tests_install
DEVELOPER_PACKAGE_DIR: /__w/openvino/openvino/developer_package_install
BUILD_DIR: /__w/openvino/openvino/openvino_build
SCCACHE_AZURE_KEY_PREFIX: 'ubuntu20_aarch64_Release'
ONNX_RUNTIME_UTILS: /__w/openvino/openvino/openvino/.ci/azure/ci_utils/onnxruntime
if: "!fromJSON(needs.smart_ci.outputs.affected_components).docs_only"
if: "!needs.smart_ci.outputs.skip_workflow"
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Install git
run: apt-get update && apt-get install --assume-yes --no-install-recommends git ca-certificates
@ -87,7 +96,7 @@ jobs:
repository: 'openvinotoolkit/openvino_contrib'
path: ${{ env.OPENVINO_CONTRIB_REPO }}
submodules: 'true'
ref: 'master'
ref: 'releases/2023/3'
#
# Print system info
@ -107,9 +116,9 @@ jobs:
apt install --assume-yes --no-install-recommends default-jdk
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: ./openvino/.github/actions/setup_python
@ -218,6 +227,20 @@ jobs:
-B ${BUILD_DIR}
cmake --build ${BUILD_DIR} --parallel --config ${{ env.CMAKE_BUILD_TYPE }}
- name: CMake configure, build and install - OpenVINO JS API
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
run: |
cmake \
-DCPACK_GENERATOR=NPM \
-DENABLE_SYSTEM_TBB=OFF -UTBB* \
-DENABLE_INTEL_GPU=OFF \
-S ${OPENVINO_REPO} \
-B ${BUILD_DIR}
cmake --build ${BUILD_DIR} --parallel
cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR_JS} -P ${BUILD_DIR}/cmake_install.cmake
#
# Upload build artifacts
#
@ -238,6 +261,14 @@ jobs:
path: ${{ env.BUILD_DIR }}/openvino_developer_package.tar.gz
if-no-files-found: 'error'
- name: Upload openvino js package
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
uses: actions/upload-artifact@v3
with:
name: openvino_js_package
path: ${{ env.INSTALL_DIR_JS }}
if-no-files-found: 'error'
- name: Upload openvino debian packages
if: ${{ 'false' }}
uses: actions/upload-artifact@v3
@ -272,6 +303,15 @@ jobs:
image: 'openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04'
affected-components: ${{ needs.smart_ci.outputs.affected_components }}
JS_API:
name: OpenVINO JS API
needs: [ Build, Smart_CI ]
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
uses: ./.github/workflows/job_openvino_js.yml
with:
runner: 'aks-linux-16-cores-arm'
container: '{"image": "openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04"}'
ONNX_Runtime:
name: ONNX Runtime Integration
if: fromJSON(needs.smart_ci.outputs.affected_components).ONNX_RT

View File

@ -58,6 +58,8 @@ jobs:
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
INSTALL_DIR: /__w/openvino/openvino/openvino_install
@ -69,6 +71,9 @@ jobs:
if: "!needs.smart_ci.outputs.skip_workflow"
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Install git
run: |
apt-get update
@ -86,7 +91,7 @@ jobs:
repository: 'openvinotoolkit/testdata'
path: ${{ env.MODELS_PATH }}
lfs: 'true'
ref: 'master'
ref: 'releases/2023/3'
#
# Print system info
@ -109,9 +114,9 @@ jobs:
update-alternatives --install /usr/bin/c++ c++ /usr/bin/clang++ 100
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: ./openvino/.github/actions/setup_python
@ -232,7 +237,7 @@ jobs:
CC_Build:
name: Conditional Compilation
needs: Build
timeout-minutes: 10
timeout-minutes: 20
defaults:
run:
shell: bash
@ -246,6 +251,8 @@ jobs:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
BUILD_DIR: /__w/openvino/openvino/openvino_build
@ -254,6 +261,9 @@ jobs:
SCCACHE_AZURE_KEY_PREFIX: ubuntu22_x86_64_cc_Release
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Install git
run: apt-get update && apt-get install --assume-yes --no-install-recommends git ca-certificates git-lfs
@ -269,7 +279,7 @@ jobs:
repository: 'openvinotoolkit/testdata'
path: ${{ env.MODELS_PATH }}
lfs: 'true'
ref: 'master'
ref: 'releases/2023/3'
- name: Download selective build statistics package
uses: actions/download-artifact@v3
@ -288,9 +298,9 @@ jobs:
run: bash ${OPENVINO_REPO}/install_build_dependencies.sh
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
#
# Build
#

View File

@ -64,6 +64,9 @@ jobs:
CCACHE_MAXSIZE: 50G
if: "!needs.smart_ci.outputs.skip_workflow"
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Install git
run: apt-get update && apt-get install --assume-yes --no-install-recommends git ca-certificates
@ -86,6 +89,7 @@ jobs:
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/json
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/gtest
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/gflags
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/telemetry
git submodule update --init -- ${OPENVINO_REPO}/src/plugins/intel_cpu
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/open_model_zoo
popd

View File

@ -73,6 +73,7 @@ jobs:
OPENVINO_REPO: ${{ github.workspace }}/openvino
OPENVINO_CONTRIB_REPO: ${{ github.workspace }}/openvino_contrib
INSTALL_DIR: ${{ github.workspace }}/openvino_install
INSTALL_DIR_JS: ${{ github.workspace }}/openvino_install/js
INSTALL_TEST_DIR: ${{ github.workspace }}/tests_install
BUILD_DIR: ${{ github.workspace }}/build
steps:
@ -190,6 +191,17 @@ jobs:
-B ${{ env.BUILD_DIR }}
cmake --build ${{ env.BUILD_DIR }} --parallel --config ${{ env.CMAKE_BUILD_TYPE }}
- name: CMake configure, build and install - OpenVINO JS API
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
run: |
cmake \
-DCPACK_GENERATOR=NPM \
-S ${{ env.OPENVINO_REPO }} \
-B ${{ env.BUILD_DIR }}
cmake --build ${{ env.BUILD_DIR }} --parallel
cmake -DCMAKE_INSTALL_PREFIX=${{ env.INSTALL_DIR_JS }} -P ${{ env.BUILD_DIR }}/cmake_install.cmake
#
# Upload build artifacts
#
@ -210,6 +222,14 @@ jobs:
path: ${{ env.BUILD_DIR }}/openvino_tests.tar.gz
if-no-files-found: 'error'
- name: Upload openvino js package
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
uses: actions/upload-artifact@v3
with:
name: openvino_js_package
path: ${{ env.INSTALL_DIR_JS }}
if-no-files-found: 'error'
Samples:
needs: [ Build, Smart_CI ]
if: fromJSON(needs.smart_ci.outputs.affected_components).samples
@ -218,6 +238,14 @@ jobs:
runner: 'macos-13'
affected-components: ${{ needs.smart_ci.outputs.affected_components }}
JS_API:
name: OpenVINO JS API
needs: [ Build, Smart_CI ]
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
uses: ./.github/workflows/job_openvino_js.yml
with:
runner: 'macos-13'
CXX_Unit_Tests:
name: C++ unit tests
needs: [ Build, Smart_CI ]

View File

@ -72,6 +72,7 @@ jobs:
OPENVINO_REPO: ${{ github.workspace }}/openvino
OPENVINO_CONTRIB_REPO: ${{ github.workspace }}/openvino_contrib
INSTALL_DIR: ${{ github.workspace }}/openvino_install
INSTALL_DIR_JS: ${{ github.workspace }}/openvino_install/js
INSTALL_TEST_DIR: ${{ github.workspace }}/tests_install
BUILD_DIR: ${{ github.workspace }}/build
steps:
@ -189,6 +190,17 @@ jobs:
-B ${{ env.BUILD_DIR }}
cmake --build ${{ env.BUILD_DIR }} --parallel --config ${{ env.CMAKE_BUILD_TYPE }}
- name: CMake configure, build and install - OpenVINO JS API
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
run: |
cmake \
-DCPACK_GENERATOR=NPM \
-S ${{ env.OPENVINO_REPO }} \
-B ${{ env.BUILD_DIR }}
cmake --build ${{ env.BUILD_DIR }} --parallel
cmake -DCMAKE_INSTALL_PREFIX=${{ env.INSTALL_DIR_JS }} -P ${{ env.BUILD_DIR }}/cmake_install.cmake
#
# Upload build artifacts
#
@ -209,6 +221,14 @@ jobs:
path: ${{ env.BUILD_DIR }}/openvino_tests.tar.gz
if-no-files-found: 'error'
- name: Upload openvino js package
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
uses: actions/upload-artifact@v3
with:
name: openvino_js_package
path: ${{ env.INSTALL_DIR_JS }}
if-no-files-found: 'error'
Samples:
needs: Build
uses: ./.github/workflows/job_samples_tests.yml
@ -216,6 +236,14 @@ jobs:
runner: 'macos-13-xlarge'
affected-components: ${{ needs.smart_ci.outputs.affected_components }}
JS_API:
name: OpenVINO JS API
needs: [ Build, Smart_CI ]
if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
uses: ./.github/workflows/job_openvino_js.yml
with:
runner: 'macos-13-xlarge'
CXX_Unit_Tests:
name: C++ unit tests
needs: [ Build, Smart_CI ]

View File

@ -29,7 +29,7 @@ jobs:
python-version: '3.10'
- name: Cache pip
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('tools/mo/requirements*.txt') }}

View File

@ -51,6 +51,8 @@ jobs:
CMAKE_BUILD_TYPE: 'Release'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
OPENVINO_REPO: /__w/openvino/openvino/openvino
OPENVINO_BUILD_DIR: /__w/openvino/openvino/openvino_build
SCCACHE_AZURE_KEY_PREFIX: webassembly_Release
@ -66,9 +68,9 @@ jobs:
submodules: 'true'
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: emcmake cmake - configure
run: |

View File

@ -51,9 +51,13 @@ jobs:
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
SCCACHE_IDLE_TIMEOUT: 0
OPENVINO_REPO: "${{ github.workspace }}\\openvino"
OPENVINO_CONTRIB_REPO: "${{ github.workspace }}\\openvino_contrib"
INSTALL_DIR: "${{ github.workspace }}\\openvino_install"
INSTALL_DIR_JS: "${{ github.workspace }}\\openvino_install\\js"
INSTALL_TEST_DIR: "${{ github.workspace }}\\tests_install"
BUILD_DIR: "${{ github.workspace }}\\openvino_build"
# TODO: specify version of compiler here
@ -72,7 +76,7 @@ jobs:
with:
repository: 'openvinotoolkit/openvino_contrib'
path: 'openvino_contrib'
ref: 'master'
ref: 'releases/2023/3'
#
# Print system info
@ -117,12 +121,16 @@ jobs:
python3 -m pip install certifi
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: Install build dependencies
run: choco install --no-progress ninja
run: |
Invoke-WebRequest https://github.com/ninja-build/ninja/releases/download/v1.11.1/ninja-win.zip -OutFile ninja-win.zip -MaximumRetryCount 10
Expand-Archive -Force ninja-win.zip
# Add it to the GitHub Path so it would be available in the subsequent steps
Add-Content -Path $env:GITHUB_PATH -Value "${{ github.workspace }}/ninja-win"
#
# Build
@ -194,6 +202,16 @@ jobs:
-B ${{ env.BUILD_DIR }}
cmake --build ${{ env.BUILD_DIR }} --parallel --config ${{ env.CMAKE_BUILD_TYPE }} --verbose
- name: CMake configure, build and install - OpenVINO JS API
if: ${{ 'false' }} # 128689
# if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
run:
cmake -DCPACK_GENERATOR=NPM -DENABLE_SYSTEM_TBB=OFF -UTBB* -S ${{ env.OPENVINO_REPO }} -B ${{ env.BUILD_DIR }}
cmake --build ${{ env.BUILD_DIR }} --parallel
cmake -DCMAKE_INSTALL_PREFIX=${{ env.INSTALL_DIR_JS }} -P ${{ env.BUILD_DIR }}/cmake_install.cmake
- name: Upload openvino package
uses: actions/upload-artifact@v3
with:
@ -208,6 +226,15 @@ jobs:
path: ${{ env.BUILD_DIR }}/openvino_tests.zip
if-no-files-found: 'error'
- name: Upload openvino js package
if: ${{ 'false' }} # 128689
# if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
uses: actions/upload-artifact@v3
with:
name: openvino_js_package
path: ${{ env.INSTALL_DIR_JS }}
if-no-files-found: 'error'
Samples:
needs: [Build, Smart_CI]
if: fromJSON(needs.smart_ci.outputs.affected_components).samples
@ -289,6 +316,51 @@ jobs:
path: ${{ env.INSTALL_TEST_DIR }}/TEST*.xml
if-no-files-found: 'error'
JS_API:
name: JS API
needs: [ Build, Smart_CI ]
defaults:
run:
shell: pwsh
runs-on: 'aks-win-4-cores-8gb'
env:
OPENVINO_JS_DIR: "${{ github.workspace }}\\openvino\\src\\bindings\\js"
OPENVINO_JS_LIBS_DIR: "${{ github.workspace }}\\openvino\\src\\bindings\\js\\node\\bin"
if: ${{ 'false' }} # 128689
# if: fromJSON(needs.smart_ci.outputs.affected_components).JS_API
steps:
- name: Fetch OpenVINO JS sources
uses: actions/checkout@v4
with:
sparse-checkout: |
src/bindings/js
path: 'openvino'
- name: Download OpenVINO js package
uses: actions/download-artifact@v3
with:
name: openvino_js_package
path: ${{ env.OPENVINO_JS_LIBS_DIR }}
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: 18
- name: Configure OpenVINO JS
working-directory: ${{ env.OPENVINO_JS_DIR }}/node
run: npm i
- name: Test OpenVINO JS
working-directory: ${{ env.OPENVINO_JS_DIR }}/node
run: npm test
- name: Test OpenVINO JS (cmd)
shell: cmd
working-directory: ${{ env.OPENVINO_JS_DIR }}/node
run: call npm test
Python_Unit_Tests:
name: Python unit tests
needs: [Build, Smart_CI]
@ -787,7 +859,7 @@ jobs:
run: python3 -m pip install -r ${{ github.workspace }}\install\tests\functional_test_utils\layer_tests_summary\requirements.txt
- name: Restore tests execution time
uses: actions/cache/restore@v3
uses: actions/cache/restore@v4
with:
path: ${{ env.PARALLEL_TEST_CACHE }}
key: ${{ runner.os }}-tests-functional-cpu-stamp-${{ github.sha }}
@ -801,7 +873,7 @@ jobs:
timeout-minutes: 60
- name: Save tests execution time
uses: actions/cache/save@v3
uses: actions/cache/save@v4
if: github.ref_name == 'master'
with:
path: ${{ env.PARALLEL_TEST_CACHE }}

View File

@ -48,12 +48,15 @@ jobs:
defaults:
run:
shell: pwsh
runs-on: aks-win-16-cores-32gb
runs-on: aks-win-8-cores-64gb
env:
CMAKE_BUILD_TYPE: 'Release'
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
SCCACHE_IDLE_TIMEOUT: 0
OPENVINO_REPO: "${{ github.workspace }}\\openvino"
INSTALL_DIR: "${{ github.workspace }}\\openvino_install"
INSTALL_TEST_DIR: "${{ github.workspace }}\\tests_install"
@ -77,7 +80,7 @@ jobs:
repository: 'openvinotoolkit/testdata'
path: 'testdata'
lfs: 'true'
ref: 'master'
ref: 'releases/2023/3'
#
# Print system info
@ -98,12 +101,16 @@ jobs:
self-hosted-runner: 'false'
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"
version: "v0.7.5"
- name: Install build dependencies
run: choco install --no-progress ninja
run: |
Invoke-WebRequest https://github.com/ninja-build/ninja/releases/download/v1.11.1/ninja-win.zip -OutFile ninja-win.zip -MaximumRetryCount 10
Expand-Archive -Force ninja-win.zip
# Add it to the GitHub Path so it would be available in the subsequent steps
Add-Content -Path $env:GITHUB_PATH -Value "${{ github.workspace }}/ninja-win"
- name: Install python dependencies
run: |
@ -250,11 +257,14 @@ jobs:
defaults:
run:
shell: pwsh
runs-on: aks-win-16-cores-32gb
runs-on: aks-win-8-cores-64gb
env:
CMAKE_BUILD_TYPE: 'Release'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
SCCACHE_IDLE_TIMEOUT: 0
OPENVINO_REPO: "${{ github.workspace }}\\openvino"
BUILD_DIR: "${{ github.workspace }}\\openvino_build"
MODELS_PATH: "${{ github.workspace }}\\testdata"
@ -275,7 +285,7 @@ jobs:
repository: 'openvinotoolkit/testdata'
path: 'testdata'
lfs: 'true'
ref: 'master'
ref: 'releases/2023/3'
- name: Download selective build statistics package
uses: actions/download-artifact@v3
@ -294,7 +304,7 @@ jobs:
self-hosted-runner: 'false'
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.5.4"

3
.gitmodules vendored
View File

@ -75,3 +75,6 @@
[submodule "src/plugins/intel_cpu/thirdparty/mlas"]
path = src/plugins/intel_cpu/thirdparty/mlas
url = https://github.com/openvinotoolkit/mlas.git
[submodule "thirdparty/telemetry"]
path = thirdparty/telemetry
url = https://github.com/openvinotoolkit/telemetry.git

View File

@ -28,6 +28,11 @@ if(POLICY CMP0091)
cmake_policy(SET CMP0091 NEW) # Enables use of MSVC_RUNTIME_LIBRARY
endif()
# Avoid warning about DOWNLOAD_EXTRACT_TIMESTAMP in CMake 3.24:
if(POLICY CMP0135)
cmake_policy(SET CMP0135 NEW)
endif()
project(OpenVINO DESCRIPTION "OpenVINO toolkit")
find_package(OpenVINODeveloperScripts REQUIRED

View File

@ -61,7 +61,7 @@ product better.
[./docs/dev](https://github.com/openvinotoolkit/openvino/tree/master/docs/dev) folder.
* **User documentation** is built from several sources and published at
[docs.openvino.ai](docs.openvino.ai), which is the recommended place for reading
[docs.openvino.ai](https://docs.openvino.ai/), which is the recommended place for reading
these documents. Use the files maintained in this repository only for editing purposes.
* The easiest way to help with documentation is to review it and provide feedback on the
@ -69,7 +69,7 @@ product better.
or think more information should be added, you can reach out to any of the documentation
contributors to discuss the potential changes.
You can also create a Pull Request directly, following the [editor's guide](./docs/CONTRIBUTING_DOCS.md).
You can also create a Pull Request directly, following the [editor's guide](./CONTRIBUTING_DOCS.md).
### Promote and Support OpenVINO
@ -151,4 +151,4 @@ We'll make sure to review your Pull Request as soon as possible and provide you
## License
By contributing to the OpenVINO project, you agree that your contributions will be
licensed under the terms stated in the [LICENSE](./LICENSE.md) file.
licensed under the terms stated in the [LICENSE](./LICENSE) file.

View File

@ -56,7 +56,7 @@ Regardless of the automated tests, you should ensure the quality of your changes
## Need Additional Help? Check these Articles
* [How to create a fork](https://help.github.com/articles/fork-a-rep)
* [How to create a fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo)
* [Install Git](https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup)
* If you want to add a new sample, please have a look at the Guide for contributing
to C++/C/Python IE samples and add the license statement at the top of new files for

View File

@ -1,5 +1,5 @@
<div align="center">
<img src="docs/img/openvino-logo-purple-black.png" width="400px">
<img src="docs/sphinx_setup/_static/images/img/openvino-logo-purple-black.png" width="400px">
[![PyPI Status](https://badge.fury.io/py/openvino.svg)](https://badge.fury.io/py/openvino)
[![Anaconda Status](https://anaconda.org/conda-forge/openvino/badges/version.svg)](https://anaconda.org/conda-forge/openvino)
@ -67,18 +67,18 @@ The OpenVINO™ Runtime can infer models on different hardware devices. This sec
<tbody>
<tr>
<td rowspan=2>CPU</td>
<td> <a href="https://docs.openvino.ai/2023.2/openvino_docs_OV_UG_supported_plugins_CPU.html#doxid-openvino-docs-o-v-u-g-supported-plugins-c-p-u">Intel CPU</a></tb>
<td> <a href="https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_CPU.html#doxid-openvino-docs-o-v-u-g-supported-plugins-c-p-u">Intel CPU</a></tb>
<td><b><i><a href="./src/plugins/intel_cpu">openvino_intel_cpu_plugin</a></i></b></td>
<td>Intel Xeon with Intel® Advanced Vector Extensions 2 (Intel® AVX2), Intel® Advanced Vector Extensions 512 (Intel® AVX-512), and AVX512_BF16, Intel Core Processors with Intel AVX2, Intel Atom Processors with Intel® Streaming SIMD Extensions (Intel® SSE), Intel® Advanced Matrix Extensions (Intel® AMX)</td>
</tr>
<tr>
<td> <a href="https://docs.openvino.ai/2023.2/openvino_docs_OV_UG_supported_plugins_CPU.html#doxid-openvino-docs-o-v-u-g-supported-plugins-c-p-u">ARM CPU</a></tb>
<td> <a href="https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_CPU.html#doxid-openvino-docs-o-v-u-g-supported-plugins-c-p-u">ARM CPU</a></tb>
<td><b><i><a href="./src/plugins/intel_cpu">openvino_arm_cpu_plugin</a></i></b></td>
<td>Raspberry Pi™ 4 Model B, Apple® Mac mini with Apple silicon
</tr>
<tr>
<td>GPU</td>
<td><a href="https://docs.openvino.ai/2023.2/openvino_docs_OV_UG_supported_plugins_GPU.html#doxid-openvino-docs-o-v-u-g-supported-plugins-g-p-u">Intel GPU</a></td>
<td><a href="https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_GPU.html#doxid-openvino-docs-o-v-u-g-supported-plugins-g-p-u">Intel GPU</a></td>
<td><b><i><a href="./src/plugins/intel_gpu">openvino_intel_gpu_plugin</a></i></b></td>
<td>Intel Processor Graphics, including Intel HD Graphics and Intel Iris Graphics</td>
</tr>
@ -102,22 +102,22 @@ OpenVINO™ Toolkit also contains several plugins which simplify loading models
</thead>
<tbody>
<tr>
<td><a href="https://docs.openvino.ai/2023.2/openvino_docs_OV_UG_supported_plugins_AUTO.html">Auto</a></td>
<td><a href="https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_supported_plugins_AUTO.html">Auto</a></td>
<td><b><i><a href="./src/plugins/auto">openvino_auto_plugin</a></i></b></td>
<td>Auto plugin enables selecting Intel device for inference automatically</td>
</tr>
<tr>
<td><a href="https://docs.openvino.ai/2023.2/openvino_docs_OV_UG_Automatic_Batching.html">Auto Batch</a></td>
<td><a href="https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Automatic_Batching.html">Auto Batch</a></td>
<td><b><i><a href="./src/plugins/auto_batch">openvino_auto_batch_plugin</a></i></b></td>
<td>Auto batch plugin performs on-the-fly automatic batching (i.e. grouping inference requests together) to improve device utilization, with no programming effort from the user</td>
</tr>
<tr>
<td><a href="https://docs.openvino.ai/2023.2/openvino_docs_OV_UG_Hetero_execution.html#doxid-openvino-docs-o-v-u-g-hetero-execution">Hetero</a></td>
<td><a href="https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Hetero_execution.html#doxid-openvino-docs-o-v-u-g-hetero-execution">Hetero</a></td>
<td><b><i><a href="./src/plugins/hetero">openvino_hetero_plugin</a></i></b></td>
<td>Heterogeneous execution enables automatic inference splitting between several devices</td>
</tr>
<tr>
<td><a href="https://docs.openvino.ai/2023.2/openvino_docs_OV_UG_Running_on_multiple_devices.html#doxid-openvino-docs-o-v-u-g-running-on-multiple-devices">Multi</a></td>
<td><a href="https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_Running_on_multiple_devices.html#doxid-openvino-docs-o-v-u-g-running-on-multiple-devices">Multi</a></td>
<td><b><i><a href="./src/plugins/auto">openvino_auto_plugin</a></i></b></td>
<td>Multi plugin enables simultaneous inference of the same model on several devices in parallel</td>
</tr>
@ -164,9 +164,9 @@ The list of OpenVINO tutorials:
## System requirements
The system requirements vary depending on platform and are available on dedicated pages:
- [Linux](https://docs.openvino.ai/2023.2/openvino_docs_install_guides_installing_openvino_linux_header.html)
- [Windows](https://docs.openvino.ai/2023.2/openvino_docs_install_guides_installing_openvino_windows_header.html)
- [macOS](https://docs.openvino.ai/2023.2/openvino_docs_install_guides_installing_openvino_macos_header.html)
- [Linux](https://docs.openvino.ai/2023.3/openvino_docs_install_guides_installing_openvino_linux_header.html)
- [Windows](https://docs.openvino.ai/2023.3/openvino_docs_install_guides_installing_openvino_windows_header.html)
- [macOS](https://docs.openvino.ai/2023.3/openvino_docs_install_guides_installing_openvino_macos_header.html)
## How to build
@ -206,6 +206,6 @@ Report questions, issues and suggestions, using:
\* Other names and brands may be claimed as the property of others.
[Open Model Zoo]:https://github.com/openvinotoolkit/open_model_zoo
[OpenVINO™ Runtime]:https://docs.openvino.ai/2023.2/openvino_docs_OV_UG_OV_Runtime_User_Guide.html
[OpenVINO Model Converter (OVC)]:https://docs.openvino.ai/2023.2/openvino_docs_model_processing_introduction.html#convert-a-model-in-cli-ovc
[OpenVINO™ Runtime]:https://docs.openvino.ai/2023.3/openvino_docs_OV_UG_OV_Runtime_User_Guide.html
[OpenVINO Model Converter (OVC)]:https://docs.openvino.ai/2023.3/openvino_docs_model_processing_introduction.html#convert-a-model-in-cli-ovc
[Samples]:https://github.com/openvinotoolkit/openvino/tree/master/samples

View File

@ -206,6 +206,8 @@ set(CMAKE_POLICY_DEFAULT_CMP0025 NEW)
set(CMAKE_POLICY_DEFAULT_CMP0026 NEW)
# CMake 3.0+ (2.8.12): MacOS "@rpath" in target's install name
set(CMAKE_POLICY_DEFAULT_CMP0042 NEW)
# CMake 3.1+: Simplify variable reference and escape sequence evaluation.
set(CMAKE_POLICY_DEFAULT_CMP0053 NEW)
# CMake 3.9+: `RPATH` settings on macOS do not affect `install_name`.
set(CMAKE_POLICY_DEFAULT_CMP0068 NEW)
# CMake 3.12+: find_package() uses <PackageName>_ROOT variables.

View File

@ -3,15 +3,15 @@
#
if(WIN32)
set(PROGRAMFILES_ENV "ProgramFiles(X86)")
set(PROGRAMFILES_ENV "ProgramFiles\(X86\)")
# check that PROGRAMFILES_ENV is defined, because in case of cross-compilation for Windows
# we don't have such variable
if(DEFINED ENV{PROGRAMFILES_ENV})
if(DEFINED ENV{${PROGRAMFILES_ENV}})
file(TO_CMAKE_PATH $ENV{${PROGRAMFILES_ENV}} PROGRAMFILES)
set(WDK_PATHS "${PROGRAMFILES}/Windows Kits/10/bin/${CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION}/x64"
"${PROGRAMFILES}/Windows Kits/10/bin/x64")
"${PROGRAMFILES}/Windows Kits/10/bin/x64")
message(STATUS "Trying to find apivalidator in: ")
foreach(wdk_path IN LISTS WDK_PATHS)
@ -19,9 +19,9 @@ if(WIN32)
endforeach()
find_host_program(ONECORE_API_VALIDATOR
NAMES apivalidator
PATHS ${WDK_PATHS}
DOC "ApiValidator for OneCore compliance")
NAMES apivalidator
PATHS ${WDK_PATHS}
DOC "ApiValidator for OneCore compliance")
if(ONECORE_API_VALIDATOR)
message(STATUS "Found apivalidator: ${ONECORE_API_VALIDATOR}")

View File

@ -25,38 +25,10 @@ macro(ov_npm_cpack_set_dirs)
set(OV_CPACK_DEVREQDIR .)
set(OV_CPACK_PYTHONDIR .)
if(WIN32)
set(OV_CPACK_LIBRARYDIR .)
set(OV_CPACK_RUNTIMEDIR .)
set(OV_CPACK_ARCHIVEDIR .)
elseif(APPLE)
set(OV_CPACK_LIBRARYDIR .)
set(OV_CPACK_RUNTIMEDIR .)
set(OV_CPACK_ARCHIVEDIR .)
else()
set(OV_CPACK_LIBRARYDIR .)
set(OV_CPACK_RUNTIMEDIR .)
set(OV_CPACK_ARCHIVEDIR .)
endif()
set(OV_CPACK_LIBRARYDIR .)
set(OV_CPACK_ARCHIVEDIR .)
set(OV_CPACK_PLUGINSDIR .)
set(OV_CPACK_IE_CMAKEDIR .)
set(OV_CPACK_NGRAPH_CMAKEDIR .)
set(OV_CPACK_OPENVINO_CMAKEDIR .)
set(OV_CPACK_DOCDIR .)
set(OV_CPACK_LICENSESDIR licenses)
set(OV_CPACK_PYTHONDIR .)
# non-native stuff
set(OV_CPACK_SHAREDIR .)
set(OV_CPACK_SAMPLESDIR .)
set(OV_CPACK_DEVREQDIR .)
unset(OV_CPACK_SHAREDIR)
# skipped during debian packaging
set(OV_CPACK_WHEELSDIR .)
set(OV_CPACK_RUNTIMEDIR .)
endmacro()
ov_npm_cpack_set_dirs()

View File

@ -122,6 +122,32 @@ function(ov_install_with_name file component)
endif()
endfunction()
#
# checks that current OpenVINO versions has previous version in RPM / DEB conflicts
#
function(ov_check_conflicts_versions var_name)
set(ov_major ${OpenVINO_VERSION_MAJOR})
set(ov_minor ${OpenVINO_VERSION_MINOR})
set(ov_patch ${OpenVINO_VERSION_PATCH})
if(ov_patch EQUAL 0)
if(ov_minor EQUAL 0)
math(EXPR ov_major "${ov_major} - 1")
else()
math(EXPR ov_minor "${ov_minor} - 1")
endif()
else()
math(EXPR ov_patch "${ov_patch} - 1")
endif()
set(ov_prev_version "${ov_major}.${ov_minor}.${ov_patch}")
# perform check
if(NOT ov_prev_version IN_LIST ${var_name})
message(FATAL_ERROR "List ${var_name} (${${var_name}}) does not contain verison ${ov_prev_version}")
endif()
endfunction()
#
# List of public OpenVINO components
#

View File

@ -193,6 +193,9 @@ ov_dependent_option (ENABLE_SYSTEM_SNAPPY "Enables use of system version of Snap
ov_dependent_option (ENABLE_PYTHON_PACKAGING "Enables packaging of Python API in APT / YUM" OFF
"ENABLE_PYTHON;UNIX" OFF)
ov_dependent_option (ENABLE_JS "Enables JS API building" ON
"NOT WIN32" OFF)
ov_option(ENABLE_OPENVINO_DEBUG "Enable output for OPENVINO_DEBUG statements" OFF)
if(NOT BUILD_SHARED_LIBS AND ENABLE_OV_TF_FRONTEND)

View File

@ -92,6 +92,8 @@ macro(ov_cpack_settings)
2023.2.0
)
ov_check_conflicts_versions(conflicting_versions)
#
# core: base dependency for each component
#

View File

@ -78,6 +78,8 @@ macro(ov_cpack_settings)
2023.2.0
)
ov_check_conflicts_versions(conflicting_versions)
find_host_program(rpmlint_PROGRAM NAMES rpmlint DOC "Path to rpmlint")
if(rpmlint_PROGRAM)
execute_process(COMMAND "${rpmlint_PROGRAM}" --version

View File

@ -59,7 +59,7 @@ function(build_docs)
if(${ENABLE_PYTHON_API})
list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "STARTED preprocessing OpenVINO Python API")
list(APPEND commands COMMAND ${Python3_EXECUTABLE} -m pip install openvino)
list(APPEND commands COMMAND ${Python3_EXECUTABLE} -m pip install openvino==2023.3)
list(APPEND commands COMMAND ${CMAKE_COMMAND} -E cmake_echo_color --green "FINISHED preprocessing OpenVINO Python API")
endif()

Binary file not shown.

Before

Width:  |  Height:  |  Size: 81 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 101 KiB

View File

@ -12,6 +12,7 @@ About OpenVINO
compatibility_and_support
system_requirements
Release Notes <openvino_release_notes>
release_policy
Additional Resources <resources>
OpenVINO is a toolkit for simple and efficient deployment of various deep learning models.
@ -24,7 +25,7 @@ OpenVINO (Open Visual Inference and Neural network Optimization) is an open-sour
Features
##############################################################
One of the main purposes of OpenVINO is to streamline the deployment of deep learning models in user applications. It optimizes and accelerates model inference, which is crucial for such domains as Generative AI, Large Language models, and use cases like object detection, classification, segmentation, and many others.
One of the main purposes of OpenVINO is to streamline the deployment of deep learning models in user applications. It optimizes and accelerates model inference, which is crucial for such domains as Generative AI, Large Language models, and use cases like object detection, classification, segmentation, and many others.
* :doc:`Model Optimization <openvino_docs_model_optimization_guide>`
@ -32,7 +33,7 @@ OpenVINO provides multiple optimization methods for both the training and post-t
* :doc:`Model Conversion and Framework Compatibility <openvino_docs_model_processing_introduction>`
Supported models can be loaded directly or converted to the OpenVINO format to achieve better performance. Supported frameworks include ONNX, PyTorch, TensorFlow, TensorFlow Lite, Keras, and PaddlePaddle.
Supported models can be loaded directly or converted to the OpenVINO format to achieve better performance. Supported frameworks include ONNX, PyTorch, TensorFlow, TensorFlow Lite, Keras, and PaddlePaddle.
* :doc:`Model Inference <openvino_docs_OV_UG_OV_Runtime_User_Guide>`
@ -40,14 +41,14 @@ OpenVINO accelerates deep learning models on various hardware platforms, ensurin
* `Deployment on a server <https://github.com/openvinotoolkit/model_server>`__
A model can be deployed either locally using OpenVINO Runtime or on a model server. Runtime is a set of C++ libraries with C and Python bindings providing a common API to deliver inference solutions. The model server enables quick model inference using external resources.
A model can be deployed either locally using OpenVINO Runtime or on a model server. Runtime is a set of C++ libraries with C and Python bindings providing a common API to deliver inference solutions. The model server enables quick model inference using external resources.
Architecture
##############################################################
To learn more about how OpenVINO works, read the Developer documentation on its `architecture <https://github.com/openvinotoolkit/openvino/blob/master/src/docs/architecture.md>`__ and `core components <https://github.com/openvinotoolkit/openvino/blob/master/src/README.md>`__.
OpenVINO Ecosystem
OpenVINO Ecosystem
##############################################################
Along with the primary components of model optimization and runtime, the toolkit also includes:

View File

@ -8,7 +8,6 @@ Compatibility and Support
:maxdepth: 1
:hidden:
openvino_supported_models
openvino_docs_OV_UG_supported_plugins_Supported_Devices
openvino_resources_supported_operations
openvino_resources_supported_operations_frontend
@ -16,9 +15,7 @@ Compatibility and Support
:doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>` - compatibility information for supported hardware accelerators.
:doc:`Supported Models <openvino_supported_models>` - a table of models officially supported by OpenVINO.
:doc:`Supported Operations <openvino_resources_supported_operations>` - a listing of framework layers supported by OpenVINO.
:doc:`Supported Operations <openvino_resources_supported_operations>` - a listing of framework layers supported by OpenVINO.
:doc:`Supported Operations <openvino_resources_supported_operations_frontend>` - a listing of layers supported by OpenVINO inference devices.

View File

@ -1,36 +0,0 @@
.. {#openvino_supported_models}
Supported Models
================
.. meta::
:description: Check the list of officially supported models in Intel®
Distribution of OpenVINO™ toolkit.
The OpenVINO team continues the effort to support as many models out-of-the-box as possible.
Based on our research and user feedback, we prioritize the most common models and test them
before every release. These models are considered officially supported.
.. button-link:: _static/download/OV_2023_models_supported.pdf
:color: primary
:outline:
:material-regular:`download;1.5em` Click for supported models [PDF]
The list is based on release 2023.0, as of June 01, 2023
| Note that the list provided here does not include all models supported by OpenVINO.
| If your model is not included but is similar to those that are, it is still very likely to work.
If your model fails to execute properly there are a few options available:
* You can create a GitHub request for the operation(s) that are missing. These requests are reviewed regularly. You will be informed if and how the request will be accommodated. Additionally, your request may trigger a reply from someone in the community who can help.
* As OpenVINO™ is open source you can enhance it with your own contribution to the GitHub repository. To learn more, see the articles on :doc:`OpenVINO Extensibility <openvino_docs_Extensibility_UG_Intro>`.

View File

@ -18,239 +18,372 @@ This page lists operations supported by OpenVINO Framework Frontend.
========================================== ==========================================================================================
PyTorch Supported Operations Limitations
========================================== ==========================================================================================
aten::__and__ Only for boolean inputs
aten::__getitem__ Supported in limitted set of patterns
aten::__not__
aten::__or__
aten::_convolution
aten::_convolution_mode
aten::_set_item
aten::abs
aten::acos
aten::acos_
aten::acosh
aten::acosh_
aten::adaptive_avg_pool2d
aten::adaptive_avg_pool3d
aten::adaptive_max_pool2d
aten::add
aten::add_
aten::addcmul
aten::addmm
aten::append Supported in limitted set of patterns
aten::arange
aten::argsort
aten::as_tensor
aten::asin
aten::asin_
aten::asinh
aten::asinh_
aten::atan
aten::atan_
aten::atanh
aten::atanh_
aten::avg_pool1d
aten::avg_pool2d
aten::avg_pool3d
aten::baddbmm
aten::batch_norm
aten::bitwise_not
aten::bmm
aten::Bool
aten::cat
aten::ceil
aten::ceil_
aten::chunk Supported in limitted set of patterns
aten::clamp
aten::clamp_max
aten::clamp_min
aten::clone
aten::contiguous
aten::conv_transpose1d
aten::conv_transpose2d
aten::conv_transpose3d
aten::conv1d
aten::conv2d
aten::conv3d
aten::convolution
aten::copy
aten::cos
aten::cos_
aten::cosh
aten::cosh_
aten::cumsum
aten::detach
aten::dim
aten::div
aten::div_
aten::dropout
aten::dropout_
aten::einsum Supported in limitted set of patterns
aten::elu
aten::embedding
aten::empty
aten::eq
aten::exp
aten::expand Supported in limitted set of patterns
aten::expand_as
aten::eye
aten::fill_
aten::flatten
aten::floor
aten::floor_
aten::floor_divide
aten::floordiv
aten::full
aten::full_like
aten::gather
aten::ge
aten::gelu
aten::glu
aten::grid_sampler
aten::group_norm
aten::gt
aten::hardsigmoid
aten::hardswish
aten::hardswish_
aten::hardtanh
aten::hardtanh_
aten::im2col
aten::index Supported in limitted set of patterns
aten::index_put_ Supported in limitted set of patterns
aten::index_select
aten::instance_norm
aten::Int
aten::IntImplicit
aten::is_grad_enabled
aten::layer_norm
aten::le
aten::leaky_relu
aten::leaky_relu_
aten::len
aten::linalg_matrix_norm
aten::linalg_norm
aten::linalg_vector_norm
aten::linear
aten::log
aten::log_
aten::log2
aten::log2_
aten::lt
aten::masked_fill
aten::masked_fill_
aten::matmul
aten::max
aten::max_pool1d
aten::max_pool2d
aten::max_pool3d
aten::mean
aten::meshgrid Supported in limitted set of patterns
aten::min
aten::mm
aten::mul
aten::mul_
aten::narrow
aten::ne
aten::neg
aten::new_empty
aten::new_full
aten::new_ones
aten::new_zeros
aten::nonzero
aten::nonzero_numpy Supported in limitted set of patterns
aten::norm
aten::numel
aten::ones
aten::ones_like
aten::pad Supported in limitted set of patterns
aten::permute
aten::pow
aten::reciprocal
aten::relu
aten::relu_
aten::relu6
aten::remainder
aten::repeat
aten::repeat_interleave
aten::reshape
aten::reshape_as
aten::roll
aten::rsqrt
aten::rsub
aten::ScalarImplicit
aten::scaled_dot_product_attention
aten::select
aten::selu
aten::selu_
aten::sigmoid
aten::sigmoid_
aten::sign
aten::silu
aten::silu_
aten::sin
aten::sin_
aten::sinh
aten::sinh_
aten::size
aten::slice
aten::softmax
aten::sort
aten::split Supported in limitted set of patterns
aten::split_with_sizes Supported in limitted set of patterns
aten::sqrt
aten::square
aten::squeeze
aten::stack Supported in limitted set of patterns
aten::sub
aten::sum
aten::tan
aten::tan_
aten::tanh
aten::tanh_
aten::tensor
aten::to
aten::topk
aten::transpose
aten::tril
aten::triu
aten::type_as
aten::unbind Supported in limitted set of patterns
aten::unfold
aten::unsqueeze
aten::unsqueeze_
aten::upsample_bicubic2d
aten::upsample_bilinear2d
aten::upsample_linear1d
aten::upsample_nearest1d
aten::upsample_nearest2d
aten::upsample_nearest3d
aten::upsample_trilinear3d
aten::var
aten::var_mean
aten::view
aten::where Supported in limitted set of patterns
aten::zeros
aten::zeros_like
prim::Constant String and None constant is only supported when used by supported operation
prim::device
prim::DictConstruct Supported in limitted set of patterns
prim::dtype Supported in limitted set of patterns
prim::GetAttr
prim::If
prim::is_cuda
prim::ListConstruct Supported in limitted set of patterns
prim::ListUnpack Supported in limitted set of patterns
prim::Loop
prim::max Supported in limitted set of patterns
prim::min Supported in limitted set of patterns
prim::NumToTensor
prim::PythonOp Supported only in tracing
prim::requires_grad
prim::TupleConstruct Supported in limitted set of patterns
prim::type
torchvision::deform_conv2d
torchvision::nms
torchvision::roi_align
aten::__and__
aten::__derive_index
aten::__getitem__
aten::__not__
aten::__or__
aten::__range_length
aten::__xor__
aten::_convolution
aten::_convolution_mode
aten::_native_multi_head_attention
aten::_set_item
aten::_shape_as_tensor
aten::_upsample_bicubic2d_aa
aten::_upsample_bilinear2d_aa
aten::_weight_norm
aten::abs
aten::acos
aten::acos_
aten::acosh
aten::acosh_
aten::adaptive_avg_pool1d
aten::adaptive_avg_pool2d
aten::adaptive_avg_pool3d
aten::adaptive_max_pool1d
aten::adaptive_max_pool2d
aten::adaptive_max_pool3d
aten::add
aten::add_
aten::addcmul
aten::addmm
aten::alias
aten::alias_copy
aten::all
aten::amax
aten::amin
aten::append Supported in limited set of patterns
aten::arange
aten::argmax
aten::argmin
aten::argsort
aten::as_strided
aten::as_tensor
aten::asin
aten::asin_
aten::asinh
aten::asinh_
aten::atan
aten::atan_
aten::atanh
aten::atanh_
aten::avg_pool1d
aten::avg_pool2d
aten::avg_pool3d
aten::baddbmm
aten::batch_norm
aten::bitwise_and
aten::bitwise_not
aten::bitwise_or
aten::bitwise_xor
aten::bmm
aten::Bool
aten::broadcast_tensors Supported in limited set of patterns
aten::broadcast_to
aten::cat
aten::cdist
aten::ceil
aten::ceil_
aten::channel_shuffle
aten::chunk Supported in limited set of patterns
aten::clamp
aten::clamp_
aten::clamp_max
aten::clamp_min
aten::clip
aten::clip_
aten::clone
aten::complex Supported in limited set of patterns
aten::concat
aten::contiguous
aten::conv1d
aten::conv2d
aten::conv3d
aten::conv_transpose1d
aten::conv_transpose2d
aten::conv_transpose3d
aten::convolution
aten::copy
aten::copy_
aten::cos
aten::cos_
aten::cosh
aten::cosh_
aten::cross
aten::cumsum
aten::dequantize
aten::detach
aten::dim
aten::div
aten::div_
aten::dropout
aten::dropout_
aten::einsum Supported in limited set of patterns
aten::elu
aten::embedding
aten::embedding_bag
aten::empty
aten::empty_like
aten::eq
aten::erf
aten::erf_
aten::erfc
aten::erfc_
aten::exp
aten::exp_
aten::expand
aten::expand_as
aten::eye
aten::fake_quantize_per_channel_affine
aten::fake_quantize_per_tensor_affine
aten::feature_dropout
aten::fft_irfftn Supported in limited set of patterns
aten::fft_rfftn Supported in limited set of patterns
aten::fill
aten::fill_
aten::fill_diagonal_
aten::flatten
aten::flip
aten::floor
aten::floor_
aten::floor_divide
aten::floordiv
aten::fmod
aten::frobenius_norm
aten::full
aten::full_like
aten::gather
aten::ge
aten::gelu
aten::glu
aten::grid_sampler
aten::group_norm
aten::gru
aten::gt
aten::hardsigmoid
aten::hardswish
aten::hardswish_
aten::hardtanh
aten::hardtanh_
aten::im2col
aten::imag Supported in limited set of patterns
aten::index Supported in limited set of patterns
aten::index_put_
aten::index_select
aten::instance_norm
aten::Int
aten::IntImplicit
aten::is_grad_enabled
aten::is_nonzero
aten::item
aten::layer_norm
aten::le
aten::leaky_relu
aten::leaky_relu_
aten::len
aten::lift
aten::lift_fresh
aten::lift_fresh_copy
aten::linalg_cross
aten::linalg_matrix_norm
aten::linalg_norm
aten::linalg_vector_norm
aten::linear
aten::linspace
aten::log
aten::log10
aten::log10_
aten::log1p
aten::log1p_
aten::log2
aten::log2_
aten::log_
aten::log_softmax
aten::logical_and
aten::logical_not
aten::logical_or
aten::logical_xor
aten::lstm
aten::lt
aten::masked_fill
aten::masked_fill_
aten::masked_scatter
aten::masked_scatter_
aten::matmul
aten::max
aten::max_pool1d
aten::max_pool1d_with_indices
aten::max_pool2d
aten::max_pool2d_with_indices
aten::max_pool3d
aten::max_pool3d_with_indices
aten::maximum
aten::mean
aten::meshgrid
aten::min
aten::minimum
aten::mm
aten::mul
aten::mul_
aten::multinomial
aten::multiply
aten::multiply_
aten::narrow
aten::ne
aten::neg
aten::new_empty
aten::new_full
aten::new_ones
aten::new_zeros
aten::nonzero
aten::nonzero_numpy Supported in limited set of patterns
aten::norm
aten::normal
aten::normal_
aten::numel
aten::numpy_T
aten::one_hot
aten::ones
aten::ones_like
aten::outer
aten::pad
aten::pairwise_distance
aten::permute
aten::pixel_shuffle
aten::pixel_unshuffle
aten::pow
aten::pow_
aten::prelu
aten::prod
aten::quantize_per_channel
aten::quantize_per_tensor
aten::rand
aten::rand_like
aten::randint
aten::randn
aten::randn_like
aten::real Supported in limited set of patterns
aten::reciprocal
aten::reflection_pad2d Supported in limited set of patterns
aten::relu
aten::relu6
aten::relu_
aten::remainder
aten::repeat
aten::repeat_interleave
aten::reshape
aten::reshape_as
aten::resolve_conj
aten::resolve_neg
aten::rnn_relu
aten::rnn_tanh
aten::roll
aten::round
aten::rsqrt
aten::rsub
aten::ScalarImplicit
aten::scaled_dot_product_attention
aten::scatter
aten::scatter_
aten::scatter_add
aten::scatter_add_
aten::scatter_reduce
aten::scatter_reduce_
aten::select
aten::selu
aten::selu_
aten::sigmoid
aten::sigmoid_
aten::sign
aten::silu
aten::silu_
aten::sin
aten::sin_
aten::sinh
aten::sinh_
aten::size
aten::slice
aten::softmax
aten::softplus
aten::sort
aten::split Supported in limited set of patterns
aten::split_with_sizes Supported in limited set of patterns
aten::sqrt
aten::square
aten::squeeze
aten::stack Supported in limited set of patterns
aten::std
aten::std_mean
aten::sub
aten::sub_
aten::sum
aten::swapaxes
aten::t
aten::t_
aten::take_along_dim
aten::tan
aten::tan_
aten::tanh
aten::tanh_
aten::tensor
aten::tensor_split Supported in limited set of patterns
aten::tile
aten::to
aten::topk
aten::transpose
aten::tril
aten::tril_
aten::triu
aten::triu_
aten::type_as
aten::unbind Supported in limited set of patterns
aten::unflatten
aten::unfold
aten::unsqueeze
aten::unsqueeze_
aten::upsample_bicubic2d
aten::upsample_bilinear2d
aten::upsample_linear1d
aten::upsample_nearest1d
aten::upsample_nearest2d
aten::upsample_nearest3d
aten::upsample_trilinear3d
aten::var
aten::var_mean
aten::view
aten::view_as
aten::where
aten::zero_
aten::zeros
aten::zeros_like
prim::Constant
prim::device
prim::DictConstruct Supported in limited set of patterns
prim::GetAttr
prim::If
prim::is_cuda
prim::ListConstruct
prim::ListUnpack
prim::Loop
prim::NumToTensor
prim::PythonOp
prim::requires_grad
prim::TupleConstruct Supported in limited set of patterns
prim::TupleIndex
prim::TupleUnpack Supported in limited set of patterns
prim::type
quantized::add
quantized::add_relu
quantized::cat
quantized::conv2d
quantized::conv2d_relu
quantized::hardswish
quantized::linear
quantized::mul
torchvision::deform_conv2d
torchvision::nms
torchvision::roi_align
========================================== ==========================================================================================
.. tab-item:: ONNX

View File

@ -7,163 +7,171 @@ Supported Operations - by Inference Devices
This page lists operations supported by OpenVINO inference devices. The table presents general information,
for a more detailed and most recent listing of operations that are implemented and tested:
.. button-link:: _static/download/operation_conformance_table_files/opset_report_omz_static.html
.. button-link:: _static/download/conformance_reports/opset_report_omz_static.html
:color: primary
:outline:
See the full conformance report table
See the full conformance report table (static)
.. button-link:: _static/download/conformance_reports/opset_report_omz_dynamic.html
:color: primary
:outline:
See the full conformance report table (dynamic)
================================= =============== ============== ================ ==================
Operations CPU (x86) GPU GNA CPU (Arm®)
================================= =============== ============== ================ ==================
Abs Supported** Supported Not Supported Supported
================================= =============== ============== ================ ==================
Abs Supported** Supported Not Supported Supported
Acos Supported** Supported Not Supported Supported****
Acosh Supported** Supported Not Supported Supported****
Activation-Clamp Supported*** Supported Supported Supported
Activation-ELU Supported*** Supported Not Supported Supported
Activation-Exp Supported*** Supported Supported Supported
Activation-Leaky ReLU Supported*** Supported Supported Not Supported
Activation-Not Supported*** Supported Not Supported Not Supported
Activation-PReLU Supported*** Supported Not Supported Supported
Activation-ReLU Supported*** Supported Supported Supported
Activation-ReLU6 Supported*** Supported Not Supported Not Supported
Activation-Sigmoid/Logistic Supported*** Supported Supported Supported
Activation-TanH Supported*** Supported Supported Supported
ArgMax Supported** Supported Not Supported Not Supported
Activation-Clamp Supported*** Supported Supported Supported
Activation-ELU Supported*** Supported Not Supported Supported
Activation-Exp Supported*** Supported Supported Supported
Activation-Leaky ReLU Supported*** Supported Supported Not Supported
Activation-Not Supported*** Supported Not Supported Not Supported
Activation-PReLU Supported*** Supported Not Supported Supported
Activation-ReLU Supported*** Supported Supported Supported
Activation-ReLU6 Supported*** Supported Not Supported Not Supported
Activation-Sigmoid/Logistic Supported*** Supported Supported Supported
Activation-TanH Supported*** Supported Supported Supported
ArgMax Supported** Supported Not Supported Not Supported
Asin Supported** Supported Not Supported Supported****
Asinh Supported** Supported Not Supported Supported****
Atan Supported** Supported Not Supported Supported****
Atanh Supported** Supported Not Supported Supported****
BatchNormalization Supported Supported Not Supported Supported
BinaryConvolution Supported Supported Not Supported Not Supported
Broadcast Supported** Supported Not Supported Supported
Ceil Supported** Supported Not Supported Supported
Concat Supported*** Supported Supported Supported
Const Supported Supported Supported Supported
Convolution-Dilated Supported Supported Not Supported Supported
Convolution-Dilated 3D Supported Supported Not Supported Not Supported
Convolution-Grouped Supported Supported Not Supported Supported
Convolution-Grouped 3D Supported Supported Not Supported Not Supported
Convolution-Ordinary Supported Supported Supported* Supported
Convolution-Ordinary 3D Supported Supported Not Supported Not Supported
BatchNormalization Supported Supported Not Supported Supported
BinaryConvolution Supported Supported Not Supported Not Supported
Broadcast Supported** Supported Not Supported Supported
Ceil Supported** Supported Not Supported Supported
Concat Supported*** Supported Supported Supported
Const Supported Supported Supported Supported
Convolution-Dilated Supported Supported Not Supported Supported
Convolution-Dilated 3D Supported Supported Not Supported Not Supported
Convolution-Grouped Supported Supported Not Supported Supported
Convolution-Grouped 3D Supported Supported Not Supported Not Supported
Convolution-Ordinary Supported Supported Supported* Supported
Convolution-Ordinary 3D Supported Supported Not Supported Not Supported
Cos Supported** Supported Not Supported Supported****
Cosh Supported** Supported Not Supported Supported****
Crop Supported Supported Supported Not Supported
Crop Supported Supported Supported Not Supported
CTCGreedyDecoder Supported** Supported** Not Supported Supported****
Deconvolution Supported Supported Not Supported Not Supported
Deconvolution 3D Supported Supported Not Supported Not Supported
DeformableConvolution Supported Supported Not Supported Not Supported
DepthToSpace Supported** Supported Not Supported Supported*
Deconvolution Supported Supported Not Supported Not Supported
Deconvolution 3D Supported Supported Not Supported Not Supported
DeformableConvolution Supported Supported Not Supported Not Supported
DepthToSpace Supported** Supported Not Supported Supported*
DetectionOutput Supported** Supported Not Supported Supported****
Eltwise-And Supported*** Supported Not Supported Supported
Eltwise-Add Supported*** Supported Not Supported Supported
Eltwise-Div Supported*** Supported Not Supported Supported
Eltwise-Equal Supported*** Supported Not Supported Supported*
Eltwise-And Supported*** Supported Not Supported Supported
Eltwise-Add Supported*** Supported Not Supported Supported
Eltwise-Div Supported*** Supported Not Supported Supported
Eltwise-Equal Supported*** Supported Not Supported Supported*
Eltwise-FloorMod Supported*** Supported Not Supported Supported****
Eltwise-Greater Supported*** Supported Not Supported Supported
Eltwise-GreaterEqual Supported*** Supported Not Supported Supported
Eltwise-Less Supported*** Supported Not Supported Supported*
Eltwise-LessEqual Supported*** Supported Not Supported Supported*
Eltwise-LogicalAnd Supported*** Supported Not Supported Supported
Eltwise-LogicalOr Supported*** Supported Not Supported Supported
Eltwise-LogicalXor Supported*** Supported Not Supported Supported
Eltwise-Max Supported*** Supported Not Supported Supported
Eltwise-Min Supported*** Supported Not Supported Supported
Eltwise-Mul Supported*** Supported Supported Supported
Eltwise-NotEqual Supported*** Supported Not Supported Supported*
Eltwise-Pow Supported*** Supported Not Supported Supported
Eltwise-Prod Supported*** Supported Supported Not Supported
Eltwise-SquaredDiff Supported*** Supported Not Supported Supported
Eltwise-Sub Supported*** Supported Supported Supported
Eltwise-Greater Supported*** Supported Not Supported Supported
Eltwise-GreaterEqual Supported*** Supported Not Supported Supported
Eltwise-Less Supported*** Supported Not Supported Supported*
Eltwise-LessEqual Supported*** Supported Not Supported Supported*
Eltwise-LogicalAnd Supported*** Supported Not Supported Supported
Eltwise-LogicalOr Supported*** Supported Not Supported Supported
Eltwise-LogicalXor Supported*** Supported Not Supported Supported
Eltwise-Max Supported*** Supported Not Supported Supported
Eltwise-Min Supported*** Supported Not Supported Supported
Eltwise-Mul Supported*** Supported Supported Supported
Eltwise-NotEqual Supported*** Supported Not Supported Supported*
Eltwise-Pow Supported*** Supported Not Supported Supported
Eltwise-Prod Supported*** Supported Supported Not Supported
Eltwise-SquaredDiff Supported*** Supported Not Supported Supported
Eltwise-Sub Supported*** Supported Supported Supported
Eltwise-Sum Supported*** Supported Supported Supported****
Erf Supported** Supported Not Supported Supported****
Exp Supported Supported Supported Supported
FakeQuantize Supported Not Supported Not Supported Supported*
Fill Supported** Not Supported Not Supported Not Supported
Flatten Supported Supported Not Supported Not Supported
Floor Supported** Supported Not Supported Supported
FullyConnected (Inner Product) Supported*** Supported Supported Supported
Gather Supported** Supported Not Supported Supported*
Exp Supported Supported Supported Supported
FakeQuantize Supported Not Supported Not Supported Supported*
Fill Supported** Not Supported Not Supported Not Supported
Flatten Supported Supported Not Supported Not Supported
Floor Supported** Supported Not Supported Supported
FullyConnected (Inner Product) Supported*** Supported Supported Supported
Gather Supported** Supported Not Supported Supported*
GatherTree Supported** Not Supported Not Supported Supported****
Gemm Supported Supported Not Supported Not Supported
GRN Supported** Supported** Not Supported Supported
Gemm Supported Supported Not Supported Not Supported
GRN Supported** Supported** Not Supported Supported
HardSigmoid Supported** Supported Not Supported Supported****
Interp Supported** Supported** Not Supported Supported*
Log Supported** Supported Supported Supported
LRN (Norm) Supported Supported Not Supported Supported*
LSTMCell Supported Supported Supported Supported
GRUCell Supported Supported Supported Supported
RNNCell Supported Supported Not Supported Supported
Interp Supported** Supported** Not Supported Supported*
Log Supported** Supported Supported Supported
LRN (Norm) Supported Supported Not Supported Supported*
LSTMCell Supported Supported Supported Supported
GRUCell Supported Supported Supported Supported
RNNCell Supported Supported Not Supported Supported
LSTMSequence Supported Supported Supported Supported****
GRUSequence Supported Supported Supported Supported****
RNNSequence Supported Supported Not Supported Supported****
LogSoftmax Supported** Supported Not Supported Supported
Memory Supported Not Supported Supported Not Supported
MVN Supported** Supported Not Supported Supported*
Neg Supported** Supported Not Supported Supported
LogSoftmax Supported** Supported Not Supported Supported
Memory Supported Not Supported Supported Not Supported
MVN Supported** Supported Not Supported Supported*
Neg Supported** Supported Not Supported Supported
NonMaxSuppression Supported** Not Supported Not Supported Supported****
Normalize Supported** Supported Not Supported Supported*
Normalize Supported** Supported Not Supported Supported*
OneHot Supported** Supported Not Supported Supported****
Pad Supported** Supported Not Supported Supported*
Permute Supported Supported Supported* Not Supported
Pooling(AVG,MAX) Supported Supported Supported Supported
Pooling(AVG,MAX) 3D Supported Supported Not Supported Supported*
Power Supported** Supported Supported* Supported
PowerFile Supported** Not Supported Not Supported Not Supported
PriorBox Supported** Supported Not Supported Supported
PriorBoxClustered Supported** Supported** Not Supported Supported
Pad Supported** Supported Not Supported Supported*
Permute Supported Supported Supported* Not Supported
Pooling(AVG,MAX) Supported Supported Supported Supported
Pooling(AVG,MAX) 3D Supported Supported Not Supported Supported*
Power Supported** Supported Supported* Supported
PowerFile Supported** Not Supported Not Supported Not Supported
PriorBox Supported** Supported Not Supported Supported
PriorBoxClustered Supported** Supported** Not Supported Supported
Proposal Supported** Supported Not Supported Supported****
PSROIPooling Supported** Supported Not Supported Supported****
Range Supported** Not Supported Not Supported Not Supported
Reciprocal Supported** Supported Not Supported Not Supported
Range Supported** Not Supported Not Supported Not Supported
Reciprocal Supported** Supported Not Supported Not Supported
ReduceAnd Supported** Supported Not Supported Supported****
ReduceL1 Supported** Supported Not Supported Supported
ReduceL2 Supported** Supported Not Supported Supported
ReduceLogSum Supported** Supported Not Supported Supported
ReduceLogSumExp Supported** Supported Not Supported Not Supported
ReduceMax Supported** Supported Not Supported Supported
ReduceMean Supported** Supported Not Supported Supported
ReduceMin Supported** Supported Not Supported Supported
ReduceL1 Supported** Supported Not Supported Supported
ReduceL2 Supported** Supported Not Supported Supported
ReduceLogSum Supported** Supported Not Supported Supported
ReduceLogSumExp Supported** Supported Not Supported Not Supported
ReduceMax Supported** Supported Not Supported Supported
ReduceMean Supported** Supported Not Supported Supported
ReduceMin Supported** Supported Not Supported Supported
ReduceOr Supported** Supported Not Supported Supported****
ReduceProd Supported** Supported Not Supported Supported
ReduceSum Supported** Supported Not Supported Supported
ReduceSumSquare Supported** Supported Not Supported Not Supported
ReduceProd Supported** Supported Not Supported Supported
ReduceSum Supported** Supported Not Supported Supported
ReduceSumSquare Supported** Supported Not Supported Not Supported
RegionYolo Supported** Supported Not Supported Supported****
ReorgYolo Supported** Supported Not Supported Supported
Resample Supported** Supported Not Supported Not Supported
Reshape Supported*** Supported Supported Supported
ReorgYolo Supported** Supported Not Supported Supported
Resample Supported** Supported Not Supported Not Supported
Reshape Supported*** Supported Supported Supported
ReverseSequence Supported** Supported Not Supported Supported****
RNN Supported Not Supported Not Supported Supported
RNN Supported Not Supported Not Supported Supported
ROIPooling Supported Supported* Not Supported Supported****
ScaleShift Supported*** Supported Supported Not Supported
ScatterUpdate Supported** Not Supported Not Supported Not Supported
Select Supported Supported Not Supported Supported
ScaleShift Supported*** Supported Supported Not Supported
ScatterUpdate Supported** Not Supported Not Supported Not Supported
Select Supported Supported Not Supported Supported
Selu Supported** Supported Not Supported Supported****
ShuffleChannels Supported** Supported Not Supported Supported
Sign Supported** Supported Not Supported Supported
Sin Supported** Supported Not Supported Supported
ShuffleChannels Supported** Supported Not Supported Supported
Sign Supported** Supported Not Supported Supported
Sin Supported** Supported Not Supported Supported
Sinh Supported** Supported Not Supported Supported****
SimplerNMS Supported** Supported Not Supported Not Supported
Slice Supported*** Supported Supported Not Supported
SoftMax Supported*** Supported Not Supported Supported
Softplus Supported** Supported Not Supported Supported
Softsign Supported** Supported Supported Not Supported
SpaceToDepth Supported** Not Supported Not Supported Supported*
SpatialTransformer Supported** Not Supported Not Supported Not Supported
Split Supported*** Supported Supported Supported
Squeeze Supported** Supported Supported Supported
StridedSlice Supported** Supported Not Supported Supported*
SimplerNMS Supported** Supported Not Supported Not Supported
Slice Supported*** Supported Supported Not Supported
SoftMax Supported*** Supported Not Supported Supported
Softplus Supported** Supported Not Supported Supported
Softsign Supported** Supported Supported Not Supported
SpaceToDepth Supported** Not Supported Not Supported Supported*
SpatialTransformer Supported** Not Supported Not Supported Not Supported
Split Supported*** Supported Supported Supported
Squeeze Supported** Supported Supported Supported
StridedSlice Supported** Supported Not Supported Supported*
Tan Supported** Supported Not Supported Supported****
TensorIterator Supported Not Supported Supported Supported
Tile Supported*** Supported** Not Supported Supported
TensorIterator Supported Not Supported Supported Supported
Tile Supported*** Supported** Not Supported Supported
TopK Supported** Supported Not Supported Supported****
Unpooling Not Supported Supported Not Supported Not Supported
Unsqueeze Supported** Supported Supported Supported
Upsampling Not Supported Supported Not Supported Not Supported
Unpooling Not Supported Supported Not Supported Not Supported
Unsqueeze Supported** Supported Supported Supported
Upsampling Not Supported Supported Not Supported Not Supported
================================= =============== ============== ================ ==================
| `*` - support is limited to the specific parameters. Refer to "Known Layer Limitations" section for the device :doc:`from the list of supported <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`.
| `*` - support is limited to the specific parameters.
| `**` - support is implemented via :doc:`Extensibility mechanism <openvino_docs_Extensibility_UG_Intro>`.
| `***` - supports NCDHW layout.
| `****` - support is implemented via runtime reference.

View File

@ -5,8 +5,8 @@ Performance Benchmarks
.. meta::
:description: Use the benchmark results for Intel® Distribution of OpenVINO™
toolkit, that may help you decide what hardware to use or how
:description: Use the benchmark results for Intel® Distribution of OpenVINO™
toolkit, that may help you decide what hardware to use or how
to plan the workload.
.. toctree::
@ -15,13 +15,12 @@ Performance Benchmarks
openvino_docs_performance_benchmarks_faq
OpenVINO Accuracy <openvino_docs_performance_int8_vs_fp32>
Performance Data Spreadsheet (download xlsx) <https://docs.openvino.ai/2023.2/_static/benchmarks_files/OV-2023.2-Performance-Data.xlsx>
openvino_docs_MO_DG_Getting_Performance_Numbers
This page presents benchmark results for `Intel® Distribution of OpenVINO™ toolkit <https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html>`__
This page presents benchmark results for `Intel® Distribution of OpenVINO™ toolkit <https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html>`__
and :doc:`OpenVINO Model Server <ovms_what_is_openvino_model_server>`, for a representative selection of public neural networks and Intel® devices.
The results may help you decide which hardware to use in your applications or plan AI workload for the hardware you have already implemented in your solutions.
The results may help you decide which hardware to use in your applications or plan AI workload for the hardware you have already implemented in your solutions.
Click the buttons below to see the chosen benchmark data.
.. grid:: 1 1 2 2
@ -44,53 +43,71 @@ Click the buttons below to see the chosen benchmark data.
:color: primary
:outline:
:expand:
:material-regular:`bar_chart;1.4em` OVMS Benchmark Graphs
For a successful deep learning inference application, the following four key metrics need to be considered:
Please visit the tabs below for more information on key performance indicators and workload parameters.
.. tab-set::
.. tab-item:: Throughput
:sync: throughput
Measures the number of inferences delivered within a latency threshold
(for example, number of Frames Per Second - FPS). When deploying a system with
deep learning inference, select the throughput that delivers the best trade-off
Measures the number of inferences delivered within a latency threshold
(for example, number of Frames Per Second - FPS). When deploying a system with
deep learning inference, select the throughput that delivers the best trade-off
between latency and power for the price and performance that meets your requirements.
.. tab-item:: Value
:sync: value
While throughput is important, what is more critical in edge AI deployments is
the performance efficiency or performance-per-cost. Application performance in
throughput per dollar of system cost is the best measure of value. The value KPI is
calculated as “Throughput measured as inferences per second / price of inference engine”.
This means for a 2 socket system 2x the price of a CPU is used. Prices are as per
While throughput is important, what is more critical in edge AI deployments is
the performance efficiency or performance-per-cost. Application performance in
throughput per dollar of system cost is the best measure of value. The value KPI is
calculated as “Throughput measured as inferences per second / price of inference engine”.
This means for a 2 socket system 2x the price of a CPU is used. Prices are as per
date of benchmarking and sources can be found as links in the Hardware Platforms (PDF) description below.
.. tab-item:: Efficiency
:sync: efficiency
System power is a key consideration from the edge to the data center. When selecting
deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider.
Intel designs provide excellent power efficiency for running deep learning workloads.
The efficiency KPI is calculated as “Throughput measured as inferences per second / TDP of
inference engine”. This means for a 2 socket system 2x the power dissipation (TDP) of a CPU is used.
System power is a key consideration from the edge to the data center. When selecting
deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider.
Intel designs provide excellent power efficiency for running deep learning workloads.
The efficiency KPI is calculated as “Throughput measured as inferences per second / TDP of
inference engine”. This means for a 2 socket system 2x the power dissipation (TDP) of a CPU is used.
TDP-values are as per date of benchmarking and sources can be found as links in the Hardware Platforms (PDF) description below.
.. tab-item:: Latency
:sync: latency
This measures the synchronous execution of inference requests and is reported in milliseconds.
Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before
the next is started. This performance metric is relevant in usage scenarios where a single image
input needs to be acted upon as soon as possible. An example would be the healthcare sector where
medical personnel only request analysis of a single ultra sound scanning image or in real-time or
near real-time applications for example an industrial robot's response to actions in its environment
This measures the synchronous execution of inference requests and is reported in milliseconds.
Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before
the next is started. This performance metric is relevant in usage scenarios where a single image
input needs to be acted upon as soon as possible. An example would be the healthcare sector where
medical personnel only request analysis of a single ultra sound scanning image or in real-time or
near real-time applications for example an industrial robot's response to actions in its environment
or obstacle avoidance for autonomous vehicles.
.. tab-item:: Workload Parameters
:sync: workloadparameters
The workload parameters affect the performance results of the different models we use for benchmarking.
Image processing models have different image size definitions and the Natural Language Processing models
have different max token list lengths. All these can be found in detail in the :doc:`FAQ section <openvino_docs_performance_benchmarks_faq>`.
All models are executed using a batch size of 1. Below are the parameters for the GenAI models we display.
* Input tokens: 1024,
* Output tokens: 128,
* number of beams: 1
For text to image:
* iteration steps: 20,
* image size (HxW): 256 x 256,
* input token length: 1024 (the tokens for GenAI models are in English).
Platforms, Configurations, Methodology
###########################################################
@ -102,53 +119,60 @@ For a listing of all platforms and configurations used for testing, refer to the
.. grid-item::
.. button-link:: _static/benchmarks_files/OV-2023.2-platform_list.pdf
.. button-link:: _static/benchmarks_files/OV-2023.3-platform_list.pdf
:color: primary
:outline:
:expand:
:material-regular:`download;1.5em` Click for Hardware Platforms [PDF]
.. button-link:: _static/benchmarks_files/OV-2023.2-system-info-detailed.xlsx
.. button-link:: _static/benchmarks_files/OV-2023.3-system-info-detailed.xlsx
:color: primary
:outline:
:expand:
:material-regular:`download;1.5em` Click for Configuration Details [XLSX]
.. button-link:: _static/benchmarks_files/OV-2023.3-Performance-Data.xlsx
:color: primary
:outline:
:expand:
:material-regular:`download;1.5em` Click for Performance Data [XLSX]
The OpenVINO benchmark setup includes a single system with OpenVINO™, as well as the benchmark application installed.
It measures the time spent on actual inference (excluding any pre or post processing) and then reports on the inferences
per second (or Frames Per Second).
It measures the time spent on actual inference (excluding any pre or post processing) and then reports on the inferences
per second (or Frames Per Second).
OpenVINO™ Model Server (OVMS) employs the Intel® Distribution of OpenVINO™ toolkit runtime libraries and exposes a set of
OpenVINO™ Model Server (OVMS) employs the Intel® Distribution of OpenVINO™ toolkit runtime libraries and exposes a set of
models via a convenient inference API over gRPC or HTTP/REST. Its benchmark results are measured with the configuration of
multiple-clients-single-server, using two hardware platforms connected by ethernet. Network bandwidth depends on both, platforms
and models under investigation. It is set not to be a bottleneck for workload intensity. The connection is dedicated
only to measuring performance.
multiple-clients-single-server, using two hardware platforms connected by ethernet. Network bandwidth depends on both, platforms
and models under investigation. It is set not to be a bottleneck for workload intensity. The connection is dedicated
only to measuring performance.
.. dropdown:: See more details about OVMS benchmark setup
The benchmark setup for OVMS consists of four main parts:
.. image:: _static/images/performance_benchmarks_ovms_02.png
:alt: OVMS Benchmark Setup Diagram
* **OpenVINO™ Model Server** is launched as a docker container on the server platform and it listens (and answers on)
requests from clients. OpenVINO™ Model Server is run on the same machine as the OpenVINO™ toolkit benchmark application
in corresponding benchmarking. Models served by OpenVINO™ Model Server are located in a local file system mounted into
* **OpenVINO™ Model Server** is launched as a docker container on the server platform and it listens (and answers on)
requests from clients. OpenVINO™ Model Server is run on the same machine as the OpenVINO™ toolkit benchmark application
in corresponding benchmarking. Models served by OpenVINO™ Model Server are located in a local file system mounted into
the docker container. The OpenVINO™ Model Server instance communicates with other components via ports over a dedicated docker network.
* **Clients** are run in separated physical machine referred to as client platform. Clients are implemented in Python3
programming language based on TensorFlow* API and they work as parallel processes. Each client waits for a response from OpenVINO™
* **Clients** are run in separated physical machine referred to as client platform. Clients are implemented in Python3
programming language based on TensorFlow* API and they work as parallel processes. Each client waits for a response from OpenVINO™
Model Server before it will send a new next request. The role played by the clients is also verification of responses.
* **Load balancer** works on the client platform in a docker container. HAProxy is used for this purpose. Its main role is
counting of requests forwarded from clients to OpenVINO™ Model Server, estimating its latency, and sharing this information by
Prometheus service. The reason of locating the load balancer on the client site is to simulate real life scenario that includes
* **Load balancer** works on the client platform in a docker container. HAProxy is used for this purpose. Its main role is
counting of requests forwarded from clients to OpenVINO™ Model Server, estimating its latency, and sharing this information by
Prometheus service. The reason of locating the load balancer on the client site is to simulate real life scenario that includes
impact of physical network on reported metrics.
* **Execution Controller** is launched on the client platform. It is responsible for synchronization of the whole measurement process,
* **Execution Controller** is launched on the client platform. It is responsible for synchronization of the whole measurement process,
downloading metrics from the load balancer, and presenting the final report of the execution.
@ -158,9 +182,9 @@ Test performance yourself
You can also test performance for your system yourself, following the guide on :doc:`getting performance numbers <openvino_docs_MO_DG_Getting_Performance_Numbers>`.
Performance of a particular application can also be evaluated virtually using `Intel® DevCloud for the Edge <https://devcloud.intel.com/edge/>`__.
It is a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit.
To learn more about it, visit `the website <https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/overview.html>`__
Performance of a particular application can also be evaluated virtually using `Intel® DevCloud for the Edge <https://devcloud.intel.com/edge/>`__.
It is a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit.
To learn more about it, visit `the website <https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/overview.html>`__
or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devcloud-enrollment/account-provisioning.html>`__.
@ -168,11 +192,11 @@ or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devc
Disclaimers
####################################
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2023.2, as of November 15, 2023.
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2023.3, as of February 13, 2024.
* OpenVINO Model Server performance results are based on release 2023.0, as of June 01, 2023.
* OpenVINO Model Server performance results are based on release 2023.3, as of February 13, 2024.
The results may not reflect all publicly available updates. Intel technologies features and benefits depend on system configuration
The results may not reflect all publicly available updates. Intel technologies features and benefits depend on system configuration
and may require enabled hardware, software, or service activation. Learn more at intel.com, or from the OEM or retailer.
See configuration disclosure for details. No product can be absolutely secure.

View File

@ -16,9 +16,7 @@ Test performance with the benchmark_app
You can run OpenVINO benchmarks in both C++ and Python APIs, yet the experience differs in each case.
The Python one is part of OpenVINO Runtime installation, while C++ is available as a code sample.
For a detailed description, see:
* :doc:`benchmark_app for C++ <openvino_inference_engine_samples_benchmark_app_README>`
* :doc:`benchmark_app for Python <openvino_inference_engine_tools_benchmark_tool_README>`.
For a detailed description, see: :doc:`benchmark_app <openvino_sample_benchmark_tool>`.
Make sure to install the latest release package with support for frameworks of the models you want to test.
For the most reliable performance benchmarks, :doc:`prepare the model for use with OpenVINO <openvino_docs_model_processing_introduction>`.
@ -87,7 +85,7 @@ slower than the subsequent ones, an aggregated value can be used for the executi
When comparing the OpenVINO Runtime performance with the framework or another reference code, make sure that both versions are as similar as possible:
- Wrap the exact inference execution (for examples, see :doc:`Benchmark app <openvino_inference_engine_samples_benchmark_app_README>`).
- Wrap the exact inference execution (for examples, see :doc:`Benchmark app <openvino_sample_benchmark_tool>`).
- Do not include model loading time.
- Ensure that the inputs are identical for OpenVINO Runtime and the framework. For example, watch out for random values that can be used to populate the inputs.
- In situations when any user-side pre-processing should be tracked separately, consider :doc:`image pre-processing and conversion <openvino_docs_OV_UG_Preprocessing_Overview>`.
@ -98,7 +96,7 @@ Internal Inference Performance Counters and Execution Graphs
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
More detailed insights into inference performance breakdown can be achieved with device-specific performance counters and/or execution graphs.
Both :doc:`C++ <openvino_inference_engine_samples_benchmark_app_README>` and :doc:`Python <openvino_inference_engine_tools_benchmark_tool_README>`
Both :doc:`C++ and Python <openvino_sample_benchmark_tool>`
versions of the *benchmark_app* support a ``-pc`` command-line parameter that outputs internal execution breakdown.
For example, the table shown below is part of performance counters for quantized

View File

@ -31,10 +31,8 @@ Performance Information F.A.Q.
All of the performance benchmarks are generated using the
open-source tool within the Intel® Distribution of OpenVINO™ toolkit
called ``benchmark_app``. This tool is available
:doc:`for C++ apps <openvino_inference_engine_samples_benchmark_app_README>`.
as well as
:doc:`for Python apps <openvino_inference_engine_tools_benchmark_tool_README>`.
called :doc:`benchmark_app <openvino_sample_benchmark_tool>`.
This tool is available for Python and C++ apps.
For a simple instruction on testing performance, see the :doc:`Getting Performance Numbers Guide <openvino_docs_MO_DG_Getting_Performance_Numbers>`.
@ -59,6 +57,10 @@ Performance Information F.A.Q.
- Meta AI
- Auto regressive language
- 4096
* - `Mistral-7b <https://huggingface.co/mistralai/Mistral-7B-v0.1>`__
- Mistral AI
- Auto regressive language
- 4096
* - `Stable-Diffusion-V2-1 <https://huggingface.co/stabilityai/stable-diffusion-2-1>`__
- Hugginface
- Latent Diffusion Model
@ -71,18 +73,14 @@ Performance Information F.A.Q.
- BERT-large
- question / answer
- 384
* - `deeplabv3 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/deeplabv3>`__
- DeepLab v3 Tf
- semantic segmentation
- 513x513
* - `efficientdet-d0 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/efficientdet-d0-tf>`__
- Efficientdet
- classification
- 512x512
* - `faster_rcnn_resnet50_coco <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/faster_rcnn_resnet50_coco>`__
- Faster RCNN TF
- object detection
- 600x1024
* - `mask_rcnn_resnet50_atrous_coco <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mask_rcnn_resnet50_atrous_coco>`__
- Mask R-CNN ResNet 50 Atrous
- object instance segmentation
- 800x1365
* - `mobilenet-v2 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mobilenet-v2-pytorch>`__
- Mobilenet V2 PyTorch
- classification
@ -94,40 +92,36 @@ Performance Information F.A.Q.
* - `ssd-mobilenet-v1-coco <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssd_mobilenet_v1_coco>`__
- ssd-mobilenet-V1-coco onnx model
- object detection
- 300x300
- 300x300
* - `ssd-resnet34-1200-onnx <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssd-resnet34-1200-onnx>`__
- ssd-resnet34 onnx model
- object detection
- 1200x1200
- 1200x1200
* - `unet-camvid-onnx-0001 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/unet-camvid-onnx-0001>`__
- U-Net
- semantic segmentation
- 368x480
* - `yolo-v3 <https://https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v3-tf>`__
- YOLO v3
- object detection
- 416x416
- 368x480
* - `yolo-v3-tiny <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v3-tiny-tf>`__
- YOLO v3 Tiny
- object detection
- 416x416
- 416x416
* - `yolov8n <https://https://github.com/ultralytics/ultralytics>`__
- Yolov8nano
- Yolov8nano
- object detection
- 608x608
.. dropdown:: Where can I purchase the specific hardware used in the benchmarking?
Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the
`Intel® AI: In Production Partners & Solutions Catalog <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html>`__.
Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the
`Intel® AI: In Production Partners & Solutions Catalog <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html>`__.
For more details, see the :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`.
documentation. Before purchasing any hardware, you can test and run
models remotely, using `Intel® DevCloud for the Edge <http://devcloud.intel.com/edge/>`__.
.. dropdown:: How can I optimize my models for better performance or accuracy?
Set of guidelines and recommendations to optimize models are available in the
Set of guidelines and recommendations to optimize models are available in the
:doc:`optimization guide <openvino_docs_deployment_optimization_guide_dldt_optimization_guide>`.
Join the conversation in the `Community Forum <https://software.intel.com/en-us/forums/intel-distribution-of-openvino-toolkit>`__ for further support.
@ -166,4 +160,3 @@ Performance Information F.A.Q.
for autonomous vehicles, where a quick response to the result of the
inference is required.

View File

@ -5,15 +5,18 @@ Model Accuracy
The following two tables present the absolute accuracy drop calculated as the accuracy difference
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8 and FP16
representations of a model on three platform architectures. Please also refer to notes below table
for more information.
The following two tables present the absolute accuracy drop calculated as the accuracy difference
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and
FP16 representations of a model on three platform architectures. Please also refer to notes below
The following two tables present the absolute accuracy drop calculated as the accuracy difference
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and
FP16 representations of a model on three platform architectures. Please also refer to notes below
the table for more information.
* A - Intel® Core™ i9-9000K (AVX2), INT8 and FP32
* B - Intel® Xeon® 6338, (VNNI), INT8 and FP32
* C - Intel® Flex-170, INT8 and FP16
* C - Intel(R) Xeon 8490H (VNNI, AMX), INT8, BF16, FP32
* D - Intel® Flex-170, INT8 and FP16
.. list-table:: Model Accuracy for INT8
:header-rows: 1
@ -24,104 +27,114 @@ for more information.
- A, INT8
- B, INT8
- C, INT8
- D, INT8
* - bert-base-cased
- SST-2_bert_cased_padded
- accuracy
- -0.76%
- 2.42%
- 2.72%
- spearman@cosine
- 3.17%
- 2.68%
- 3.00%
- 2.73%
* - bert-large-uncased-whole-word-masking-squad-0001
- SQUAD_v1_1_bert_msl384_mql64_ds128_lowercase
- F1
- 0.07%
- -0.03%
- 0.13%
- 0.11%
* - deeplabv3
- VOC2012_segm
- mean_iou
- 0.49%
- 0.23%
- -0.16%
* - efficientdet-d0
- COCO2017_detection_91cl
- coco_precision
- -0.84%
- -0.59%
- -0.62%
- -0.63%
* - faster_rcnn_resnet50_coco
* - mask_rcnn_resnet50_atrous_coco
- COCO2017_detection_91cl_bkgr
- coco_orig_precision
- -0.19%
- -0.19%
- -0.04%
- 0.03%
- 0.08%
- 0.11%
- 0.07%
* - mobilenet-v2
- ImageNet2012
- accuracy @ top1
-
- %
- -0.97%
- -0.97%
- -0.95%
* - resnet-50
- ImageNet2012
- accuracy @ top1
- -0.09%
- -0.12%
- -0.20%
- -0.19%
* - ssd-mobilenet-v1-coco
- COCO2017_detection_80cl_bkgr
- coco-precision
- -2.97%
- -0.29%
- -0.26%
- -0.13%
- -0.15%
* - ssd-resnet34-1200
- COCO2017_detection_80cl_bkgr
- map
- -0.03%
- -0.06%
- -0.01%
- 0.04%
* - ssd-mobilenet-v1-coco
- COCO2017_detection_80cl_bkgr
- coco-precision
- -2.97%
- -0.29%
- -0.31%
- -0.26%
* - unet-camvid-onnx-0001
- CamVid_12cl
- mean_iou @ mean
- -6.32%
- 6.40%
- 6.40%
* - yolo_v3
- COCO2017_detection_80cl
- map
- -0.13%
- -0.26%
- -0.44%
- 6.41%
- 6.40%
* - yolo_v3_tiny
- COCO2017_detection_80cl
- map
- -0.11%
- -0.13%
- -0.15%
- %
- -0.23%
- -0.24%
- -0.66%
* - yolo_v8n
- COCO2017_detection_80cl
- map
- 0.27%
- 0.23%
- 0.17%
- -0.02%
- -0.03%
- -0.06%
- -0.06%
* - chatGLM2-6b
- lambada openai
- ppl
-
- 17.595
-
-
- 17.38
- 17.41
- 17.17
* - Llama-2-7b-chat
- Wiki, StackExch, Crawl
- ppl
-
- 3.268
-
-
- 3.24
- 3.24
- 3.25
* - Stable-Diffusion-V2-1
- LIAON-5B
- CLIP
-
-
-
-
* - Mistral-7b
- proprietary Mistral.ai
- ppl
-
-
-
-
- 3.29
- 3.47
- 3.49
.. list-table:: Model Accuracy for FP32 and FP16 (FP16: Flex-170 only)
.. list-table:: Model Accuracy for BF16, FP32 and FP16 (FP16: Flex-170 only. BF16: Xeon(R) 8490H only)
:header-rows: 1
* - OpenVINO™ Model name
@ -129,53 +142,55 @@ for more information.
- Metric Name
- A, FP32
- B, FP32
- C, FP16
- C, FP32
- C, BF16
- D, FP16
* - bert-base-cased
- SST-2_bert_cased_padded
- accuracy
- spearman@cosine
- 0.00%
- 0.00%
- 0.00%
- -0.09%
- 0.00%
* - bert-large-uncased-whole-word-masking-squad-0001
- SQUAD_v1_1_bert_msl384_mql64_ds128_lowercase
- F1
- 0.04%
- 0.04%
- 0.04%
* - deeplabv3
- VOC2012_segm
- mean_iou
- 0.00%
- 0.00%
- 0.00%
- 0.06%
- 0.04%
* - efficientdet-d0
- COCO2017_detection_91cl
- coco_precision
- -0.02%
- -0.02%
- -0.02%
* - faster_rcnn_resnet50_coco
- -0.02%
- -0.03%
* - mask_rcnn_resnet50_atrous_coco
- COCO2017_detection_91cl_bkgr
- coco_orig_precision
- 0.00%
-
- 0.00%
- -0.01%
- -0.01%
- %
- -0.18%
- 0.02%
* - mobilenet-v2
- ImageNet2012
- accuracy @ top1
- 0.00%
- 0.00%
- 0.00%
- -0.04%
- 0.02%
* - resnet-50
- ImageNet2012
- accuracy @ top1
- 0.02%
- 0.02%
- 0.00%
- 0.00%
- 0.00%
* - ssd-mobilenet-v1-coco
- COCO2017_detection_80cl_bkgr
- coco-precision
- 0.01%
- 0.01%
- 0.01%
* - ssd-resnet34-1200
@ -184,49 +199,74 @@ for more information.
- 0.00%
- 0.00%
- 0.00%
- -0.02%
- 0.02%
* - ssd-mobilenet-v1-coco
- COCO2017_detection_80cl_bkgr
- coco-precision
- 0.01%
- 0.01%
- 0.01%
- 0.05%
- -0.03%
* - unet-camvid-onnx-0001
- CamVid_12cl
- mean_iou @ mean
- 0.00%
- 0.00%
- 0.00%
* - yolo_v3
- COCO2017_detection_80cl
- map
- 0.00%
- 0.00%
- 0.00%
- -0.03%
- -0.03%
* - yolo_v3_tiny
- COCO2017_detection_80cl
- map
- -0.04%
- -0.04%
- 0.02%
- %
- 0.00%
- 0.00%
- 0.00%
- -0.02%
* - yolo_v8n
- COCO2017_detection_80cl
- map
- 0.00%
- 0.00%
- 0.00%
- 0.05%
- -0.03%
* - chatGLM2-6b
- lambada-openai
- lambada openai
- ppl
-
- 17.488
-
-
- 17.48
- 17.56
-
- 17.49
* - Llama-2-7b-chat
- Wiki, StackExch, Crawl
- ppl
-
- 3.262
-
-
- 3.26
- 3.26
-
-
* - Stable-Diffusion-V2-1
- LIAON-5B
- CLIP
-
-
-
-
- 22.48
* - Mistral-7b
- proprietary Mistral.ai
- ppl
-
-
-
- 3.19
- 3.18
-
-
Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop.
For perplexity the values do not indicate a deviation from a reference but are the actual measured accuracy for the model.
Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop.
For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured
accuracy for the model.

View File

@ -0,0 +1,175 @@
.. {#release_policy}
Release Policy
=============================================================================
OpenVINO offers releases of three different types, each targeting a different use case:
* `Regular releases <#regular-releases>`__
* `Long-Term Support <#long-term-support-releases>`__
* `Nightly <#nightly-releases>`__
Regular releases
####################
OpenVINO™ is published multiple times a year, when significant new features and bug fixes have
been completed and validated. For each regular release, a dedicated development branch in GitHub
is created, targeting changes such as:
* New features of gold quality, as well as Beta features, labeled as “preview.”
* Key bug fixes.
* Newest hardware support.
Each regular release is supported until the next version arrives, making it suitable for:
* Most typical use cases (the recommended release type).
* Products requiring frequent changes in supported hardware, libraries, operating systems, and models.
Long-Term Support releases
###########################
Each year's final release becomes a Long-Term Support (LTS) version, which continues to receive
bug fixes and security updates, even after newer versions are published. Therefore, LTS may be
used for production environments where:
* There is no need for frequent changes in hardware or model support.
* New optimizations are not prioritized.
* Upgrading is challenging, e.g., due to high software complexity.
* A legacy feature, discontinued in newer OpenVINO versions, is still required.
**LTS Lifecycle**
* LTS is typically published at the end of every year cycle.
* LTS uses the branch of the last yearly regular release.
* LTS aim to receive an update once a year.
* Security updates are offered for the duration of the entire LTS period, which is two years
(or until superseded by two consecutive LTS versions).
* Updates targeting newly discovered bugs are offered for the period of one year.
.. note::
LTS releases may offer limited distribution options.
**Components covered by LTS**
Not all components associated with the OpenVINO™ toolkit are covered by the LTS policy.
The following elements are not guaranteed to receive updates:
* Components in the deprecation period.
* Preview features (highlighted in the release notes).
* Components not directly connected to the OpenVINO™ workflow, such as: Samples, demos, and Jupyter notebooks.
* OpenVINO tools, such as NNCF and OVMS.
* Code samples used in component testing.
Nightly releases
###########################
OpenVINO nightly releases are the first source of newly added features and priority bug fixes
reported for the previous versions, as a preview of the most recent changes. They are:
* Released every workday.
* Based on the master branch of the OpenVINO GitHub repository.
* Not fit for production environments.
* Offered with limited distribution options:
Since their validation scope is limited, **they should never be used for production purposes**.
Instead, they may serve:
* Early integration testing.
* Community contribution development and integration.
* Tracking development progress.
.. tab-set::
.. tab-item:: Downloadable Archives
:sync: archives-s3
1. Go to `OpenVINO Nightly Packages <https://storage.openvinotoolkit.org/repositories/openvino/packages/nightly/>`__.
2. Select a package you want to install.
3. Download the archive for your platform.
4. Unpack the archive in a convenient location.
5. Once unpacked, proceed as with a regular OpenVINO archive (see :doc:`installation guides <openvino_docs_install_guides_overview>`).
.. tab-item:: OV Wheels on S3
:sync: wheels-s3
A PyPI repository deployed on AWS S3 (`see more details <https://peps.python.org/pep-0503/>`__)
enables the use of regular PyPI without the need to rename wheels. Installation commands vary depending
on the branch:
.. tab-set::
.. tab-item:: Master
:sync: master
.. code-block:: py
pip install --pre openvino --extra-index-url
https://storage.openvinotoolkit.org/simple/wheels/nightly
.. tab-item:: Release
:sync: release
* This command includes **Release Candidates**.
* To use ``extra-index-url``, you need to pass a link containing ``simple``.
* The ``--pre`` allows the installation of dev-builds.
.. code-block:: py
pip install --pre openvino --extra-index-url
https://storage.openvinotoolkit.org/simple/wheels/pre-release
.. tab-item:: OV Wheels on PyPi (not recommended)
:sync: wheels-pypi
Install OV Wheels from PyPI:
.. code-block:: py
pip install openvino-nightly
Additional Information
#########################
| **Determining the OpenVINO Version**
| If you need to operate on a specific OpenVINO release, and you are not sure which version
is included in the installed package, you can verify it in one of two ways:
.. tab-set::
.. tab-item:: Python
:sync: python
Execute the following command within the installed package:
.. code-block:: python
python3 -c "import openvino; print(openvino.__version__)"
.. tab-item:: Archives
:sync: archives
You can find the file version in:
.. code-block:: text
<UNZIPPED_ARCHIVE_ROOT>/runtime/version.txt
| **Issue Reporting**
| To report issues, use the `Intel® Premier Support <https://www.intel.com/content/www/us/en/design/support/ips/training/welcome.html>`__
clearly stating the issue, impact, and the expected timeline.
| **Distribution:**
* `Selector tool <https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/download.html>`__ of all distribution options.
* Source code distribution: `GitHub <https://github.com/openvinotoolkit/openvino>`__ and
`Gitee <https://gitee.com/openvinotoolkit-prc/openvino>`__ .
* Binary distribution:
* Download from `OpenVINO storage <https://storage.openvinotoolkit.org/repositories/openvino/packages/>`__
* `pypi.org <https://pypi.org/project/openvino-dev/>`__
* `DockerHub* <https://hub.docker.com/u/openvino>`__

View File

@ -1,314 +1,345 @@
.. {#openvino_release_notes}
OpenVINO Release Notes
======================
========================================
The Intel® Distribution of OpenVINO™ toolkit is an open-source solution for optimizing
and deploying AI inference in domains such as computer vision,automatic speech
recognition, natural language processing, recommendation systems, and generative AI.
With its plug-in architecture, OpenVINO enables developers to write once and deploy
anywhere. We are proud to announce the release of OpenVINO 2023.2 introducing a range
of new features, improvements, and deprecations aimed at enhancing the developer
experience.
New and changed in 2023.2
2023.3 (LTS) - 24.01.2024
###########################
Summary of major features and improvements
++++++++++++++++++++++++++++++++++++++++++++
* More Generative AI coverage and framework integrations to minimize code changes.
* More Generative AI coverage and framework integrations to minimize code changes.
* **Expanded model support for direct PyTorch model conversion** - automatically convert
additional models directly from PyTorch or execute via ``torch.compile`` with OpenVINO
as the backend.
* **New and noteworthy models supported** - we have enabled models used for chatbots,
instruction following, code generation, and many more, including prominent models
like Llava, chatGLM, Bark (text to audio) and LCM (Latent Consistency Models, an
optimized version of Stable Diffusion).
* **Easier optimization and conversion of Hugging Face models** - compress LLM models
to Int8 with the Hugging Face Optimum command line interface and export models to
the OpenVINO IR format.
* **OpenVINO is now available on Conan** - a package manager which allows more seamless
package management for large scale projects for C and C++ developers.
* Introducing `OpenVINO Gen AI repository <https://github.com/openvinotoolkit/openvino.genai>`__
on GitHub that demonstrates native C and C++ pipeline samples for Large Language Models
(LLMs). String tensors are now supported as inputs and tokenizers natively to reduce
overhead and ease production.
* New and noteworthy models validated; Mistral, Zephyr, Qwen, ChatGLM3, and Baichuan
* New Jupyter Notebooks for
`Latent Consistency Models (LCM) <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/263-latent-consistency-models-image-generation>`__
and `Distil-Whisper <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/267-distil-whisper-asr>`__.
Updated `LLM Chatbot notebook <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/254-llm-chatbot>`__
to include LangChain, Neural Chat, TinyLlama, ChatGLM3, Qwen, Notus and Youri models.
* Torch.compile is now fully integrated with OpenVINO, which now includes a hardware
'options' parameter allowing for seamless inference hardware selection by leveraging
the plugin architecture in OpenVINO.
* Broader Large Language Model (LLM) support and more model compression techniques.
* Broader Large Language Model (LLM) support and more model compression techniques.
* Accelerate inference for LLM models on Intel® CoreTM CPU and iGPU with the
use of Int8 model weight compression.
* Expanded model support for dynamic shapes for improved performance on GPU.
* Preview support for Int4 model format is now included. Int4 optimized model
weights are now available to try on Intel® Core™ CPU and iGPU, to accelerate
models like Llama 2 and chatGLM2.
* The following Int4 model compression formats are supported for inference
in runtime:
* Generative Pre-training Transformer Quantization (GPTQ); with GPTQ-compressed
models, you can access them through the Hugging Face repositories.
* Native Int4 compression through Neural Network Compression Framework (NNCF).
* As part of the Neural Network Compression Framework (NNCF), INT4 weight compression model
formats are now fully supported on Intel® Xeon® CPUs in addition to Intel® Core™ and iGPU,
adding more performance, lower memory usage, and accuracy opportunity when using LLMs.
* Improved performance of transformer based LLM on CPU and GPU using stateful model technique
to increase memory efficiency where internal states are shared among multiple iterations of
inference.
* Easier optimization and conversion of Hugging Face models - compress LLM models to INT8
and INT4 with Hugging Face Optimum command line interface and export models to OpenVINO
format. Note this is part of `Optimum-Intel <https://huggingface.co/docs/optimum/intel/index>`__
which needs to be installed separately.
* Tokenizer and TorchVision transform support is now available in the OpenVINO runtime
(via new API) requiring less preprocessing code and enhancing performance by automatically
handling this model setup. More details on Tokenizers support in Ecosystem section.
* More portability and performance to run AI at the edge, in the cloud, or locally.
* **In 2023.1 we announced full support for ARM** architecture, now we have improved
performance by enabling FP16 model formats for LLMs and integrating additional
acceleration libraries to improve latency.
* Full support for 5th Gen Intel® Xeon® Scalable processors (codename Emerald Rapids).
* Further optimized performance on Intel® Core™ Ultra (codename Meteor Lake) CPU with
latency hint, by leveraging both P-core and E-cores.
* Improved performance on ARM platforms using throughput hint, which increases efficiency
in utilization of CPU cores and memory bandwidth.
* Preview JavaScript API to enable node JS development to access JavaScript binding via
source code. See details below.
* Improved `model serving of LLMs <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/llm_text_generation>`__
through OpenVINO Model Server. This not only enables LLM serving over KServe v2 gRPC
and REST APIs for more flexibility, but also improves throughput by running processing
like tokenization on the server side. More details in the Ecosystem section.
Support Change and Deprecation Notices
++++++++++++++++++++++++++++++++++++++++++
* The OpenVINO™ Development Tools package (pip install openvino-dev) is deprecated
and will be removed from installation options and distribution channels with
2025.0. To learn more, refer to the
:doc:`OpenVINO Legacy Features and Components page <openvino_legacy_features>`.
To ensure optimal performance, install the OpenVINO package (pip install openvino),
which includes essential components such as OpenVINO Runtime, OpenVINO Converter,
and Benchmark Tool.
* The OpenVINO™ Development Tools package (pip install openvino-dev) is deprecated and will be
removed from installation options and distribution channels beginning with 2025.0.
For more details, refer to the :doc:`OpenVINO Legacy Features and Components <openvino_legacy_features>`
page.
* Ubuntu 18.04 support will be discontinued in the 2023.3 LTS release. The recommended version
of Ubuntu is 22.04.
* Starting in release 2023.3 OpenVINO will no longer support Python 3.7 due to the Python
community discontinuing support. Update to a newer version (currently 3.8-3.11) to avoid
interruptions.
* All ONNX Frontend legacy API (known as ONNX_IMPORTER_API) will no longer be available in 2024.0 release.
* ``PerfomanceMode.UNDEFINED`` property as part of the OpenVINO Python API will be
discontinued in the 2024.0 release.
* Tools:
* :doc:`Deployment Manager <openvino_docs_install_guides_deployment_manager_tool>`
is deprecated and will be removed in the 2024.0 release.
is deprecated and will be supported for two years according to our LTS policy.
Visit our :doc:`selector tool <openvino_docs_install_guides_overview>` to see
package distribution options or our :doc:`deployment guide <openvino_deployment_guide>`
documentation.
* Accuracy Checker is deprecated and will be discontinued with 2024.0.
* Post-Training Optimization Tool (POT) is deprecated and will be
discontinued with 2024.0.
* Model Optimizer is deprecated and will be fully supported up until the 2025.0
release. Model conversion to the OpenVINO format should be performed through
OpenVINO Model Converter, which is part of the PyPI package. Follow the
:doc:`Model Optimizer to OpenVINO Model Converter transition <openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition>`
guide for smoother transition. Known limitations are TensorFlow model with
TF1 Control flow and object detection models. These limitations relate to
the gap in TensorFlow direct conversion capabilities which will be addressed
in upcoming releases.
* PyTorch 1.13 support is deprecated in Neural Network Compression Framework (NNCF)
* Post-Training Optimization Tool (POT) has been deprecated and the 2023.3 LTS will be
the last release that will support the tool. Developers are encouraged to use the Neural
Network Compression Framework (NNCF) for this feature.
* Model Optimizer is deprecated and will be fully supported until the 2025.0 release.
We encourage developers to perform model conversion through OpenVINO Model Converter
(API call: OVC). Follow the
:doc:`model conversion transition guide <openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition>`
for more details.
* Deprecated support for a `git patch <https://github.com/openvinotoolkit/nncf/tree/develop/third_party_integration/huggingface_transformers>`__
for NNCF integration with `huggingface/transformers <https://github.com/huggingface/transformers>`__.
The recommended approach is to use `huggingface/optimum-intel <https://github.com/huggingface/optimum-intel>`__
for applying NNCF optimization on top of models from Hugging Face.
* Support for Apache MXNet, Caffe, and Kaldi model formats is deprecated and will be
discontinued with the 2024.0 release.
* Runtime:
* Intel® Gaussian & Neural Accelerator (Intel®GNA) will be deprecated in a future
release. We encourage developers to use the Neural Processing Unit (NPU) for
low powered systems like Intel® Core™ Ultra or 14th generation and beyond.
* OpenVINO C++/C/Python 1.0 APIs will be discontinued with 2024.0.
* Python 3.7 support has been discontinued.
* Intel® Gaussian & Neural Accelerator (Intel® GNA) will be deprecated in a future release.
We encourage developers to use the Neural Processing Unit (NPU) for low-powered systems
like Intel® CoreTM Ultra or 14th generation and beyond.
* OpenVINO C++/C/Python 1.0 APIs are deprecated and will be discontinued in the 2024.0 release.
Please use API 2.0 in your applications going forward to avoid disruption.
* OpenVINO property Affinity API will be deprecated from 2024.0 and will be discontinued in 2025.0.
It will be replaced with CPU binding configurations (``ov::hint::enable_cpu_pinning``).
OpenVINO™ Development Tools
++++++++++++++++++++++++++++++++++++++++++
List of components and their changes:
------------------------------------------
* :doc:`OpenVINO Model Converter tool <openvino_docs_model_processing_introduction>`
now supports the original framework shape format.
* `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__
* Added data-free Int4 weight compression support for LLMs in OpenVINO IR with
``nncf.compress_weights()``.
* Improved quantization time of LLMs with NNCF PTQ API for ``nncf.quantize()``
and ``nncf.quantize_with_accuracy_control()``.
* Added support for SmoothQuant and ChannelAlighnment algorithms in NNCF HyperParameter
Tuner for automatic optimization of their hyperparameters during quantization.
* Added quantization support for the ``IF`` operation of models in OpenVINO format
to speed up such models.
* NNCF Post-training Quantization for PyTorch backend is now supported with
``nncf.quantize()`` and the common implementation of quantization algorithms.
* Added support for PyTorch 2.1. PyTorch 1.13 support has been deprecated.
OpenVINO™ Runtime (previously known as Inference Engine)
---------------------------------------------------------
* Weight compression API, ``nncf.compress_weights()``, has been extended by:
* OpenVINO Common
* When using the 'all_layers' parameter, it compresses the model, including embeddings
and final layers, to the 4-bit format. This helps make the model footprint smaller
and improves performance, but it might impact the model accuracy. By default, this
parameter is disabled, and the backup precision (INT8) is assigned for the embeddings
and last layers.
* When using INT8_SYM compression mode for better performance of the compressed model
in case of 8-bit weight compression you might experience an impact on model accuracy
therefore by default, we use INT8_ASYM mode to better balance performance and accuracy.
* We implemented a 4-bit data-aware weight compression feature, introducing the 'dataset'
optional parameter in ``nncf.compress_weights()``. This parameter can be utilized to
mitigate accuracy loss in compressed models. It's important to note that enabling
this option will extend the compression time.
* Post-training Quantization with Accuracy Control, ``nncf.quantize_with_accuracy_control()``,
has been extended by the 'restore_mode' optional parameter to revert weights to INT8
instead of the original precision. This parameter helps to reduce the size of the
quantized model and improves its performance. By default, it is disabled and model
weights are reverted to the original precision in ``nncf.quantize_with_accuracy_control()``.
* Operations for reference implementations updated from legacy API to API 2.0.
* Symbolic transformation introduced the ability to remove Reshape operations
surrounding MatMul operations.
* OpenVINO Python API
* Better support for the ``openvino.properties`` submodule, which now allows the use
of properties directly, without additional parenthesis. Example use-case:
``{openvino.properties.cache_dir: “./some_path/”}``.
* Added missing properties: ``execution_devices`` and ``loaded_from_cache``.
* Improved error propagation on imports from OpenVINO package.
* AUTO device plug-in (AUTO)
* o Provided additional option to improve performance of cumulative throughput
(or MULTI), where part of CPU resources can be reserved for GPU inference
when GPU and CPU are both used for inference (using ``ov::hint::enable_cpu_pinning(true)``).
This avoids the performance issue of CPU resource contention where there
is not enough CPU resources to schedule tasks for GPU
(`PR #19214 <https://github.com/openvinotoolkit/openvino/pull/19214>`__).
* CPU
* Introduced support of GPTQ quantized Int4 models, with improved performance
compared to Int8 weight-compressed or FP16 models. In the CPU plugin,
the gain in performance is achieved by FullyConnected acceleration with
4bit weight decompression
(`PR #20607 <https://github.com/openvinotoolkit/openvino/pull/20607>`__).
* Improved performance of Int8 weight-compressed large language models on
some platforms, such as 13th Gen Intel Core
(`PR #20607 <https://github.com/openvinotoolkit/openvino/pull/20607>`__).
* Further reduced memory consumption of select large language models on
CPU platforms with AMX and AVX512 ISA, by eliminating extra memory copy
with a unified weight layout
(`PR #19575 <https://github.com/openvinotoolkit/openvino/pull/19575>`__).
* Fixed performance issue observed in 2023.1 release on select Xeon CPU
platform with improved thread workload partitioning matching L2 cache
utilization
(`PR #20436 <https://github.com/openvinotoolkit/openvino/pull/20436>`__).
* Extended support of configuration (enable_cpu_pinning) on Windows
platforms to allow fine-grain control on CPU resource used for inference
workload, by binding inference thread to CPU cores
(`PR #19418 <https://github.com/openvinotoolkit/openvino/pull/19418>`__).
* Optimized YoloV8n and YoloV8s model performance for BF16/FP32 precision.
* Optimized Falcon model on 4th Gen Intel® Xeon® Scalable Processors.
* Enabled support for FP16 inference precision on ARM.
* GPU
* Enhanced inference performance for Large Language Models.
* Introduced int8 weight compression to boost LLM performance.
(`PR #19548 <https://github.com/openvinotoolkit/openvino/pull/19548>`__).
* Implemented Int4 GPTQ weight compression for improved LLM performance.
* Optimized constant weights for LLMs, resulting in better memory usage
and faster model loading.
* Optimized gemm (general matrix multiply) and fc (fully connected) for
enhanced performance on iGPU.
(`PR #19780 <https://github.com/openvinotoolkit/openvino/pull/19780>`__).
* Completed GPU plugin migration to API 2.0.
* Added support for oneDNN 3.3 version.
OpenVINO™ Runtime
++++++++++++++++++++++++
* Model Import Updates
* TensorFlow Framework Support
* TensorFlow Framework Support
* Supported conversion of models from memory in keras.Model and tf.function formats.
`PR #19903 <https://github.com/openvinotoolkit/openvino/pull/19903>`__
* Supported TF 2.14.
`PR #20385 <https://github.com/openvinotoolkit/openvino/pull/20385>`__
* Supported TF1 While Control flow construction w/o TensorArray operations
(`PR #20800 <https://github.com/openvinotoolkit/openvino/pull/20800>`__).
* Support for complex tensors has been added
(`PR #20860 <https://github.com/openvinotoolkit/openvino/pull/20860>`__),
(`PR #21477 <https://github.com/openvinotoolkit/openvino/pull/21477>`__).
* Provided fixes for the following:
* PyTorch Framework Support
* Accept any model file extension for frozen protobuf format
(`PR #21508 <https://github.com/openvinotoolkit/openvino/pull/21508>`__).
* Correct ArgMin/ArgMax translators for repeating elements case
(`PR #21364 <https://github.com/openvinotoolkit/openvino/pull/21364>`__).
* Correct PartitionedCall translator when numbers of external and internal
body inputs mismatch
(`PR #20825 <https://github.com/openvinotoolkit/openvino/pull/20825>`__).
* Supported Int4 GPTQ models.
* New operations supported.
* PyTorch Framework Support
* ONNX Framework Support
* Added support of nested dictionaries and lists as example input.
* Disabled ``torch.jit.freeze`` in default model tracing scenario and
improved support for models without freezing, extending model
coverage and improving accuracy for some models.
* Added support for ONNX version 1.14.1
(`PR #18359 <https://github.com/openvinotoolkit/openvino/pull/18359>`__)
* ONNX Framework Support
* Switched to ONNX 1.15.0 as a supported version of original framework
(`PR #20929 <https://github.com/openvinotoolkit/openvino/pull/20929>`__).
* CPU
* Full support for 5th Gen Intel® Xeon® Scalable processors (codename Emerald Rapids)
with sub-numa (SNC) and efficient core resource scheduling to improve performance.
* Further optimized performance on Intel® Core™ Ultra (codename Meteor Lake) CPU with
latency hint, by leveraging both P-core and E-cores.
* Further improved performance of LLMs in INT4 weight compression, especially on 1st
token latency and on 4th and 5th Gen of Intel Xeon platforms (codename Sapphire
Rapids and Emerald Rapids) with AMX capabilities.
* Improved performance of transformer-based LLM using stateful model technique to
increase memory efficiency where internal states (KV cache) are shared among multiple
iterations of inference. The stateful model implementation supports both greedy search
and beam search (preview) for LLMs. This technique also reduces the memory footprint
of LLMs, where Intel Core and Ultra platforms like Raptor Lake and Meteor Lake can
run INT4 models, such as Llama v2 7B.
* Improved performance on ARM platforms with throughput hint, by increasing
efficiency in usage of the CPU cores and memory bandwidth.
* GPU
* Full support for Intel® Core™ Ultra (codename Meteor Lake) integrated graphics.
* For LLMs, the first inference latency for INT8 and INT4 weight-compressed models has
been improved on iGPU thanks to more efficient context processing. Overall average
token latency for INT8 and INT4 has also been enhanced on iGPU with graph compilation
optimization, various host overhead optimization, and dynamic padding support for GEMM.
* Stateful model is functionally supported for LLMs.
* Model caching for dynamically shaped models is now supported. Model loading time is
improved for these models, including LLMs.
* API for switching between size mode (model caching) and speed mode (kernel caching)
is introduced.
* The model cache file name is changed to be independent of GPU driver versions.
The GPU will not generate separate model cache files when the driver is updated.
* Compilation time for Stable Diffusion models has been improved.
* NPU
* NPU plugin is available as part of OpenVINO. With the Intel(R) Core Ultra NPU driver
installed, inference can run on the NPU device.
* AUTO device plug-in (AUTO)
* Introduced the round-robin policy to AUTO cumulative throughput hint, which dispatches
inference requests to multiple devices (such as multiple GPU devices) in the round-robin
sequence, instead of in the device priority sequence. The device priority sequence
remains as the default configuration.
* AUTO loads stateful models to GPU or CPU per device priority, since GPU now supports
stateful model inference.
* OpenVINO Common
* Enhanced support of String tensors has been implemented, enabling the use of operators
and models that rely on string tensors. This update also enhances the capability in
the torchvision preprocessing (`PR #21244 <https://github.com/openvinotoolkit/openvino/pull/20929>`__).
* A new feature has been added that enables the selection of P-Cores for model compilation
on CPU device(s) with hybrid architecture (i.e. Intel® Core™ 12th Gen and beyond).
This will reduce compilation time compared to previous implementation where P-cores
and E-cores are used randomly by OS scheduling.
* OpenVINO JavaScript API (preview feature)
* We've introduced a preview version of
`JS API <https://github.com/openvinotoolkit/openvino/tree/master/src/bindings/js>`__
for OpenVINO runtime in this release. We hope that you will try this feature and
provide your feedback through GitHub `issues <https://github.com/openvinotoolkit/openvino/issues>`__.
* Known limitations:
* Only supported in manylinux and x86 (Windows, ARM, ARM64, and macOS have not been tested)
* Node.js version >= 18.16
* CMake version < 3.14 is not supported
* gcc compiler version < 7 is not supported
* OpenVINO Python API
* Introducing string tensor support for Python API.
* Added support for the following:
* Create ov.Tensor from Python lists
* Create ov.Tensor from empty numpy arrays.
* Constants from empty numpy arrays.
* Autogenerated get/set methods for Node attributes.
* Inference functions (``InferRequest.infer/start_async``, ``CompiledModel.__call__`` etc.) support OVDict as the input.
* PILLOW interpolation modes bindings. (`PR #21188 <https://github.com/openvinotoolkit/openvino/pull/21188>`__ external contribution: @meetpatel0963)
* Torchvision to :doc:`OpenVINO preprocessing <openvino_docs_OV_UG_string_tensors>`
converter documentation has been added to OpenVINO docs.
OpenVINO Ecosystem
+++++++++++++++++++++++++++++++++++++++++++++
OpenVINO Model Server
--------------------------
* OpenVINO Tokenizer (Preview feature)
Introduced an extension of the KServe gRPC API, enabling streaming input and
output for servables with Mediapipe graphs. This extension ensures the persistence
of Mediapipe graphs within a user session, improving processing performance.
This enhancement supports stateful graphs, such as tracking algorithms, and
enables the use of source calculators.
(`see additional documentation <https://github.com/openvinotoolkit/model_server/blob/main/docs/streaming_endpoints.md>`__)
* OpenVINO Tokenizer adds text processing operations to OpenVINO:
* Mediapipe framework has been updated to the version 0.10.3.
* model_api used in the openvino inference Mediapipe calculator has been updated
and included with all its features.
* Added a demo showcasing gRPC streaming with Mediapipe graph.
(`see here <https://github.com/openvinotoolkit/model_server/tree/main/demos/mediapipe/holistic_tracking>`__)
* Added parameters for gRPC quota configuration and changed default gRPC channel
arguments to add rate limits. It will minimize the risks of impact of the service
from uncontrolled flow of requests.
* Updated python clients requirements to match wide range of python versions from 3.6 to 3.11
* Text PrePostprocessing without third-party dependencies
* Convert a HuggingFace tokenizer into the OpenVINO model tokenizer and the
detokenizer using a CLI tool or Python API
* Connect a tokenizer and a model to get a single model with text input
Learn more about the changes in https://github.com/openvinotoolkit/model_server/releases
Jupyter Notebook Tutorials
-----------------------------
* The following notebooks have been updated or newly added:
* `LaBSE <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/220-cross-lingual-books-alignment>`__
Cross-lingual Books Alignment With Transformers
* `LLM chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot>`__
Create LLM-powered Chatbot
* Updated to include Int4 weight compression and Zephyr 7B model
* `Bark Text-to-Speech <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/256-bark-text-to-audio>`__
Text-to-Speech generation using Bark
* `LLaVA Multimodal Chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/257-llava-multimodal-chatbot>`__
Visual-language assistant with LLaVA
* `BLIP-Diffusion - Subject-Driven Generation <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/258-blip-diffusion-subject-generation>`__
Subject-driven image generation and editing using BLIP Diffusion
* `DeciDiffusion <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/259-decidiffusion-image-generation>`__
Image generation with DeciDiffusion
* `Fast Segment Anything <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/261-fast-segment-anything>`__
Object segmentations with FastSAM
* `SoftVC VITS Singing Voice Conversion <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/262-softvc-voice-conversion>`__
* `QR Code Monster <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/264-qrcode-monster>`__
Generate creative QR codes with ControlNet QR Code Monster
* `Würstchen <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/265-wuerstchen-image-generation>`__
Text-to-image generation with Würstchen
* `Distil-Whisper <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/267-distil-whisper-asr>`__
Automatic speech recognition using Distil-Whisper and OpenVINO™
* OpenVINO Tokenizer models work only on the CPU device
* Supported platforms: Linux (x86 and ARM), Windows and Mac (x86 and ARM)
* Added optimization support (8-bit quantization, weight compression)
by NNCF for the following notebooks:
* OpenVINO Model Server
* Added support for serving pipelines with custom nodes implemented as a
`python code <https://github.com/openvinotoolkit/model_server/blob/main/docs/python_support/quickstart.md>`__
This greatly simplifies exposing GenAI algorithms based on Hugging Face
and Optimum libraries. It can be also applied for arbitrary pre and
post-processing in model serving pipelines.
* Included a new set of model serving demos that use custom nodes with python
code. These include LLM `text generation <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/llm_text_generation>`__,
`stable diffusion <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/stable_diffusion>`__,
and `seq2seq translation <https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/seq2seq_translation>`__.
* Improved video stream analysis `demo <https://github.com/openvinotoolkit/model_server/tree/main/demos/real_time_stream_analysis/python>`__.
A simple client example can now process the
video stream from a local camera, video file or RTSP stream.
* Learn more about these changes on
`GitHub <https://github.com/openvinotoolkit/model_server/releases>`__.
* Jupyter Notebook Tutorials
* The following notebooks have been updated or newly added:
* `Sound generation with AudioLDM2 and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/270-sound-generation-audioldm2>`__.
* `Single-step image generation using SDXL-turbo and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/271-sdxl-turbo>`__.
* `Paint by Example using Diffusion models and OpenVINO™ <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/272-paint-by-example>`__.
* `LLM-powered chatbot using Stable-Zephyr-3b and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/273-stable-zephyr-3b-chatbot>`__.
* `Object segmentations with EfficientSAM and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/274-efficient-sam>`__.
* `Create an LLM-powered RAG system using OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot/254-rag-chatbot.ipynb>`__
- Demonstrates an integration with LangChain.
* `High-resolution image generation with Segmind-VegaRT and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/248-stable-diffusion-xl/248-segmind-vegart.ipynb>`__.
* `Text-to-Image Generation with LCM LoRA and ControlNet Conditioning <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/263-latent-consistency-models-image-generation/263-lcm-lora-controlnet.ipynb>`__.
* `LLM Instruction-following pipeline with OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/275-llm-question-answering>`__ -
Demonstrates how to run an instruction-following text generation pipeline using
tiny-llama-1b-chat, phi-2, dolly-v2-3b, red-pajama-3b-instruct and mistral-7b models.
* `LLM chabot notebook <https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/254-llm-chatbot>`__
updated with support for new LLMs and INT4/INT8 Weight Compression: TinyLlama-1b-chat,
Mistral-7B, neural-chat-7b, notus-7b, ChatGLM3, youri-7b-chat (for Japanese language).
* Added optimization support (8-bit quantization, weight compression) by NNCF for the following notebooks:
* `Image generation with Würstchen and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/265-wuerstchen-image-generation>`__
* `QR-code monster <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/264-qrcode-monster>`__
* `INT4-compression support for LLaVA multimodal chatbot <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/257-llava-multimodal-chatbot>`__
* `Distil-whisper quantization <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/267-distil-whisper-asr>`__
* `Image generation with DeepFloyd IF <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/238-deepfloyd-if>`__
* `Instruction following using Databricks Dolly 2.0 <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/240-dolly-2-instruction-following>`__
* `Visual Question Answering and Image Captioning using BLIP <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/233-blip-visual-language-processing>`__
* `Grammatical Error Correction <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/214-grammar-correction>`__
* `Universal segmentation with OneFormer <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/249-oneformer-segmentation>`__
* `Visual-language assistant with LLaVA and OpenVINO <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/257-llava-multimodal-chatbot>`__
* `Image editing with InstructPix2Pix <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/231-instruct-pix2pix-image-editing>`__
* `MMS: Scaling Speech Technology to 1000+ languages <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/255-mms-massively-multilingual-speech>`__
* `Image generation with Latent Consistency Model <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/263-latent-consistency-models-image-generation>`__
* `Object segmentations with FastSAM <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/261-fast-segment-anything>`__
* `Automatic speech recognition using Distil-Whisper <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/267-distil-whisper-asr>`__
Known issues
++++++++++++++++++++++++++++++++++++++++++++
| **ID - 118179**
| *Component* - Python API, Plugins
| **ID - 127202**
| *Component* - CPU Plugin
| *Description:*
| When input byte sizes are matching, inference methods accept incorrect inputs
in copy mode (share_inputs=False). Example: [1, 4, 512, 512] is allowed when
[1, 512, 512, 4] is required by the model.
| Deeplabv3 model from TF framework shows lower performance than previous
release. This is because the TopK layer in the model is now correctly
conducting the stable sort as specified by the model, slower than the
previous unstable sort.
| *Workaround:*
| Pass inputs which shape and layout match model ones.
| **ID - 124181**
| *Component* - CPU plugin
| *Description:*
| On CPU platform with L2 cache size less than 256KB, such as i3 series of 8th
Gen Intel CORE platforms, some models may hang during model loading.
| *Workaround:*
| Rebuild the software from OpenVINO master or use the next OpenVINO release.
| **ID - 121959**
| *Component* - CPU plugin
| *Description:*
| During inference using latency hint on selected hybrid CPU platforms
(such as 12th or 13th Gen Intel CORE), there is a sporadic occurrence of
increased latency caused by the operating system scheduling of P-cores or
E-cores during OpenVINO initialization.
| *Workaround:*
| This will be fixed in the next OpenVINO release.
| This release has the correct behavior. If performance is critical,
please use the previous version of OpenVINO, or tune the model.
| **ID - 123101**
| *Component* - GPU plugin
| *Component* - GPU plugin
| *Description:*
| Hung up of GPU plugin on A770 Graphics (dGPU) in case of
large batch size (1750).
| Hung up of GPU plugin on A770 Graphics (dGPU) in case of large
batch size (1750).
| *Workaround:*
| Decrease the batch size, wait for fixed driver released.
| Decrease the batch size, and wait for the fixed driver released.
Included in This Release
+++++++++++++++++++++++++++++++++++++++++++++
@ -320,19 +351,19 @@ three types of operating systems: Windows, Linux, and macOS.
|| Component || License | Location |
+================================+===================================+=================+=================+=======================+=================================================+
|| OpenVINO (Inference Engine) C++ Runtime || Dual licensing: || <install_root>/runtime/* |
|| Unified API to integrate the inference with application logic || Intel® OpenVINO™ Distribution License (Version May 2021) || <install_root>/runtime/include/* |
|| OpenVINO (Inference Engine) Headers || Apache 2.0 || |
|| Unified API to integrate the inference with application logic || Intel® OpenVINO™ Distribution License (Version May 2021) || <install_root>/runtime/include/* |
|| OpenVINO (Inference Engine) Headers || Apache 2.0 || |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| OpenVINO (Inference Engine) Pythion API || Apache 2.0 || <install_root>/python/* |
|| OpenVINO (Inference Engine) Python API || Apache 2.0 || <install_root>/python/* |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| OpenVINO (Inference Engine) Samples || Apache 2.0 || <install_root>/samples/* |
|| Samples that illustrate OpenVINO C++/ Python API usage || || |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
|| [Deprecated] Deployment manager || Apache 2.0 || <install_root>/tools/deployment_manager/* |
|| The Deployment Manager is a Python* command-line tool that || || |
|| creates a deployment package by assembling the model, IR files, || || |
|| your application, and associated dependencies into a runtime || || |
|| package for your target device. || || |
|| [Deprecated] Deployment manager || Apache 2.0 || <install_root>/tools/deployment_manager/* |
|| The Deployment Manager is a Python command-line tool that || || |
|| creates a deployment package by assembling the model, IR files, || || |
|| your application, and associated dependencies into a runtime || || |
|| package for your target device. || || |
+--------------------------------------------------------------------+-----------------------------------------------------------+-------------------------------------------------+
@ -360,7 +391,7 @@ enabled hardware, software or service activation. Learn more at
`http://www.intel.com/ <http://www.intel.com/>`__
or from the OEM or retailer.
No computer system can be absolutely secure.
No computer system can be absolutely secure.
Intel, Atom, Arria, Core, Movidius, Xeon, OpenVINO, and the Intel logo are trademarks
of Intel Corporation in the U.S. and/or other countries.
@ -371,18 +402,18 @@ Other names and brands may be claimed as the property of others.
Copyright © 2023, Intel Corporation. All rights reserved.
For more complete information about compiler optimizations, see our Optimization Notice.
Performance varies by use, configuration and other factors. Learn more at
For more complete information about compiler optimizations, see our Optimization Notice.
Performance varies by use, configuration and other factors. Learn more at
`www.Intel.com/PerformanceIndex <www.Intel.com/PerformanceIndex>`__.
Download
+++++++++++++++++++++++++++++++++++++++++++++
`The OpenVINO product selector tool <https://docs.openvino.ai/install>`__
provides easy access to the right packages that match your desired OS, version,
provides easy access to the right packages that match your desired OS, version,
and distribution options.

View File

@ -2,13 +2,16 @@
System Requirements
===================
Certain hardware requires specific drivers to work properly with OpenVINO.
These drivers, including Linux* kernels, might require updates to your operating system,
which is not part of OpenVINO installation. Refer to your hardware's documentation
for updating instructions.
.. note::
Certain hardware (including but not limited to GPU, GNA, and latest CPUs) requires manual
installation of specific drivers and/or other software components to work correctly and/or
to utilize hardware capabilities at their best. This might require updates to operating
system, including but not limited to Linux kernel, please refer to their documentation
for details. These modifications should be handled by user and are not part of OpenVINO
installation.
CPU
@ -20,26 +23,24 @@ CPU
* Intel Atom® processor with Intel® SSE4.2 support
* Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with Intel® HD Graphics
* 6th - 13th generation Intel® Core™ processors
* Intel®Core™Ultra (codenameMeteor Lake)
* Intel® Xeon® Scalable Processors (code name Skylake)
* 2nd Generation Intel® Xeon® Scalable Processors (code name Cascade Lake)
* 3rd Generation Intel® Xeon® Scalable Processors(code nameCooper Lakeand Ice Lake)
* 4th Generation Intel® Xeon® Scalable Processors(code name Sapphire Rapids)
* ARM* and ARM64 CPUs; Apple M1, M2 and Raspberry Pi
* 6th - 14th generation Intel® Core™ processors
* Intel® Core™ Ultra (codename Meteor Lake)
* 1st - 5th generation Intel® Xeon® Scalable Processors
* ARM and ARM64 CPUs; Apple M1, M2, and Raspberry Pi
.. tab-item:: Supported Operating Systems
* Ubuntu 22.04 long-term support (LTS), 64-bit (Kernel 5.15+)
* Ubuntu 20.04 long-term support (LTS), 64-bit (Kernel 5.15+)
* Ubuntu 18.04 long-term support (LTS) with limitations, 64-bit (Kernel 5.4+)
* Windows* 10
* Windows* 11
* macOS* 10.15 and above, 64-bit
* Windows 10
* Windows 11
* macOS 10.15 and above, 64-bit
* macOS 11 and above, ARM64
* Red Hat Enterprise Linux* 8, 64-bit
* Debian 9 ARM64 and ARM
* CentOS 7 64-bit
* CentOS 7
* Red Hat Enterprise Linux 8, 64-bit
* Ubuntu 18 ARM64
* Debian 9 ARM
GPU
##########
@ -53,7 +54,7 @@ GPU
* Intel® Iris® Pro Graphics
* Intel® Iris® Xe Graphics
* Intel® Iris® Xe Max Graphics
* Intel® Arc ™ GPU Series
* Intel® Arc™ GPU Series
* Intel® Data Center GPU Flex Series
* Intel® Data Center GPU Max Series
@ -63,7 +64,7 @@ GPU
* Ubuntu 20.04 long-term support (LTS), 64-bit
* Windows 10, 64-bit
* Windows 11, 64-bit
* Centos 7
* CentOS 7
* Red Hat Enterprise Linux 8, 64-bit
.. tab-item:: Additional considerations
@ -72,24 +73,24 @@ GPU
Distribution of OpenVINO™ toolkit package.
* A chipset that supports processor graphics is required for Intel® Xeon®
processors. Processor graphics are not included in all processors. See
`Product Specifications <https://ark.intel.com/>`__
for information about your processor.
* Although this release works with Ubuntu 20.04 for discrete graphic cards,
Ubuntu 20.04 is not POR for discrete graphics drivers, so OpenVINO support
is limited.
`Product Specifications <https://ark.intel.com/>`__
for information about your processor.
* While this release of OpenVINO supports Ubuntu 20.04, the driver stack
for Intel discrete graphic cards does not fully support Ubuntu 20.04.
We recommend using Ubuntu 22.04 when executing on discrete graphics.
* The following minimum (i.e., used for old hardware) OpenCL™ driver's versions
were used during OpenVINO internal validation: 22.43 for Ubuntu 22.04, 21.48
for Ubuntu 20.04 and 21.49 for Red Hat Enterprise Linux 8.
for Ubuntu 20.04 and 21.49 for Red Hat Enterprise Linux 8.
NPU and GNA
#############################
Intel® Neural Processing Unit
################################
.. tab-set::
.. tab-item:: Operating Systems for NPU
* Ubuntu 22.04 long-term support (LTS), 64-bit
* Windows 11, 64-bit
* Windows 11, 64-bit (22H2, 23H2)
.. tab-item:: Operating Systems for GNA
@ -100,10 +101,23 @@ NPU and GNA
.. tab-item:: Additional considerations
* These Accelerators require drivers that are not included in the
Intel® Distribution of OpenVINO™ toolkit package.
* These Accelerators require :doc:`drivers <openvino_docs_install_guides_configurations_for_intel_npu>`
that are not included in the Intel® Distribution of OpenVINO™ toolkit package.
* Users can access the NPU plugin through the OpenVINO archives on
the download page.
the :doc:`download page <openvino_docs_install_guides_overview>`.
Intel® Gaussian & Neural Accelerator
##########################################
.. tab-set::
.. tab-item:: Operating Systems for GNA
* Ubuntu 22.04 long-term support (LTS), 64-bit
* Ubuntu 20.04 long-term support (LTS), 64-bit
* Windows 10, 64-bit
* Windows 11, 64-bit
Operating systems and developer environment
@ -111,7 +125,7 @@ Operating systems and developer environment
.. tab-set::
.. tab-item:: Linux
.. tab-item:: Linux OS
* Ubuntu 22.04 with Linux kernel 5.15+
* Ubuntu 20.04 with Linux kernel 5.15+
@ -127,16 +141,13 @@ Operating systems and developer environment
* `GNU Compiler Collection (GCC) <https://www.gnu.org/software/gcc/>`__ 7.5 and above
* `CMake <https://cmake.org/download/>`__ 3.10 or higher
Higher versions of kernel might be required for 10th Gen Intel® Core™ Processors,
11th Gen Intel® Core™ Processors, 11th Gen Intel® Core™ Processors S-Series Processors,
12th Gen Intel® Core™ Processors, 13th Gen Intel® Core™ Processors, Intel® Core™ Ultra
Processors, or 4th Gen Intel® Xeon® Scalable Processors to support CPU, GPU, GNA or
hybrid-cores CPU capabilities.
Higher versions of kernel might be required for 10th Gen Intel® Core™ Processors, 11th Gen
Intel® Core™ Processors, 11th Gen Intel® Core™ Processors S-Series Processors, 12th Gen
Intel® Core™ Processors, 13th Gen Intel® Core™ Processors, 14th Gen
Intel® Core™ Processors, Intel® Core™ Ultra Processors, 4th Gen Intel® Xeon® Scalable Processors
or 5th Gen Intel® Xeon® Scalable Processors to support CPU, GPU, GNA or hybrid-cores CPU capabilities.
.. tab-item:: Windows
* Windows 10
* Windows 11
.. tab-item:: Windows 10 and 11
Build environment components:
@ -158,26 +169,25 @@ Operating systems and developer environment
.. tab-item:: DL frameworks versions:
* TensorFlow* 1.15, 2.12
* MxNet* 1.9.0
* ONNX* 1.14.1
* PaddlePaddle* 2.4
* TensorFlow 1.15, 2.12
* MxNet 1.9.0
* ONNX 1.14.1
* PaddlePaddle 2.4
This package can be installed on other versions of DL Framework
but only the version specified here is fully validated.
This package can be installed on other versions of DL Frameworks
but only the versions specified here are fully validated.
.. note::
OpenVINO Python binaries and binaries on Windows/CentOS7/MACOS(x86) are built
with oneTBB libraries. Other binaries on Ubuntu and Redhat OSes are built with
legacy TBB which is released by OS distribution. OpenVINO can be built with
either oneTBB or legacy TBB by the user on all OS systems listed. System
compatibility and performance are improved on Hybrid CPUs,
OpenVINO Python binaries and binaries on Windows, CentOS 7, and macOS (x86) are built
with oneTBB libraries, and others on Ubuntu and RedHat systems are built with
legacy TBB which is released by OS distribution. OpenVINO can be built from source
with either oneTBB or legacy TBB on all the systems listed here. System
compatibility and performance are improved on Hybrid CPUs
such as 12th Gen Intel Core and above.
Legal Information
+++++++++++++++++++++++++++++++++++++++++++++
@ -202,7 +212,7 @@ enabled hardware, software or service activation. Learn more at
`http://www.intel.com/ <http://www.intel.com/>`__
or from the OEM or retailer.
No computer system can be absolutely secure.
No computer system can be absolutely secure.
Intel, Atom, Arria, Core, Movidius, Xeon, OpenVINO, and the Intel logo are trademarks
of Intel Corporation in the U.S. and/or other countries.
@ -213,9 +223,9 @@ Other names and brands may be claimed as the property of others.
Copyright © 2023, Intel Corporation. All rights reserved.
For more complete information about compiler optimizations, see our Optimization Notice.
Performance varies by use, configuration and other factors. Learn more at
For more complete information about compiler optimizations, see our Optimization Notice.
Performance varies by use, configuration and other factors. Learn more at
`www.Intel.com/PerformanceIndex <www.Intel.com/PerformanceIndex>`__.

View File

@ -5,8 +5,8 @@ OpenVINO Extensibility Mechanism
.. meta::
:description: Explore OpenVINO™ Extensibility API, which allows adding
support for models with custom operations and their further implementation
:description: Explore OpenVINO™ Extensibility API, which allows adding
support for models with custom operations and their further implementation
in applications.
.. toctree::
@ -16,18 +16,18 @@ OpenVINO Extensibility Mechanism
openvino_docs_Extensibility_UG_add_openvino_ops
openvino_docs_Extensibility_UG_Frontend_Extensions
openvino_docs_Extensibility_UG_GPU
.. toctree::
:maxdepth: 1
:hidden:
openvino_docs_transformations
OpenVINO Plugin Developer Guide <openvino_docs_ie_plugin_dg_overview>
The Intel® Distribution of OpenVINO™ toolkit supports neural-network models trained with various frameworks, including
TensorFlow, PyTorch, ONNX, TensorFlow Lite, and PaddlePaddle (OpenVINO support for Apache MXNet, Caffe, and Kaldi is currently
being deprecated and will be removed entirely in the future). The list of supported operations is different for each of the supported frameworks.
TensorFlow, PyTorch, ONNX, TensorFlow Lite, and PaddlePaddle (OpenVINO support for Apache MXNet, Caffe, and Kaldi is currently
being deprecated and will be removed entirely in the future). The list of supported operations is different for each of the supported frameworks.
To see the operations supported by your framework, refer to :doc:`Supported Framework Operations <openvino_resources_supported_operations_frontend>`.
Custom operations, which are not included in the list, are not recognized by OpenVINO out-of-the-box. The need for custom operation may appear in two cases:
@ -80,32 +80,32 @@ Registering Extensions
A custom operation class and a new mapping frontend extension class object should be registered to be usable in OpenVINO runtime.
.. note::
.. note::
This documentation is derived from the `Template extension <https://github.com/openvinotoolkit/openvino/tree/master/src/core/template_extension/new>`__, which demonstrates the details of extension development. It is based on minimalistic ``Identity`` operation that is a placeholder for your real custom operation. Review the complete, fully compilable code to see how it works.
Use the ``:ref:`ov::Core::add_extension <doxid-classov_1_1_core_1a68d0dea1cbcd42a67bea32780e32acea>``` method to load the extensions to the ``:ref:`ov::Core <doxid-classov_1_1_core>``` object. This method allows loading library with extensions or extensions from the code.
Use the ``ov::Core::add_extension`` method to load the extensions to the ``ov::Core`` object. This method allows loading library with extensions or extensions from the code.
Load Extensions to Core
+++++++++++++++++++++++
Extensions can be loaded from a code with the ``:ref:`ov::Core::add_extension <doxid-classov_1_1_core_1a68d0dea1cbcd42a67bea32780e32acea>``` method:
Extensions can be loaded from a code with the ``ov::Core::add_extension`` method:
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_extensions.py
:language: python
:fragment: [add_extension]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_extensions.cpp
:language: cpp
:fragment: [add_extension]
The ``Identity`` is a custom operation class defined in :doc:`Custom Operation Guide <openvino_docs_Extensibility_UG_add_openvino_ops>`. This is sufficient to enable reading OpenVINO IR which uses the ``Identity`` extension operation emitted by Model Optimizer. In order to load original model directly to the runtime, add a mapping extension:
@ -130,7 +130,7 @@ When Python API is used, there is no way to implement a custom OpenVINO operatio
Python can still be used to map and decompose operations when only operations from the standard OpenVINO operation set are used.
.. _create_a_library_with_extensions:
Create a Library with Extensions
++++++++++++++++++++++++++++++++
@ -142,7 +142,7 @@ An extension library should be created in the following cases:
To create an extension library, for example, to load the extensions into Model Optimizer, perform the following:
1. Create an entry point for extension library. OpenVINO provides the ``:ref:`OPENVINO_CREATE_EXTENSIONS() <doxid-core_2include_2openvino_2core_2extension_8hpp_1acdadcfa0eff763d8b4dadb8a9cb6f6e6>``` macro, which allows to define an entry point to a library with OpenVINO Extensions.
1. Create an entry point for extension library. OpenVINO provides the ``OPENVINO_CREATE_EXTENSIONS()`` macro, which allows to define an entry point to a library with OpenVINO Extensions.
This macro should have a vector of all OpenVINO Extensions as an argument.
Based on that, the declaration of an extension class might look like the following:
@ -162,7 +162,7 @@ This CMake script finds OpenVINO, using the ``find_package`` CMake command.
3. Build the extension library, running the commands below:
.. code-block:: sh
$ cd src/core/template_extension/new
$ mkdir build
$ cd build
@ -173,16 +173,16 @@ This CMake script finds OpenVINO, using the ``find_package`` CMake command.
4. After the build, you may use the path to your extension library to load your extensions to OpenVINO Runtime:
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_extensions.py
:language: python
:fragment: [add_extension_lib]
.. tab-item:: C++
:sync: cpp
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_extensions.cpp
:language: cpp
@ -194,5 +194,5 @@ See Also
* :doc:`OpenVINO Transformations <openvino_docs_transformations>`
* :doc:`Using OpenVINO Runtime Samples <openvino_docs_OV_UG_Samples_Overview>`
* :doc:`Hello Shape Infer SSD sample <openvino_inference_engine_samples_hello_reshape_ssd_README>`
* :doc:`Hello Shape Infer SSD sample <openvino_sample_hello_reshape_ssd>`

View File

@ -5,7 +5,7 @@ How to Implement Custom GPU Operations
.. meta::
:description: Learn the details of custom kernel support for the GPU device to
:description: Learn the details of custom kernel support for the GPU device to
enable operations not supported by OpenVINO.
@ -16,20 +16,20 @@ The GPU codepath abstracts many details about OpenCL. You need to provide the ke
There are two options for using the custom operation configuration file:
* Include a section with your kernels into the automatically-loaded ``<lib_path>/cldnn_global_custom_kernels/cldnn_global_custom_kernels.xml`` file.
* Call the ``:ref:`ov::Core::set_property() <doxid-classov_1_1_core_1aa953cb0a1601dbc9a34ef6ba82b8476e>``` method from your application with the ``"CONFIG_FILE"`` key and the configuration file name as a value before loading the network that uses custom operations to the plugin:
* Call the ``ov::Core::set_property()`` method from your application with the ``"CONFIG_FILE"`` key and the configuration file name as a value before loading the network that uses custom operations to the plugin:
.. tab-set::
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/gpu/custom_kernels_api.py
:language: python
:fragment: [part0]
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/gpu/custom_kernels_api.cpp
:language: cpp
:fragment: [part0]
@ -43,10 +43,10 @@ feature a dedicated command-line option ``-c`` to load custom kernels. For examp
$ ./classification_sample -m <path_to_model>/bvlc_alexnet_fp16.xml -i ./validation_set/daily/227x227/apron.bmp -d GPU
-c <absolute_path_to_config>/custom_layer_example.xml
.. _config-file-format:
.. _config-file-format:
Configuration File Format
#########################
#########################
The configuration file is expected to follow the ``.xml`` file structure
with a node of the type ``CustomLayer`` for every custom operation you provide.
@ -344,7 +344,7 @@ Example Kernel
.. _debugging-tips:
.. note::
.. note::
As described in the previous section, all items such as the ``INPUT0_TYPE`` are actually defined as OpenCL (pre-)compiler inputs by OpenVINO for efficiency reasons. See the `Debugging Tips <#debugging-tips>`__ below for information on debugging the results.
Debugging Tips

View File

@ -5,8 +5,8 @@ Custom OpenVINO™ Operations
.. meta::
:description: Explore OpenVINO™ Extension API which enables registering
custom operations to support models with operations
:description: Explore OpenVINO™ Extension API which enables registering
custom operations to support models with operations
not supported by OpenVINO.
OpenVINO™ Extension API allows you to register custom operations to support models with operations which OpenVINO™ does not support out-of-the-box. This capability requires writing code in C++, so if you are using Python to develop your application you need to build a separate shared library implemented in C++ first and load it in Python using ``add_extension`` API. Please refer to :ref:`Create library with extensions <create_library_with_extensions>` for more details on library creation and usage. The remining part of this document describes how to implement an operation class.
@ -14,7 +14,7 @@ OpenVINO™ Extension API allows you to register custom operations to support mo
Operation Class
###############
To add your custom operation, create a new class that extends ``ov::Op``, which is in turn derived from ``:ref:`ov::Node <doxid-classov_1_1_node>```, the base class for all graph operations in OpenVINO™. To add ``ov::Op``, include the next file:
To add your custom operation, create a new class that extends ``ov::Op``, which is in turn derived from ``ov::Node``, the base class for all graph operations in OpenVINO™. To add ``ov::Op``, include the next file:
.. doxygensnippet:: ./src/core/template_extension/new/identity.hpp
:language: cpp
@ -24,9 +24,9 @@ Follow the steps below to add a custom operation:
1. Add the ``OPENVINO_OP`` macro which defines a ``NodeTypeInfo`` object that identifies the type of the operation to the graph users and helps with dynamic type resolution. The type info of an operation currently consists of a string operation identifier and a string for operation version.
2. Implement default constructor and constructors that optionally take the operation inputs and attributes as parameters.
2. Implement default constructor and constructors that optionally take the operation inputs and attributes as parameters.
3. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``:ref:`ov::Node <doxid-classov_1_1_node>```. Set the inferred shape and element type of the output using ``set_output_type``.
3. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``ov::Node``. Set the inferred shape and element type of the output using ``set_output_type``.
4. Override the ``clone_with_new_inputs`` method, which enables graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.
@ -40,9 +40,9 @@ Based on that, declaration of an operation class can look as follows:
Operation Constructors
++++++++++++++++++++++
OpenVINO™ operation contains two constructors:
OpenVINO™ operation contains two constructors:
* Default constructor, which enables you to create an operation without attributes
* Default constructor, which enables you to create an operation without attributes
* Constructor that creates and validates an operation with specified inputs and attributes
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
@ -52,7 +52,7 @@ OpenVINO™ operation contains two constructors:
``validate_and_infer_types()``
++++++++++++++++++++++++++++++
``:ref:`ov::Node::validate_and_infer_types <doxid-classov_1_1_node_1ac5224b5be848ec670d2078d9816d12e7>``` method validates operation attributes and calculates output shapes using attributes of the operation.
``ov::Node::validate_and_infer_types`` method validates operation attributes and calculates output shapes using attributes of the operation.
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
:language: cpp
@ -61,7 +61,7 @@ OpenVINO™ operation contains two constructors:
``clone_with_new_inputs()``
+++++++++++++++++++++++++++
``:ref:`ov::Node::clone_with_new_inputs <doxid-classov_1_1_node_1a04cb103fa069c3b7944ab7c44d94f5ff>``` method creates a copy of the operation with new inputs.
``ov::Node::clone_with_new_inputs`` method creates a copy of the operation with new inputs.
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
:language: cpp
@ -70,7 +70,7 @@ OpenVINO™ operation contains two constructors:
``visit_attributes()``
++++++++++++++++++++++
``:ref:`ov::Node::visit_attributes <doxid-classov_1_1_node_1a9743b56d352970486d17dae2416d958e>``` method enables you to visit all operation attributes.
``ov::Node::visit_attributes`` method enables you to visit all operation attributes.
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
:language: cpp
@ -79,7 +79,7 @@ OpenVINO™ operation contains two constructors:
``evaluate() and has_evaluate()``
+++++++++++++++++++++++++++++++++
``:ref:`ov::Node::evaluate <doxid-classov_1_1_node_1acfb82acc8349d7138aeaa05217c7014e>``` method enables you to apply constant folding to an operation.
``ov::Node::evaluate`` method enables you to apply constant folding to an operation.
.. doxygensnippet:: ./src/core/template_extension/new/identity.cpp
:language: cpp

View File

@ -96,6 +96,6 @@ Detailed Guides
API References
##############
* `OpenVINO Plugin API <https://docs.openvino.ai/2023.2/groupov_dev_api.html>`__
* `OpenVINO Transformation API <https://docs.openvino.ai/2023.2/groupie_transformation_api.html>`__
* `OpenVINO Plugin API <https://docs.openvino.ai/2023.3/api/c_cpp_api/group__ov__dev__api.html>`__
* `OpenVINO Transformation API <https://docs.openvino.ai/2023.3/api/c_cpp_api/group__ie__transformation__api.html>`__

View File

@ -329,7 +329,7 @@ After that you should quantize model by the :doc:`Model Quantizer <omz_tools_dow
Inference
+++++++++
The simplest way to infer the model and collect performance counters is :doc:`Benchmark Application <openvino_inference_engine_samples_benchmark_app_README>`.
The simplest way to infer the model and collect performance counters is :doc:`Benchmark Application <openvino_sample_benchmark_tool>`.
.. code-block:: sh

View File

@ -5,7 +5,7 @@ Step 2. Markup Transformations
.. meta::
:description: Learn about markup transformations, which are used to create
:description: Learn about markup transformations, which are used to create
attributes for input and output ports and operations during runtime.
.. toctree::
@ -21,7 +21,7 @@ Step 2. Markup Transformations
PropagatePrecisions <openvino_docs_OV_UG_lpt_PropagatePrecisions>
AlignQuantizationIntervals <openvino_docs_OV_UG_lpt_AlignQuantizationIntervals>
AlignQuantizationParameters <openvino_docs_OV_UG_lpt_AlignQuantizationParameters>
CreateAttribute <openvino_docs_OV_UG_lpt_CreateAttribute>
CreatePrecisionsDependentAttribute <openvino_docs_OV_UG_lpt_CreatePrecisionsDependentAttribute>
PropagateThroughPrecisionPreserved <openvino_docs_OV_UG_lpt_PropagateThroughPrecisionPreserved>
@ -31,13 +31,13 @@ Step 2. Markup Transformations
This step defines the optimal ``FakeQuantize`` decomposition precisions for the best inference performance via operations markup with runtime attribute instances. Attributes are created for input and output ports and operations. Transformations do not change the operation output port precisions. A model markup low precision logic is decomposed and implemented into the following common markup transformations. The order of transformations is important:
1. :doc:`MarkupBias <openvino_docs_OV_UG_lpt_MarkupBias>`
2. :doc:`MarkupCanBeQuantized <openvino_docs_OV_UG_lpt_MarkupCanBeQuantized>`
3. :doc:`MarkupPrecisions <openvino_docs_OV_UG_lpt_MarkupPrecisions>`
4. :doc:`MarkupPerTensorQuantization <openvino_docs_OV_UG_lpt_MarkupPerTensorQuantization>`
5. :doc:`MarkupAvgPoolPrecisionPreserved <openvino_docs_OV_UG_lpt_MarkupAvgPoolPrecisionPreserved>`
6. :doc:`PropagatePrecisions <openvino_docs_OV_UG_lpt_PropagatePrecisions>`
7. :doc:`AlignQuantizationIntervals <openvino_docs_OV_UG_lpt_AlignQuantizationIntervals>`
8. :doc:`AlignQuantizationParameters <openvino_docs_OV_UG_lpt_AlignQuantizationParameters>`
2. :doc:`MarkupCanBeQuantized <openvino_docs_OV_UG_lpt_MarkupCanBeQuantized>`
3. :doc:`MarkupPrecisions <openvino_docs_OV_UG_lpt_MarkupPrecisions>`
4. :doc:`MarkupPerTensorQuantization <openvino_docs_OV_UG_lpt_MarkupPerTensorQuantization>`
5. :doc:`MarkupAvgPoolPrecisionPreserved <openvino_docs_OV_UG_lpt_MarkupAvgPoolPrecisionPreserved>`
6. :doc:`PropagatePrecisions <openvino_docs_OV_UG_lpt_PropagatePrecisions>`
7. :doc:`AlignQuantizationIntervals <openvino_docs_OV_UG_lpt_AlignQuantizationIntervals>`
8. :doc:`AlignQuantizationParameters <openvino_docs_OV_UG_lpt_AlignQuantizationParameters>`
.. list-table::
:header-rows: 1
@ -47,16 +47,16 @@ This step defines the optimal ``FakeQuantize`` decomposition precisions for the
- Use attributes
* - MarkupBias
- Bias
-
-
* - MarkupCanBeQuantized
- Precisions
-
-
* - MarkupPrecisions
- Precisions,PrecisionPreserved
-
-
* - MarkupPerTensorQuantization
- PerTensorQuantization
-
-
* - MarkupAvgPoolPrecisionPreserved
- AvgPoolPrecisionPreserved
- Precisions, PrecisionPreserved
@ -70,16 +70,16 @@ This step defines the optimal ``FakeQuantize`` decomposition precisions for the
- QuantizationAlignment
- PrecisionPreserved, PerTensorQuantization
.. note::
.. note::
The same type of attribute instances can be created in different transformations. This approach is the result of the transformation single-responsibility principle. For example, ``Precision`` attribute instances are created in ``MarkupCanBeQuantized`` and ``MarkupPrecisions`` transformations, but the reasons for their creation are different
Common markup transformations can be decomposed into simpler utility markup transformations. The order of Markup utility transformations is not important:
* :doc:`CreateAttribute <openvino_docs_OV_UG_lpt_CreateAttribute>`
* :doc:`CreatePrecisionsDependentAttribute <openvino_docs_OV_UG_lpt_CreatePrecisionsDependentAttribute>`
* :doc:`PropagateThroughPrecisionPreserved <openvino_docs_OV_UG_lpt_PropagateThroughPrecisionPreserved>`
* :doc:`PropagateToInput <openvino_docs_OV_UG_lpt_PropagateToInput>`
* :doc:`UpdateSharedPrecisionPreserved <openvino_docs_OV_UG_lpt_UpdateSharedPrecisionPreserved>`
* :doc:`CreateAttribute <openvino_docs_OV_UG_lpt_CreateAttribute>`
* :doc:`CreatePrecisionsDependentAttribute <openvino_docs_OV_UG_lpt_CreatePrecisionsDependentAttribute>`
* :doc:`PropagateThroughPrecisionPreserved <openvino_docs_OV_UG_lpt_PropagateThroughPrecisionPreserved>`
* :doc:`PropagateToInput <openvino_docs_OV_UG_lpt_PropagateToInput>`
* :doc:`UpdateSharedPrecisionPreserved <openvino_docs_OV_UG_lpt_UpdateSharedPrecisionPreserved>`
Let's explore all transformations and their relations in detail, using one and the same model:
@ -90,12 +90,12 @@ The original model key features:
* The first ``concat1`` concatenation operation has not quantized ``convolution1`` consumer.
* The second ``concat2`` concatenation operation has quantized ``convolution2`` consumer with requirements:
* The second ``concat2`` concatenation operation has quantized ``convolution2`` consumer with requirements:
* support ``unsigned int8`` on activations,
* per-tensor quantization.
* Between the ``concat2`` concatenation operation and ``Convolution`` there is an ``AvgPool`` operation, which mathematically should return an ``f32`` tensor. But the ``MarkupAvgPoolPrecisionPreserved`` transformation is active. This allows the low precision transformation, that goes after the ``AvgPool``, to propagate low precision tensor to the next consumer.
* Between the ``concat2`` concatenation operation and ``Convolution`` there is an ``AvgPool`` operation, which mathematically should return an ``f32`` tensor. But the ``MarkupAvgPoolPrecisionPreserved`` transformation is active. This allows the low precision transformation, that goes after the ``AvgPool``, to propagate low precision tensor to the next consumer.
Transformations are run with the following parameters:
@ -127,8 +127,8 @@ Model display features (here and below):
The transformation is required and includes two tasks:
1. Mark operation input ports (create ``Precision`` attribute instance) by provided restrictions: input port index and required precisions. Restrictions are provided as input argument in ``:ref:`ov::pass::low_precision::LowPrecision <doxid-classov_1_1pass_1_1low__precision_1_1_low_precision>``` constructor.
2. Mark precision preserved operations.
1. Mark operation input ports (create ``Precision`` attribute instance) by provided restrictions: input port index and required precisions. Restrictions are provided as input argument in ``ov::pass::low_precision::LowPrecision`` constructor.
2. Mark precision preserved operations.
No attributes are required before the transformation. Changes in the example model after ``MarkupPrecisions`` transformation:
@ -191,7 +191,7 @@ Result model:
.. image:: _static/images/step2_markup5.svg
:alt: PropagatePrecisions
.. note::
.. note::
``AlignQuantizationIntervals`` and ``AlignQuantizationParameters`` transformations are required if the model has quantized concatenation operations.
6. AlignQuantizationIntervals

View File

@ -12,11 +12,11 @@ Plugin API Reference
:maxdepth: 1
:hidden:
../groupov_dev_api
../groupie_transformation_api
../api/c_cpp_api/group__ov__dev__api
../api/c_cpp_api/group__ie__transformation__api
The guides below provides extra API references needed for OpenVINO plugin development:
* `OpenVINO Plugin API <https://docs.openvino.ai/2023.2/groupov_dev_api.html>`__
* `OpenVINO Transformation API <https://docs.openvino.ai/2023.2/groupie_transformation_api.html>`__
* `OpenVINO Plugin API <https://docs.openvino.ai/2023.3/api/c_cpp_api/group__ov__dev__api.html>`__
* `OpenVINO Transformation API <https://docs.openvino.ai/2023.3/api/c_cpp_api/group__ie__transformation__api.html>`__

View File

@ -5,7 +5,7 @@ Overview of Transformations API
.. meta::
:description: Learn how to apply additional model optimizations or transform
:description: Learn how to apply additional model optimizations or transform
unsupported subgraphs and operations, using OpenVINO™ Transformations API.
@ -17,19 +17,19 @@ Overview of Transformations API
openvino_docs_Extensibility_UG_matcher_pass
openvino_docs_Extensibility_UG_graph_rewrite_pass
OpenVINO Transformation mechanism allows to develop transformation passes to modify ``:ref:`ov::Model <doxid-classov_1_1_model>```. You can use this mechanism to apply additional optimizations to the original Model or transform unsupported subgraphs and operations to new operations which are supported by the plugin.
OpenVINO Transformation mechanism allows to develop transformation passes to modify ``ov::Model``. You can use this mechanism to apply additional optimizations to the original Model or transform unsupported subgraphs and operations to new operations which are supported by the plugin.
This guide contains all necessary information that you need to start implementing OpenVINO™ transformations.
Working with Model
##################
Before the moving to transformation part it is needed to say several words about functions which allow to modify ``:ref:`ov::Model <doxid-classov_1_1_model>```.
This chapter extends the :doc:`model representation guide <openvino_docs_OV_UG_Model_Representation>` and shows an API that allows us to manipulate with ``:ref:`ov::Model <doxid-classov_1_1_model>```.
Before the moving to transformation part it is needed to say several words about functions which allow to modify ``ov::Model``.
This chapter extends the :doc:`model representation guide <openvino_docs_OV_UG_Model_Representation>` and shows an API that allows us to manipulate with ``ov::Model``.
Working with node input and output ports
++++++++++++++++++++++++++++++++++++++++
First of all let's talk about ``:ref:`ov::Node <doxid-classov_1_1_node>``` input/output ports. Each OpenVINO™ operation has input and output ports except cases when operation has ``Parameter`` or ``Constant`` type.
First of all let's talk about ``ov::Node`` input/output ports. Each OpenVINO™ operation has input and output ports except cases when operation has ``Parameter`` or ``Constant`` type.
Every port belongs to its node, so using a port we can access parent node, get shape and type for particular input/output, get all consumers in case of output port, and get producer node in case of input port.
With output port we can set inputs for newly created operations.
@ -49,13 +49,13 @@ Let's start with OpenVINO™ helper functions. The most popular function is ``ov
We will review real replacement case where Negative operation is replaced with Multiply.
.. image:: ./_static/images/ngraph_replace_node.png
.. image:: ./_static/images/ngraph_replace_node.png
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
:language: cpp
:fragment: [ov:replace_node]
``:ref:`ov::replace_node <doxid-namespaceov_1a75d84ee654edb73fe4fb18936a5dca6d>``` has a constraint that number of output ports for both of ops must be the same; otherwise, it raises an exception.
``ov::replace_node`` has a constraint that number of output ports for both of ops must be the same; otherwise, it raises an exception.
The alternative way to do the same replacement is the following:
@ -71,7 +71,7 @@ Another transformation example is insertion.
:language: cpp
:fragment: [ov:insert_node]
The alternative way to the insert operation is to make a node copy and use ``:ref:`ov::replace_node() <doxid-namespaceov_1a75d84ee654edb73fe4fb18936a5dca6d>```:
The alternative way to the insert operation is to make a node copy and use ``ov::replace_node()``:
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
:language: cpp
@ -88,16 +88,16 @@ To eliminate operation, OpenVINO™ has special method that considers all limita
:language: cpp
:fragment: [ov:eliminate_node]
``:ref:`ov::replace_output_update_name() <doxid-namespaceov_1a75ba2120e573883bd96bb19c887c6a1d>``` in case of successful replacement it automatically preserves friendly name and runtime info.
``ov::replace_output_update_name()`` in case of successful replacement it automatically preserves friendly name and runtime info.
.. _transformations_types:
Transformations types
Transformations types
#####################
OpenVINO™ Runtime has three main transformation types:
* :doc:`Model pass <openvino_docs_Extensibility_UG_model_pass>` - straightforward way to work with ``:ref:`ov::Model <doxid-classov_1_1_model>``` directly
* :doc:`Model pass <openvino_docs_Extensibility_UG_model_pass>` - straightforward way to work with ``ov::Model`` directly
* :doc:`Matcher pass <openvino_docs_Extensibility_UG_matcher_pass>` - pattern-based transformation approach
* :doc:`Graph rewrite pass <openvino_docs_Extensibility_UG_graph_rewrite_pass>` - container for matcher passes needed for efficient execution
@ -108,12 +108,12 @@ Transformation conditional compilation
Transformation library has two internal macros to support conditional compilation feature.
* ``:ref:`MATCHER_SCOPE(region) <doxid-conditional__compilation_2include_2openvino_2cc_2pass_2itt_8hpp_1a3d1377542bcf3e305c33a1b683cc77df>``` - allows to disable the MatcherPass if matcher isn't used. The region name should be unique. This macro creates a local variable ``matcher_name`` which you should use as a matcher name.
* ``:ref:`RUN_ON_MODEL_SCOPE(region) <doxid-conditional__compilation_2include_2openvino_2cc_2pass_2itt_8hpp_1ab308561b849d47b9c820506ec73c4a30>``` - allows to disable run_on_model pass if it isn't used. The region name should be unique.
* ``MATCHER_SCOPE(region)`` - allows to disable the MatcherPass if matcher isn't used. The region name should be unique. This macro creates a local variable ``matcher_name`` which you should use as a matcher name.
* ``RUN_ON_MODEL_SCOPE(region)`` - allows to disable run_on_model pass if it isn't used. The region name should be unique.
.. _transformation_writing_essentials:
Transformation writing essentials
Transformation writing essentials
#################################
When developing a transformation, you need to follow these transformation rules:
@ -121,7 +121,7 @@ When developing a transformation, you need to follow these transformation rules:
1. Friendly Names
+++++++++++++++++
Each ``:ref:`ov::Node <doxid-classov_1_1_node>``` has an unique name and a friendly name. In transformations we care only about friendly name because it represents the name from the model.
Each ``ov::Node`` has an unique name and a friendly name. In transformations we care only about friendly name because it represents the name from the model.
To avoid losing friendly name when replacing node with other node or subgraph, set the original friendly name to the latest node in replacing subgraph. See the example below.
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
@ -133,8 +133,8 @@ In more advanced cases, when replaced operation has several outputs and we add a
2. Runtime Info
+++++++++++++++
Runtime info is a map ``std::map<std::string, :ref:`ov::Any <doxid-classov_1_1_any>`>`` located inside ``:ref:`ov::Node <doxid-classov_1_1_node>``` class. It represents additional attributes in ``:ref:`ov::Node <doxid-classov_1_1_node>```.
These attributes can be set by users or by plugins and when executing transformation that changes ``:ref:`ov::Model <doxid-classov_1_1_model>``` we need to preserve these attributes as they will not be automatically propagated.
Runtime info is a map ``std::map<std::string, ov::Any>`` located inside ``ov::Node`` class. It represents additional attributes in ``ov::Node>``.
These attributes can be set by users or by plugins and when executing transformation that changes ``ov::Model`` we need to preserve these attributes as they will not be automatically propagated.
In most cases, transformations have the following types: 1:1 (replace node with another node), 1:N (replace node with a sub-graph), N:1 (fuse sub-graph into a single node), N:M (any other transformation).
Currently, there is no mechanism that automatically detects transformation types, so we need to propagate this runtime information manually. See the examples below.
@ -143,21 +143,21 @@ Currently, there is no mechanism that automatically detects transformation types
:language: cpp
:fragment: [ov:copy_runtime_info]
When transformation has multiple fusions or decompositions, ``:ref:`ov::copy_runtime_info <doxid-namespaceov_1a3bb5969a95703b4b4fd77f6f58837207>``` must be called multiple times for each case.
When transformation has multiple fusions or decompositions, ``ov::copy_runtime_info`` must be called multiple times for each case.
.. note:: ``copy_runtime_info`` removes ``rt_info`` from destination nodes. If you want to keep it, you need to specify them in source nodes like this: ``copy_runtime_info({a, b, c}, {a, b})``
3. Constant Folding
+++++++++++++++++++
If your transformation inserts constant sub-graphs that need to be folded, do not forget to use ``:ref:`ov::pass::ConstantFolding() <doxid-classov_1_1pass_1_1_constant_folding>``` after your transformation or call constant folding directly for operation.
If your transformation inserts constant sub-graphs that need to be folded, do not forget to use ``ov::pass::ConstantFolding()`` after your transformation or call constant folding directly for operation.
The example below shows how constant subgraph can be constructed.
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
:language: cpp
:fragment: [ov:constant_subgraph]
Manual constant folding is more preferable than ``:ref:`ov::pass::ConstantFolding() <doxid-classov_1_1pass_1_1_constant_folding>``` because it is much faster.
Manual constant folding is more preferable than ``ov::pass::ConstantFolding()`` because it is much faster.
Below you can find an example of manual constant folding:
@ -167,28 +167,28 @@ Below you can find an example of manual constant folding:
.. _common_mistakes:
Common mistakes in transformations
Common mistakes in transformations
##################################
In transformation development process:
* Do not use deprecated OpenVINO™ API. Deprecated methods has the ``OPENVINO_DEPRECATED`` macros in its definition.
* Do not pass ``shared_ptr<Node>`` as an input for other node if type of node is unknown or it has multiple outputs. Use explicit output port.
* If you replace node with another node that produces different shape, remember that new shape will not be propagated until the first ``validate_nodes_and_infer_types`` call for ``:ref:`ov::Model <doxid-classov_1_1_model>```. If you are using ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>```, it will automatically call this method after each transformation execution.
* Do not forget to call the ``:ref:`ov::pass::ConstantFolding <doxid-classov_1_1pass_1_1_constant_folding>``` pass if your transformation creates constant subgraphs.
* If you replace node with another node that produces different shape, remember that new shape will not be propagated until the first ``validate_nodes_and_infer_types`` call for ``ov::Model``. If you are using ``ov::pass::Manager``, it will automatically call this method after each transformation execution.
* Do not forget to call the ``ov::pass::ConstantFolding`` pass if your transformation creates constant subgraphs.
* Use latest OpSet if you are not developing downgrade transformation pass.
* When developing a callback for ``:ref:`ov::pass::MatcherPass <doxid-classov_1_1pass_1_1_matcher_pass>```, do not change nodes that come after the root node in topological order.
* When developing a callback for ``ov::pass::MatcherPass``, do not change nodes that come after the root node in topological order.
.. _using_pass_manager:
Using pass manager
##################
``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` is a container class that can store the list of transformations and execute them. The main idea of this class is to have high-level representation for grouped list of transformations.
``ov::pass::Manager`` is a container class that can store the list of transformations and execute them. The main idea of this class is to have high-level representation for grouped list of transformations.
It can register and apply any `transformation pass <#transformations_types>`__ on model.
In addition, ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` has extended debug capabilities (find more information in the `how to debug transformations <#how_to_debug_transformations>`__ section).
In addition, ``ov::pass::Manager`` has extended debug capabilities (find more information in the `how to debug transformations <#how_to_debug_transformations>`__ section).
The example below shows basic usage of ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>```
The example below shows basic usage of ``ov::pass::Manager``
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
:language: cpp
@ -199,16 +199,16 @@ Another example shows how multiple matcher passes can be united into single Grap
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
:language: cpp
:fragment: [matcher_pass:manager2]
.. _how_to_debug_transformations:
How to debug transformations
.. _how_to_debug_transformations:
How to debug transformations
############################
If you are using ``ngraph::pass::Manager`` to run sequence of transformations, you can get additional debug capabilities by using the following environment variables:
.. code-block:: cpp
OV_PROFILE_PASS_ENABLE=1 - enables performance measurement for each transformation and prints execution status
OV_ENABLE_VISUALIZE_TRACING=1 - enables visualization after each transformation. By default, it saves dot and svg files.
@ -218,6 +218,6 @@ If you are using ``ngraph::pass::Manager`` to run sequence of transformations, y
See Also
########
* :doc:`OpenVINO™ Model Representation <openvino_docs_OV_UG_Model_Representation>`
* :doc:`OpenVINO™ Model Representation <openvino_docs_OV_UG_Model_Representation>`
* :doc:`OpenVINO™ Extensions <openvino_docs_Extensibility_UG_Intro>`

View File

@ -5,11 +5,11 @@ OpenVINO Graph Rewrite Pass
.. meta::
:description: Get to know how Graph Rewrite handles running multiple matcher passes on
:description: Get to know how Graph Rewrite handles running multiple matcher passes on
ov::Model in a single graph traversal.
``:ref:`ov::pass::GraphRewrite <doxid-classov_1_1pass_1_1_graph_rewrite>``` serves for running multiple matcher passes on ``:ref:`ov::Model <doxid-classov_1_1_model>``` in a single graph traversal.
``ov::pass::GraphRewrite`` serves for running multiple matcher passes on ``ov::Model`` in a single graph traversal.
Example:
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
@ -18,13 +18,13 @@ Example:
In addition, GraphRewrite handles nodes that were registered by MatcherPasses during their execution. This nodes will be added to the beginning of the sequence with nodes for pattern matching.
.. note::
.. note::
When using ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` temporary GraphRewrite is used to execute single MatcherPass.
When using ``ov::pass::Manager`` temporary GraphRewrite is used to execute single MatcherPass.
GraphRewrite has two algorithms for MatcherPasses execution. First algorithm is straightforward. It applies each MatcherPass in registration order to current node.
.. image:: ./_static/images/graph_rewrite_execution.png
.. image:: ./_static/images/graph_rewrite_execution.png
But it is not really efficient when you have a lot of registered passes. So first of all GraphRewrite checks that all MatcherPass patterns has type-based root node (it means that type of this node is not hidden into predicate).
And then creates map from registered MatcherPasses. That helps to avoid additional cost of applying each MatcherPass for each node.

View File

@ -5,11 +5,11 @@ OpenVINO Matcher Pass
.. meta::
:description: Learn how to create a pattern, implement a callback, register
the pattern and Matcher to execute MatcherPass transformation
:description: Learn how to create a pattern, implement a callback, register
the pattern and Matcher to execute MatcherPass transformation
on a model.
``:ref:`ov::pass::MatcherPass <doxid-classov_1_1pass_1_1_matcher_pass>``` is used for pattern-based transformations.
``ov::pass::MatcherPass`` is used for pattern-based transformations.
Template for MatcherPass transformation class
@ -22,7 +22,7 @@ Template for MatcherPass transformation class
:fragment: [graph_rewrite:template_transformation_cpp]
To use ``:ref:`ov::pass::MatcherPass <doxid-classov_1_1pass_1_1_matcher_pass>```, you need to complete these steps:
To use ``ov::pass::MatcherPass``, you need to complete these steps:
1. Create a pattern
2. Implement a callback
@ -34,10 +34,10 @@ So let's go through each of these steps.
Create a pattern
################
Pattern is a single root ``:ref:`ov::Model <doxid-classov_1_1_model>```. But the only difference is that you do not need to create a model object, you just need to create and connect opset or special pattern operations.
Pattern is a single root ``ov::Model``. But the only difference is that you do not need to create a model object, you just need to create and connect opset or special pattern operations.
Then you need to take the last created operation and put it as a root of the pattern. This root node will be used as a root node in pattern matching.
.. note::
.. note::
Any nodes in a pattern that have no consumers and are not registered as root will not be used in pattern matching.
.. doxygensnippet:: docs/snippets/ov_model_snippets.cpp
@ -60,13 +60,13 @@ Callback is an action applied to every pattern entrance. In general, callback is
The example above shows the callback structure and how Matcher can be used for accessing nodes detected by pattern.
Callback return value is ``true`` if root node was replaced and another pattern cannot be applied to the same root node; otherwise, it is ``false``.
.. note::
.. note::
It is not recommended to manipulate with nodes that are under root node. This may affect GraphRewrite execution as it is expected that all nodes that come after root node in topological order are valid and can be used in pattern matching.
MatcherPass also provides functionality that allows reporting of the newly created nodes that can be used in additional pattern matching.
If MatcherPass was registered in ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` or ``:ref:`ov::pass::GraphRewrite <doxid-classov_1_1pass_1_1_graph_rewrite>```, these registered nodes will be added for additional pattern matching.
That means that matcher passes registered in ``:ref:`ov::pass::GraphRewrite <doxid-classov_1_1pass_1_1_graph_rewrite>``` will be applied to these nodes.
If MatcherPass was registered in ``ov::pass::Manager`` or ``ov::pass::GraphRewrite``, these registered nodes will be added for additional pattern matching.
That means that matcher passes registered in ``ov::pass::GraphRewrite`` will be applied to these nodes.
The example below shows how single MatcherPass can fuse sequence of operations using the ``register_new_node`` method.
@ -74,7 +74,7 @@ The example below shows how single MatcherPass can fuse sequence of operations u
:language: cpp
:fragment: [matcher_pass:relu_fusion]
.. note::
.. note::
If you register multiple nodes, please add them in topological order. We do not topologically sort these nodes as it is a time-consuming operation.
Register pattern and Matcher
@ -82,12 +82,12 @@ Register pattern and Matcher
The last step is to register Matcher and callback inside the MatcherPass pass. To do this, call the ``register_matcher`` method.
.. note::
.. note::
Only one matcher can be registered for a single MatcherPass class.
.. code-block:: cpp
// Register matcher and callback
register_matcher(m, callback);
@ -103,13 +103,13 @@ MatcherPass has multiple ways to be executed:
:language: cpp
:fragment: [matcher_pass:run_on_node]
* Run on ``:ref:`ov::Model <doxid-classov_1_1_model>``` using GraphRewrite - this approach gives ability to run MatcherPass on whole ``:ref:`ov::Model <doxid-classov_1_1_model>```. Moreover, multiple MatcherPass transformation can be registered in a single GraphRewite to be executed in a single graph traversal.
* Run on ``ov::Model`` using GraphRewrite - this approach gives ability to run MatcherPass on whole ``ov::Model``. Moreover, multiple MatcherPass transformation can be registered in a single GraphRewite to be executed in a single graph traversal.
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
:language: cpp
:fragment: [matcher_pass:graph_rewrite]
* Run on ``:ref:`ov::Model <doxid-classov_1_1_model>``` using ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>``` - this approach helps you to register MatcherPass for execution on ``:ref:`ov::Model <doxid-classov_1_1_model>``` as another transformation types.
* Run on ``ov::Model`` using ``ov::pass::Manager`` - this approach helps you to register MatcherPass for execution on ``ov::Model`` as another transformation types.
.. doxygensnippet:: docs/snippets/template_pattern_transformation.cpp
:language: cpp
@ -125,8 +125,8 @@ And for these cases OpenVINO™ provides additional helpers to construct pattern
There are two main helpers:
1. ``:ref:`ov::pass::pattern::any_input <doxid-namespaceov_1_1pass_1_1pattern_1a8ed84c3eed4610f117ee10d86d500e02>``` - helps to express inputs if their types are undefined.
2. ``:ref:`ov::pass::pattern::wrap_type <doxid-namespaceov_1_1pass_1_1pattern_1adfcd6031c95d7bace5f084e2aa105af8>`<T>`` - helps to express nodes of pattern without specifying node attributes.
1. ``ov::pass::pattern::any_input`` - helps to express inputs if their types are undefined.
2. ``ov::pass::pattern::wrap_type <T>`` - helps to express nodes of pattern without specifying node attributes.
Let's go through the example to have better understanding of how it works:

View File

@ -5,25 +5,25 @@ OpenVINO Model Pass
.. meta::
:description: Learn how to use Model Pass transformation class to take entire
:description: Learn how to use Model Pass transformation class to take entire
ov::Model as input and process it.
``:ref:`ov::pass::ModelPass <doxid-classov_1_1pass_1_1_model_pass>``` is used for transformations that take entire ``:ref:`ov::Model <doxid-classov_1_1_model>``` as an input and process it.
``ov::pass::ModelPass`` is used for transformations that take entire ``ov::Model`` as an input and process it.
Template for ModelPass transformation class
.. doxygensnippet:: docs/snippets/template_model_transformation.hpp
:language: cpp
.. doxygensnippet:: docs/snippets/template_model_transformation.hpp
:language: cpp
:fragment: [model_pass:template_transformation_hpp]
.. doxygensnippet:: docs/snippets/template_model_transformation.cpp
:language: cpp
:fragment: [model_pass:template_transformation_cpp]
Using ``:ref:`ov::pass::ModelPass <doxid-classov_1_1pass_1_1_model_pass>```, you need to override the ``run_on_model`` method where you will write the transformation code.
Using ``ov::pass::ModelPass``, you need to override the ``run_on_model`` method where you will write the transformation code.
Return value is ``true`` if the original model has changed during transformation (new operation was added, or operations replacement was made, or node attributes were changed); otherwise, it is ``false``.
Also ``:ref:`ov::pass::ModelPass <doxid-classov_1_1pass_1_1_model_pass>``` based transformations can be executed via ``:ref:`ov::pass::Manager <doxid-classov_1_1pass_1_1_manager>```.
Also ``ov::pass::ModelPass`` based transformations can be executed via ``ov::pass::Manager``.
See Also
########

View File

@ -19,6 +19,7 @@ Operation Specifications
Asin-1 <openvino_docs_ops_arithmetic_Asin_1>
Asinh-3 <openvino_docs_ops_arithmetic_Asinh_3>
Assign-3 <openvino_docs_ops_infrastructure_Assign_3>
Assign-6 <openvino_docs_ops_infrastructure_Assign_6>
Atan-1 <openvino_docs_ops_arithmetic_Atan_1>
Atanh-3 <openvino_docs_ops_arithmetic_Atanh_3>
AvgPool-1 <openvino_docs_ops_pooling_AvgPool_1>
@ -162,6 +163,7 @@ Operation Specifications
Range-4 <openvino_docs_ops_generation_Range_4>
RDFT-9<openvino_docs_ops_signals_RDFT_9>
ReadValue-3 <openvino_docs_ops_infrastructure_ReadValue_3>
ReadValue-6 <openvino_docs_ops_infrastructure_ReadValue_6>
ReLU-1 <openvino_docs_ops_activation_ReLU_1>
ReduceL1-4 <openvino_docs_ops_reduction_ReduceL1_4>
ReduceL2-4 <openvino_docs_ops_reduction_ReduceL2_4>

View File

@ -5,7 +5,7 @@ Swish
.. meta::
:description: Learn about Swish-4 - an element-wise, activation operation, which
:description: Learn about Swish-4 - an element-wise, activation operation, which
can be performed on a single tensor in OpenVINO.
**Versioned name**: *Swish-4*
@ -55,7 +55,7 @@ Example: Second input ``beta`` provided
<dim>256</dim>
<dim>56</dim>
</port>
<port id="1"> < !-- beta value: 2.0 -->
<port id="1"> <!-- beta value: 2.0 -->
</port>
</input>
<output>

View File

@ -5,7 +5,7 @@ CumSum
.. meta::
:description: Learn about CumSum-3 - an element-wise, arithmetic operation, which
:description: Learn about CumSum-3 - an element-wise, arithmetic operation, which
can be performed on a single tensor in OpenVINO.
**Versioned name**: *CumSum-3*
@ -24,7 +24,7 @@ To perform the summation in the opposite direction of the axis, set reverse attr
* **Description**: If the attribute is set to ``true``, then exclusive sums are returned, the ``j-th`` element is not included in the ``j-th`` sum. Otherwise, the inclusive sum of the first ``j`` elements for the ``j-th`` element is calculated.
* **Range of values**:
* ``false`` - include the top element
* ``true`` - do not include the top element
* **Type**: ``boolean``
@ -35,7 +35,7 @@ To perform the summation in the opposite direction of the axis, set reverse attr
* **Description**: If set to ``true`` will perform the sums in reverse direction.
* **Range of values**:
* ``false`` - do not perform sums in reverse direction
* ``true`` - perform sums in reverse direction
* **Type**: ``boolean``
@ -63,16 +63,16 @@ To perform the summation in the opposite direction of the axis, set reverse attr
.. code-block:: xml
:force:
<layer ... type="CumSum" exclusive="0" reverse="0">
<input>
<port id="0"> < !-- input value is: [1., 2., 3., 4., 5.] -->
<port id="0"> <!-- input value is: [1., 2., 3., 4., 5.] -->
<dim>5</dim>
</port>
<port id="1"/> < !-- axis value is: 0 -->
<port id="1"/> <!-- axis value is: 0 -->
</input>
<output>
<port id="2"> < !-- output value is: [1., 3., 6., 10., 15.] -->
<port id="2"> <!-- output value is: [1., 3., 6., 10., 15.] -->
<dim>5</dim>
</port>
</output>
@ -82,16 +82,16 @@ To perform the summation in the opposite direction of the axis, set reverse attr
.. code-block:: xml
:force:
<layer ... type="CumSum" exclusive="1" reverse="0">
<input>
<port id="0"> < !-- input value is: [1., 2., 3., 4., 5.] -->
<port id="0"> <!-- input value is: [1., 2., 3., 4., 5.] -->
<dim>5</dim>
</port>
<port id="1"/> < !-- axis value is: 0 -->
<port id="1"/> <!-- axis value is: 0 -->
</input>
<output>
<port id="2"> < !-- output value is: [0., 1., 3., 6., 10.] -->
<port id="2"> <!-- output value is: [0., 1., 3., 6., 10.] -->
<dim>5</dim>
</port>
</output>
@ -101,16 +101,16 @@ To perform the summation in the opposite direction of the axis, set reverse attr
.. code-block:: xml
:force:
<layer ... type="CumSum" exclusive="0" reverse="1">
<input>
<port id="0"> < !-- input value is: [1., 2., 3., 4., 5.] -->
<port id="0"> <!-- input value is: [1., 2., 3., 4., 5.] -->
<dim>5</dim>
</port>
<port id="1"/> < !-- axis value is: 0 -->
<port id="1"/> <!-- axis value is: 0 -->
</input>
<output>
<port id="2"> < !-- output value is: [15., 14., 12., 9., 5.] -->
<port id="2"> <!-- output value is: [15., 14., 12., 9., 5.] -->
<dim>5</dim>
</port>
</output>
@ -120,7 +120,7 @@ To perform the summation in the opposite direction of the axis, set reverse attr
.. code-block:: xml
:force:
<layer ... type="CumSum" exclusive="1" reverse="1">
<input>
<port id="0"> < -- input value is: [1., 2., 3., 4., 5.] -->

View File

@ -5,7 +5,7 @@ Sqrt
.. meta::
:description: Learn about Sqrt-1 - an element-wise, arithmetic operation, which
:description: Learn about Sqrt-1 - an element-wise, arithmetic operation, which
can be performed on a single tensor in OpenVINO.
**Versioned name**: *Sqrt-1*
@ -48,12 +48,12 @@ Sqrt
<layer ... type="Sqrt">
<input>
<port id="0">
<dim>4</dim> < !-- float input values: [4.0, 7.0, 9.0, 10.0] -->
<dim>4</dim> <!-- float input values: [4.0, 7.0, 9.0, 10.0] -->
</port>
</input>
<output>
<port id="1">
<dim>4</dim> < !-- float output values: [2.0, 2.6457512, 3.0, 3.1622777] -->
<dim>4</dim> <!-- float output values: [2.0, 2.6457512, 3.0, 3.1622777] -->
</port>
</output>
</layer>
@ -66,12 +66,12 @@ Sqrt
<layer ... type="Sqrt">
<input>
<port id="0">
<dim>4</dim> < !-- int input values: [4, 7, 9, 10] -->
<dim>4</dim> <!-- int input values: [4, 7, 9, 10] -->
</port>
</input>
<output>
<port id="1">
<dim>4</dim> < !-- int output values: [2, 3, 3, 3] -->
<dim>4</dim> <!-- int output values: [2, 3, 3, 3] -->
</port>
</output>
</layer>

View File

@ -5,7 +5,7 @@ IsFinite
.. meta::
:description: Learn about IsFinite-10 - an element-wise, comparison operation, which
:description: Learn about IsFinite-10 - an element-wise, comparison operation, which
can be performed on a single tensor in OpenVINO.
**Versioned name**: *IsFinite-10*
@ -64,12 +64,12 @@ IsFinite
<layer ... type="IsFinite">
<input>
<port id="0" precision="FP32">
<dim>4</dim> < !-- Input value is: [NaN, 2.1, 3.7, Inf] -->
<dim>4</dim> <!-- Input value is: [NaN, 2.1, 3.7, Inf] -->
</port>
</input>
<output>
<port id="1" precision="BOOL">
<dim>4</dim> < !-- Output value is: [False, True, True, False] -->
<dim>4</dim> <!-- Output value is: [False, True, True, False] -->
</port>
</output>
</layer>

View File

@ -5,7 +5,7 @@ Select
.. meta::
:description: Learn about Select-1 - an element-wise, condition operation, which
:description: Learn about Select-1 - an element-wise, condition operation, which
can be performed on three given tensors in OpenVINO.
**Versioned name**: *Select-1*
@ -58,21 +58,21 @@ Select
<layer ... type="Select">
<input>
<port id="0"> < !-- cond value is: [[false, false], [true, false], [true, true]] -->
<port id="0"> <!-- cond value is: [[false, false], [true, false], [true, true]] -->
<dim>3</dim>
<dim>2</dim>
</port>
<port id="1"> < !-- then value is: [[-1, 0], [1, 2], [3, 4]] -->
<port id="1"> <!-- then value is: [[-1, 0], [1, 2], [3, 4]] -->
<dim>3</dim>
<dim>2</dim>
</port>
<port id="2"> < !-- else value is: [[11, 10], [9, 8], [7, 6]] -->
<port id="2"> <!-- else value is: [[11, 10], [9, 8], [7, 6]] -->
<dim>3</dim>
<dim>2</dim>
</port>
</input>
<output>
<port id="1"> < !-- output value is: [[11, 10], [1, 8], [3, 4]] -->
<port id="1"> <!-- output value is: [[11, 10], [1, 8], [3, 4]] -->
<dim>3</dim>
<dim>2</dim>
</port>

View File

@ -5,7 +5,7 @@ ConvolutionBackpropData
.. meta::
:description: Learn about ConvolutionBackpropData-1 - a 1D, 2D or 3D convolution operation, which
:description: Learn about ConvolutionBackpropData-1 - a 1D, 2D or 3D convolution operation, which
can be performed on input and kernel tensors in OpenVINO.
**Versioned name**: *ConvolutionBackpropData-1*
@ -24,11 +24,11 @@ When output shape is specified as an input tensor ``output_shape`` then it speci
.. code-block:: xml
:force:
if auto_pads != None:
pads_begin[i] = 0
pads_end[i] = 0
Y_i = stride[i] * (X_i - 1) + ((K_i - 1) * dilations[i] + 1) - pads_begin[i] - pads_end[i] + output_padding[i]
where ``K_i`` filter kernel dimension along spatial axis ``i``.
@ -37,7 +37,7 @@ If ``output_shape`` is specified, ``pads_begin`` and ``pads_end`` are ignored, a
.. code-block:: xml
:force:
total_padding[i] = stride[i] * (X_i - 1) + ((K_i - 1) * dilations[i] + 1) - output_shape[i] + output_padding[i]
if auto_pads != SAME_UPPER:
pads_begin[i] = total_padding[i] // 2
@ -81,7 +81,7 @@ If ``output_shape`` is specified, ``pads_begin`` and ``pads_end`` are ignored, a
* *auto_pad*
* **Description**: *auto_pad* has the same definition as *auto_pad* for a regular Convolution but applied in the backward way, for the output tensor.
* *explicit*: use explicit padding values from ``pads_begin`` and ``pads_end``.
* *same_upper* the input is padded to match the output size. In case of odd padding value an extra padding is added at the end.
* *same_lower* the input is padded to match the output size. In case of odd padding value an extra padding is added at the beginning.
@ -105,7 +105,7 @@ If ``output_shape`` is specified, ``pads_begin`` and ``pads_end`` are ignored, a
* **2**: Convolution kernel tensor of type *T1* and rank 3, 4 or 5. Layout is ``[C_INPUT, C_OUTPUT, Z, Y, X]`` (number of input channels, number of output channels, spatial axes Z, Y, X). Spatial size of the kernel is derived from the shape of this input and aren't specified by any attribute. **Required.**
* **3**: ``output_shape`` is 1D tensor of type *T2* that specifies spatial shape of the output. If specified, *padding amount* is deduced from relation of input and output spatial shapes according to formulas in the description. If not specified, *output shape* is calculated based on the ``pads_begin`` and ``pads_end`` or completely according to ``auto_pad``. **Optional.**
* **Note**: Type of the convolution (1D, 2D or 3D) is derived from the rank of the input tensors and not specified by any attribute:
* 1D convolution (input tensors rank 3) means that there is only one spatial axis X,
* 2D convolution (input tensors rank 4) means that there are two spatial axes Y, X,
* 3D convolution (input tensors rank 5) means that there are three spatial axes Z, Y, X.
@ -125,7 +125,7 @@ If ``output_shape`` is specified, ``pads_begin`` and ``pads_end`` are ignored, a
.. code-block:: xml
:force:
<layer id="5" name="upsampling_node" type="ConvolutionBackpropData">
<data dilations="1,1" pads_begin="1,1" pads_end="1,1" strides="2,2" output_padding="0,0" auto_pad="explicit"/>
<input>
@ -156,7 +156,7 @@ If ``output_shape`` is specified, ``pads_begin`` and ``pads_end`` are ignored, a
.. code-block:: xml
:force:
<layer id="5" name="upsampling_node" type="ConvolutionBackpropData">
<data dilations="1,1" pads_begin="0,0" pads_end="0,0" strides="3,3" output_padding="2,2" auto_pad="explicit"/>
<input>
@ -187,7 +187,7 @@ If ``output_shape`` is specified, ``pads_begin`` and ``pads_end`` are ignored, a
.. code-block:: xml
:force:
<layer id="5" name="upsampling_node" type="ConvolutionBackpropData">
<data dilations="1,1" pads_begin="1,1" pads_end="1,1" strides="1,1" output_padding="0,0" auto_pad="valid"/>
<input>
@ -204,7 +204,7 @@ If ``output_shape`` is specified, ``pads_begin`` and ``pads_end`` are ignored, a
<dim>3</dim>
</port>
<port id="2">
<dim>2</dim> < !-- output_shape value is: [450, 450]-->
<dim>2</dim> <!-- output_shape value is: [450, 450]-->
</port>
</input>
<output>

View File

@ -5,7 +5,7 @@ PriorBoxClustered
.. meta::
:description: Learn about PriorBoxClustered-1 - an object detection operation,
:description: Learn about PriorBoxClustered-1 - an object detection operation,
which can be performed on two 1D input tensors.
**Versioned name**: *PriorBoxClustered-1*
@ -94,7 +94,7 @@ If *clip* is defined, the coordinates of prior boxes are recalculated with the f
* *step (step_w, step_h)*
* **Description**: *step (step_w, step_h)* is a distance between box centers. For example, *step* equal 85 means that the distance between neighborhood prior boxes centers is 85. If both *step_h* and *step_w* are 0 then they are updated with value of *step*. If after that they are still 0 then they are calculated as input image width(height) divided with first input width(height).
* **Description**: *step (step_w, step_h)* is a distance between box centers. For example, *step* equal 85 means that the distance between neighborhood prior boxes centers is 85. If both *step_h* and *step_w* are 0 then they are updated with value of *step*. If after that they are still 0 then they are calculated as input image width(height) divided with first input width(height).
* **Range of values**: floating-point positive number
* **Type**: ``float``
* **Default value**: 0.0
@ -139,10 +139,10 @@ If *clip* is defined, the coordinates of prior boxes are recalculated with the f
<data clip="false" height="44.0,10.0,30.0,19.0,94.0,32.0,61.0,53.0,17.0" offset="0.5" step="16.0" variance="0.1,0.1,0.2,0.2" width="86.0,13.0,57.0,39.0,68.0,34.0,142.0,50.0,23.0"/>
<input>
<port id="0">
<dim>2</dim> < !-- [10, 19] -->
<dim>2</dim> <!-- [10, 19] -->
</port>
<port id="1">
<dim>2</dim> < !-- [180, 320] -->
<dim>2</dim> <!-- [180, 320] -->
</port>
</input>
<output>

View File

@ -6,7 +6,7 @@ PriorBox
.. meta::
:description: Learn about PriorBox-1 - an object detection operation,
:description: Learn about PriorBox-1 - an object detection operation,
which can be performed on two required input tensors.
**Versioned name**: *PriorBox-1*
@ -22,44 +22,44 @@ PriorBox
1. First calculates *center_x* and *center_y* of prior box:
.. math::
W \equiv Width \quad Of \quad Image \\ H \equiv Height \quad Of \quad Image
* If step equals 0:
.. math::
center_x=(w+0.5) \\ center_y=(h+0.5)
* else:
.. math::
center_x=(w+offset)*step \\ center_y=(h+offset)*step \\ w \subset \left( 0, W \right ) \\ h \subset \left( 0, H \right )
2. Then, for each :math:`s \subset \left( 0, min\_sizes \right )` calculates coordinates of prior boxes:
.. math::
xmin = \frac{\frac{center_x - s}{2}}{W}
.. math::
ymin = \frac{\frac{center_y - s}{2}}{H}
.. math::
xmax = \frac{\frac{center_x + s}{2}}{W}
.. math::
ymin = \frac{\frac{center_y + s}{2}}{H}
3. If *clip* attribute is set to true, each output value is clipped between :math:`\left< 0, 1 \right>`.
@ -186,10 +186,10 @@ PriorBox
<data aspect_ratio="2.0" clip="false" density="" fixed_ratio="" fixed_size="" flip="true" max_size="38.46" min_size="16.0" offset="0.5" step="16.0" variance="0.1,0.1,0.2,0.2"/>
<input>
<port id="0">
<dim>2</dim> < !-- values: [24, 42] -->
<dim>2</dim> <!-- values: [24, 42] -->
</port>
<port id="1">
<dim>2</dim> < !-- values: [384, 672] -->
<dim>2</dim> <!-- values: [384, 672] -->
</port>
</input>
<output>

View File

@ -5,7 +5,7 @@ PriorBox
.. meta::
:description: Learn about PriorBox-8 - an object detection operation,
:description: Learn about PriorBox-8 - an object detection operation,
which can be performed on two required input tensors.
**Versioned name**: *PriorBox-8*
@ -21,41 +21,41 @@ PriorBox
1. First, it calculates *center_x* and *center_y* of a prior box:
.. math::
W \equiv Width \quad Of \quad Image \\ H \equiv Height \quad Of \quad Image
* If step equals 0:
.. math::
center_x=(w+0.5) \\ center_y=(h+0.5)
* else:
.. math::
center_x=(w+offset)*step \\ center_y=(h+offset)*step \\ w \subset \left( 0, W \right ) \\ h \subset \left( 0, H \right )
2. Then, it calculates coordinates of prior boxes for each :math:`s \subset \left( 0, min\_sizes \right )` :
.. math::
xmin = \frac{\frac{center_x - s}{2}}{W}
.. math::
ymin = \frac{\frac{center_y - s}{2}}{H}
.. math::
xmax = \frac{\frac{center_x + s}{2}}{W}
.. math::
ymin = \frac{\frac{center_y + s}{2}}{H}
3. If *clip* attribute is set to true, each output value is clipped between :math:`\left< 0, 1 \right>`.
@ -82,7 +82,7 @@ PriorBox
* **Description**: *flip* is a flag that denotes that each *aspect_ratio* is duplicated and flipped. For example, *flip* equals 1 and *aspect_ratio* equals ``[4.0,2.0]``, meaning that the aspect_ratio is equal to ``[4.0,2.0,0.25,0.5]``.
* **Range of values**:
* false or 0 - each *aspect_ratio* is flipped
* true or 1 - each *aspect_ratio* is not flipped
* **Type**: ``boolean``
@ -193,10 +193,10 @@ PriorBox
<data aspect_ratio="2.0" clip="false" density="" fixed_ratio="" fixed_size="" flip="true" max_size="38.46" min_size="16.0" offset="0.5" step="16.0" variance="0.1,0.1,0.2,0.2"/>
<input>
<port id="0">
<dim>2</dim> < !-- values: [24, 42] -->
<dim>2</dim> <!-- values: [24, 42] -->
</port>
<port id="1">
<dim>2</dim> < !-- values: [384, 672] -->
<dim>2</dim> <!-- values: [384, 672] -->
</port>
</input>
<output>

View File

@ -5,7 +5,7 @@ RegionYolo
.. meta::
:description: Learn about RegionYolo-1 - an object detection operation,
:description: Learn about RegionYolo-1 - an object detection operation,
which can be performed on a 4D input tensor.
**Versioned name**: *RegionYolo-1*
@ -65,7 +65,7 @@ RegionYolo
* **Description**: *do_softmax* is a flag that specifies the inference method and affects how the number of regions is determined. It also affects output shape. If it is 0, then output shape is 4D, and 2D otherwise.
* **Range of values**:
* *false* - do not perform softmax
* *true* - perform softmax
* **Type**: ``boolean``
@ -100,7 +100,7 @@ RegionYolo
.. code-block:: xml
:force:
< !-- YOLO V3 example -->
<!-- YOLO V3 example -->
<layer type="RegionYolo" ... >
<data anchors="10,14,23,27,37,58,81,82,135,169,344,319" axis="1" classes="80" coords="4" do_softmax="0" end_axis="3" mask="0,1,2" num="6"/>
<input>
@ -120,8 +120,8 @@ RegionYolo
</port>
</output>
</layer>
< !-- YOLO V2 Example -->
<!-- YOLO V2 Example -->
<layer type="RegionYolo" ... >
<data anchors="1.08,1.19,3.42,4.41,6.63,11.38,9.42,5.11,16.62,10.52" axis="1" classes="20" coords="4" do_softmax="1" end_axis="3" num="5"/>
<input>

View File

@ -5,7 +5,7 @@ Eye
.. meta::
:description: Learn about Eye-9 - a generation operation, which can be
:description: Learn about Eye-9 - a generation operation, which can be
performed on three required and one optional input tensors.
**Versioned name**: *Eye-9*
@ -23,13 +23,13 @@ Example 1. *Eye* output with ``output_type`` = ``i32``:
.. code-block:: xml
:force:
num_rows = 3
num_columns = 4
diagonal_index = 2
output = [[0 0 1 0]
[0 0 0 1]
[0 0 0 0]]
@ -38,13 +38,13 @@ Example 2. *Eye* output with ``output_type`` = ``i32``:
.. code-block:: xml
:force:
num_rows = 3
num_columns = 4
diagonal_index = -1
output = [[0 0 0 0]
[1 0 0 0]
[0 1 0 0]]
@ -53,13 +53,13 @@ Example 3. *Eye* output with ``output_type`` = ``f16``:
.. code-block:: xml
:force:
num_rows = 2
diagonal_index = 5
batch_shape = [1, 2]
output = [[[[0. 0.]
[0. 0.]]
[[0. 0.]
@ -97,13 +97,13 @@ Example 3. *Eye* output with ``output_type`` = ``f16``:
.. code-block:: xml
:force:
<layer ... name="Eye" type="Eye">
<data output_type="i8"/>
<input>
<port id="0" precision="I32"/> < !-- num rows: 5 -->
<port id="1" precision="I32"/> < !-- num columns: 5 -->
<port id="2" precision="I32"/> < !-- diagonal index -->
<port id="0" precision="I32"/> <!-- num rows: 5 -->
<port id="1" precision="I32"/> <!-- num columns: 5 -->
<port id="2" precision="I32"/> <!-- diagonal index -->
</input>
<output>
<port id="3" precision="I8" names="Eye:0">
@ -117,14 +117,14 @@ Example 3. *Eye* output with ``output_type`` = ``f16``:
.. code-block:: xml
:force:
<layer ... name="Eye" type="Eye">
<data output_type="f32"/>
<input>
<port id="0" precision="I32"/> < !-- num rows -->
<port id="1" precision="I32"/> < !-- num columns -->
<port id="2" precision="I32"/> < !-- diagonal index -->
<port id="3" precision="I32"/> < !-- batch_shape : [2, 3] -->
<port id="0" precision="I32"/> <!-- num rows -->
<port id="1" precision="I32"/> <!-- num columns -->
<port id="2" precision="I32"/> <!-- diagonal index -->
<port id="3" precision="I32"/> <!-- batch_shape : [2, 3] -->
</input>
<output>
<port id="3" precision="F32" names="Eye:0">

View File

@ -91,10 +91,10 @@ Example 3 - 2D tensor, without replacement
* **Description**: controls whether to sample with replacement (classes can be sampled multiple times).
* **Range of values**: `true`, `false`
* ``true`` - class indices can be sampled multiple times.
* ``false`` - class indices will not repeat in the output and the size of ``probs``' ``class_size`` dimension is required to be larger or equal to *num_samples* value. Might affect performance.
* **Type**: `bool`
* **Required**: *Yes*
@ -149,16 +149,16 @@ Example 3 - 2D tensor, without replacement
<layer ... name="Multinomial" type="Multinomial">
<data convert_type="f32", with_replacement="true", log_probs="false", global_seed="234", op_seed="148"/>
<input>
<port id="0" precision="FP32"> < !-- probs value: [[0.1, 0.5, 0.4]] -->
<dim>1</dim> < !-- batch size of 2 -->
<port id="0" precision="FP32"> <!-- probs value: [[0.1, 0.5, 0.4]] -->
<dim>1</dim> <!-- batch size of 2 -->
<dim>3</dim>
</port>
<port id="1" precision="I32"/> < !-- num_samples value: 5 -->
<port id="1" precision="I32"/> <!-- num_samples value: 5 -->
</input>
<output>
<port id="3" precision="I32" names="Multinomial:0">
<dim>1</dim> < !--dimension depends on input batch size -->
<dim>5</dim> < !--dimension depends on num_samples -->
<dim>1</dim> <!--dimension depends on input batch size -->
<dim>5</dim> <!--dimension depends on num_samples -->
</port>
</output>
</layer>
@ -171,16 +171,16 @@ Example 3 - 2D tensor, without replacement
<layer ... name="Multinomial" type="Multinomial">
<data convert_type="f32", with_replacement="true", log_probs="true", global_seed="234", op_seed="148"/>
<input>
<port id="0" precision="FP32"> < !-- probs value: [[-1, 1, 2], [50, 1, 21]] -->
<dim>2</dim> < !-- batch size of 2 -->
<port id="0" precision="FP32"> <!-- probs value: [[-1, 1, 2], [50, 1, 21]] -->
<dim>2</dim> <!-- batch size of 2 -->
<dim>3</dim>
</port>
<port id="1" precision="I32"/> < !-- num_samples value: 10 -->
<port id="1" precision="I32"/> <!-- num_samples value: 10 -->
</input>
<output>
<port id="3" precision="I32" names="Multinomial:0">
<dim>2</dim> < !--dimension depends on input batch size -->
<dim>10</dim> < !--dimension depends on num_samples -->
<dim>2</dim> <!--dimension depends on input batch size -->
<dim>10</dim> <!--dimension depends on num_samples -->
</port>
</output>
</layer>
@ -193,16 +193,16 @@ Example 3 - 2D tensor, without replacement
<layer ... name="Multinomial" type="Multinomial">
<data convert_type="f32", with_replacement="false", log_probs="false", global_seed="234", op_seed="148"/>
<input>
<port id="0" precision="FP32"> < !-- probs value: [[0.1, 0.5, 0.4]] -->
<dim>2</dim> < !-- batch size of 2 -->
<port id="0" precision="FP32"> <!-- probs value: [[0.1, 0.5, 0.4]] -->
<dim>2</dim> <!-- batch size of 2 -->
<dim>3</dim>
</port>
<port id="1" precision="I32"/> < !-- num_samples value: 2 -->
<port id="1" precision="I32"/> <!-- num_samples value: 2 -->
</input>
<output>
<port id="3" precision="I32" names="Multinomial:0">
<dim>2</dim> < !-- batch size of 2 -->
<dim>2</dim> < !-- 2 unique samples of classes -->
<dim>2</dim> <!-- batch size of 2 -->
<dim>2</dim> <!-- 2 unique samples of classes -->
</port>
</output>
</layer>

View File

@ -5,7 +5,7 @@ RandomUniform
.. meta::
:description: Learn about RandomUniform-8 - a generation operation, which can be
:description: Learn about RandomUniform-8 - a generation operation, which can be
performed on three required input tensors.
**Versioned name**: *RandomUniform-8*
@ -16,10 +16,10 @@ RandomUniform
**Detailed description**:
*RandomUniform* operation generates random numbers from a uniform distribution in the range ``[minval, maxval)``.
The generation algorithm is based on underlying random integer generator that uses Philox algorithm. Philox algorithm
is a counter-based pseudo-random generator, which produces uint32 values. Single invocation of Philox algorithm returns
four result random values, depending on the given *key* and *counter* values. *Key* and *counter* are initialized
*RandomUniform* operation generates random numbers from a uniform distribution in the range ``[minval, maxval)``.
The generation algorithm is based on underlying random integer generator that uses Philox algorithm. Philox algorithm
is a counter-based pseudo-random generator, which produces uint32 values. Single invocation of Philox algorithm returns
four result random values, depending on the given *key* and *counter* values. *Key* and *counter* are initialized
with *global_seed* and *op_seed* attributes respectively.
If both seed values equal to zero, RandomUniform generates non-deterministic sequence.
@ -32,7 +32,7 @@ If both seed values equal to zero, RandomUniform generates non-deterministic seq
Link to the original paper `Parallel Random Numbers: As Easy as 1, 2, 3 <https://www.thesalmons.org/john/random123/papers/random123sc11.pdf>`__.
The result of Philox is calculated by applying a fixed number of *key* and *counter* updating so-called "rounds".
The result of Philox is calculated by applying a fixed number of *key* and *counter* updating so-called "rounds".
This implementation uses 4x32_10 version of Philox algorithm, where number of rounds = 10.
Suppose we have *n* which determines *n*-th 4 elements of random sequence.
@ -43,7 +43,7 @@ In each round *key*, *counter* and *n* are splitted to pairs of uint32 values:
R = cast\_to\_uint32(value)\\
L = cast\_to\_uint32(value >> 32),
where *cast\_to\_uint32* - static cast to uint32, *value* - uint64 input value, *L*, *R* - uint32
where *cast\_to\_uint32* - static cast to uint32, *value* - uint64 input value, *L*, *R* - uint32
result values, >> - bitwise right shift.
Then *n* and *counter* are updated with the following formula:
@ -68,7 +68,7 @@ Values :math:`L'_{n}, R'_{n}, L'_{counter}, R'_{counter}` are resulting four ran
Float values between [0..1) are obtained from 32-bit integers by the following rules.
Float16 is formatted as follows: *sign* (1 bit) *exponent* (5 bits) *mantissa* (10 bits). The value is interpreted
Float16 is formatted as follows: *sign* (1 bit) *exponent* (5 bits) *mantissa* (10 bits). The value is interpreted
using following formula:
.. math::
@ -99,7 +99,7 @@ where x is uint32 generated random value.
Float32 is formatted as follows: *sign* (1 bit) *exponent* (8 bits) *mantissa* (23 bits). The value is interpreted using following formula:
.. math::
(-1)^{sign} * 1, mantissa * 2 ^{exponent - 127}
@ -117,7 +117,7 @@ So the resulting float value is:
.. code-block:: xml
:force:
val = ((exponent << 23) | x & 0x7fffffu) - 1.0,
where x is uint32 generated random value.
@ -125,7 +125,7 @@ where x is uint32 generated random value.
Double is formatted as follows: *sign* (1 bit) *exponent* (11 bits) *mantissa* (52 bits). The value is interpreted using following formula:
.. math::
(-1)^{sign} * 1, mantissa * 2 ^{exponent - 1023}
@ -133,7 +133,7 @@ so to obtain double values *sign*, *exponent* and *mantissa* are set as follows:
.. code-block:: xml
:force:
sign = 0
exponent = 1023 - representation of a zero exponent.
mantissa = 52 right bits from two concatinated uint32 values from random integer generator.
@ -143,7 +143,7 @@ So the resulting double is obtained as follows:
.. code-block:: xml
:force:
mantissa_h = x0 & 0xfffffu; // upper 20 bits of mantissa
mantissa_l = x1; // lower 32 bits of mantissa
mantissa = (mantissa_h << 32) | mantissa_l;
@ -156,7 +156,7 @@ To obtain a value in a specified range each value is processed with the followin
For float values:
.. math::
result = x * (maxval - minval) + minval,
where *x* is random float or double value between [0..1).
@ -174,7 +174,7 @@ Example 1. *RandomUniform* output with ``global_seed`` = 150, ``op_seed`` = 10,
.. code-block:: xml
:force:
input_shape = [ 3, 3 ]
output = [[0.7011236 0.30539632 0.93931055]
[0.9456035 0.11694777 0.50770056]
@ -185,7 +185,7 @@ Example 2. *RandomUniform* output with ``global_seed`` = 80, ``op_seed`` = 100,
.. code-block:: xml
:force:
input_shape = [ 2, 2 ]
minval = 2
@ -200,7 +200,7 @@ Example 3. *RandomUniform* output with ``global_seed`` = 80, ``op_seed`` = 100,
.. code-block:: xml
:force:
input_shape = [ 2, 3 ]
minval = 50
@ -261,11 +261,11 @@ Example 3. *RandomUniform* output with ``global_seed`` = 80, ``op_seed`` = 100,
<layer ... name="RandomUniform" type="RandomUniform">
<data output_type="f32" global_seed="234" op_seed="148"/>
<input>
<port id="0" precision="I32"> < !-- shape value: [2, 3, 10] -->
<port id="0" precision="I32"> <!-- shape value: [2, 3, 10] -->
<dim>3</dim>
</port>
<port id="1" precision="FP32"/> < !-- min value -->
<port id="2" precision="FP32"/> < !-- max value -->
<port id="1" precision="FP32"/> <!-- min value -->
<port id="2" precision="FP32"/> <!-- max value -->
</input>
<output>
<port id="3" precision="FP32" names="RandomUniform:0">

View File

@ -5,7 +5,7 @@ Range
.. meta::
:description: Learn about Range-1 - a generation operation, which can be
:description: Learn about Range-1 - a generation operation, which can be
performed on three required input tensors.
**Versioned name**: *Range-1*
@ -46,7 +46,7 @@ For a positive ``step``:
for a negative ``step``:
.. math::
start>=val[i]>stop,
@ -66,16 +66,16 @@ where
<layer ... type="Range">
<input>
<port id="0"> < !-- start value: 2 -->
<port id="0"> <!-- start value: 2 -->
</port>
<port id="1"> < !-- stop value: 23 -->
<port id="1"> <!-- stop value: 23 -->
</port>
<port id="2"> < !-- step value: 3 -->
<port id="2"> <!-- step value: 3 -->
</port>
</input>
<output>
<port id="3">
<dim>7</dim> < !-- [ 2, 5, 8, 11, 14, 17, 20] -->
<dim>7</dim> <!-- [ 2, 5, 8, 11, 14, 17, 20] -->
</port>
</output>
</layer>
@ -88,16 +88,16 @@ where
<layer ... type="Range">
<input>
<port id="0"> < !-- start value: 23 -->
<port id="0"> <!-- start value: 23 -->
</port>
<port id="1"> < !-- stop value: 2 -->
<port id="1"> <!-- stop value: 2 -->
</port>
<port id="2"> < !-- step value: -3 -->
<port id="2"> <!-- step value: -3 -->
</port>
</input>
<output>
<port id="3">
<dim>7</dim> < !-- [23, 20, 17, 14, 11, 8, 5] -->
<dim>7</dim> <!-- [23, 20, 17, 14, 11, 8, 5] -->
</port>
</output>
</layer>

View File

@ -5,7 +5,7 @@ Range
.. meta::
:description: Learn about Range-4 - a generation operation, which can be
:description: Learn about Range-4 - a generation operation, which can be
performed on three required input tensors.
**Versioned name**: *Range-4*
@ -81,16 +81,16 @@ This is aligned with PyTorch's operation ``torch.arange``, to align with tensorf
<layer ... type="Range">
<data output_type="i32">
<input>
<port id="0"> < !-- start value: 2 -->
<port id="0"> <!-- start value: 2 -->
</port>
<port id="1"> < !-- stop value: 23 -->
<port id="1"> <!-- stop value: 23 -->
</port>
<port id="2"> < !-- step value: 3 -->
<port id="2"> <!-- step value: 3 -->
</port>
</input>
<output>
<port id="3">
<dim>7</dim> < !-- [ 2, 5, 8, 11, 14, 17, 20] -->
<dim>7</dim> <!-- [ 2, 5, 8, 11, 14, 17, 20] -->
</port>
</output>
</layer>
@ -104,16 +104,16 @@ This is aligned with PyTorch's operation ``torch.arange``, to align with tensorf
<layer ... type="Range">
<data output_type="i32">
<input>
<port id="0"> < !-- start value: 23 -->
<port id="0"> <!-- start value: 23 -->
</port>
<port id="1"> < !-- stop value: 2 -->
<port id="1"> <!-- stop value: 2 -->
</port>
<port id="2"> < !-- step value: -3 -->
<port id="2"> <!-- step value: -3 -->
</port>
</input>
<output>
<port id="3">
<dim>7</dim> < !-- [23, 20, 17, 14, 11, 8, 5] -->
<dim>7</dim> <!-- [23, 20, 17, 14, 11, 8, 5] -->
</port>
</output>
</layer>
@ -127,16 +127,16 @@ This is aligned with PyTorch's operation ``torch.arange``, to align with tensorf
<layer ... type="Range">
<data output_type="f32">
<input>
<port id="0"> < !-- start value: 1 -->
<port id="0"> <!-- start value: 1 -->
</port>
<port id="1"> < !-- stop value: 2.5 -->
<port id="1"> <!-- stop value: 2.5 -->
</port>
<port id="2"> < !-- step value: 0.5 -->
<port id="2"> <!-- step value: 0.5 -->
</port>
</input>
<output>
<port id="3">
<dim>3</dim> < !-- [ 1.0, 1.5, 2.0] -->
<dim>3</dim> <!-- [ 1.0, 1.5, 2.0] -->
</port>
</output>
</layer>

View File

@ -5,7 +5,7 @@ I420toBGR
.. meta::
:description: Learn about I420toBGR-8 - an image processing operation, which
:description: Learn about I420toBGR-8 - an image processing operation, which
can be performed to convert image from I420 to BGR format.
**Versioned name**: *I420toBGR-8*
@ -70,19 +70,19 @@ Same as specified for :doc:`I420toRGB <openvino_docs_ops_image_I420toRGB_8>` ope
<layer ... type="I420toBGR">
<input>
<port id="0">  < !-- Y plane -->
<port id="0">  <!-- Y plane -->
<dim>1</dim>
<dim>480</dim>
<dim>640</dim>
<dim>1</dim>
</port>
<port id="1">  < !-- U plane -->
<port id="1">  <!-- U plane -->
<dim>1</dim>
<dim>240</dim>
<dim>320</dim>
<dim>1</dim>
</port>
<port id="2">  < !-- V plane -->
<port id="2">  <!-- V plane -->
<dim>1</dim>
<dim>240</dim>
<dim>320</dim>

View File

@ -5,7 +5,7 @@ I420toRGB
.. meta::
:description: Learn about I420toRGB-8 - an image processing operation, which
:description: Learn about I420toRGB-8 - an image processing operation, which
can be performed to convert image from I420 to RGB format.
**Versioned name**: *I420toRGB-8*
@ -113,19 +113,19 @@ Input I420 image tensor shall have ``NHWC (also known as NYXC)`` layout and can
<layer ... type="I420toRGB">
<input>
<port id="0">  < !-- Y plane -->
<port id="0">  <!-- Y plane -->
<dim>1</dim>
<dim>480</dim>
<dim>640</dim>
<dim>1</dim>
</port>
<port id="1">  < !-- U plane -->
<port id="1">  <!-- U plane -->
<dim>1</dim>
<dim>240</dim>
<dim>320</dim>
<dim>1</dim>
</port>
<port id="2">  < !-- V plane -->
<port id="2">  <!-- V plane -->
<dim>1</dim>
<dim>240</dim>
<dim>320</dim>

View File

@ -5,7 +5,7 @@ Interpolate
.. meta::
:description: Learn about I420toRGB-8 - an image processing operation, which
:description: Learn about I420toRGB-8 - an image processing operation, which
can be performed on two required tensors.
**Versioned name**: *Interpolate-1*
@ -91,7 +91,7 @@ This is a scalar that specifies padding for each spatial dimension.
<dim>80</dim>
</port>
<port id="1">
<dim>2</dim>  < !--The values in this input are [50, 60] -->
<dim>2</dim>  <!--The values in this input are [50, 60] -->
</port>
</input>
<output>

View File

@ -5,7 +5,7 @@ Interpolate
.. meta::
:description: Learn about Interpolate-11 - an image processing operation, which
:description: Learn about Interpolate-11 - an image processing operation, which
can be performed on two required and one optional tensor.
**Versioned name**: *Interpolate-11*
@ -129,13 +129,13 @@ Interpolate
<dim>80</dim>
</port>
<port id="1">
<dim>2</dim> < !--The values in this input are [24, 160] -->
<dim>2</dim> <!--The values in this input are [24, 160] -->
</port>
<port id="2">
<dim>2</dim> < !--The values in this input are [0.5, 2.0] -->
<dim>2</dim> <!--The values in this input are [0.5, 2.0] -->
</port>
<port id="3">
<dim>2</dim> < !--The values in this input are [2, 3] (axes). -->
<dim>2</dim> <!--The values in this input are [2, 3] (axes). -->
</port>
</input>
<output>

View File

@ -5,7 +5,7 @@ Interpolate
.. meta::
:description: Learn about Interpolate-4 - an image processing operation, which
:description: Learn about Interpolate-4 - an image processing operation, which
can be performed on three required and one optional tensor.
**Versioned name**: *Interpolate-4*
@ -128,7 +128,7 @@ Calculations are performed according to the following rules.
import math
import numpy as np
from enum import Enum, unique
class GetNearestPixel:
def __init__(self, mode: str):
self.func = {
@ -138,37 +138,37 @@ Calculations are performed according to the following rules.
'ceil': GetNearestPixel.ceil_func,
'simple': GetNearestPixel.simple_func
}[mode]
def __call__(self, x_original, is_downsample):
return self.func(x_original, is_downsample)
@staticmethod
def prefer_floor_func(x_original, is_downsample):
if x_original == int(x_original) + 0.5:
return int(math.floor(x_original))
else:
return int(round(x_original))
@staticmethod
def prefer_ceil_func(x_original, is_downsample):
return int(round(x_original))
@staticmethod
def floor_func(x_original, is_downsample):
return int(math.floor(x_original))
@staticmethod
def ceil_func(x_original, is_downsample):
return int(math.ceil(x_original))
@staticmethod
def simple_func(x_original, is_downsample):
if is_downsample:
return int(math.ceil(x_original))
else:
return int(x_original)
class GetOriginalCoordinate:
def __init__(self, mode: str):
self.func = {
@ -178,31 +178,31 @@ Calculations are performed according to the following rules.
'tf_half_pixel_for_nn': GetOriginalCoordinate.tf_half_pixel_for_nn_func,
'align_corners': GetOriginalCoordinate.align_corners_func
}[mode]
def __call__(self, x_resized, x_scale, length_resized, length_original):
return self.func(x_resized, x_scale, length_resized, length_original)
@staticmethod
def half_pixel_func(x_resized, x_scale, length_resized, length_original):
return ((x_resized + 0.5) / x_scale) - 0.5
@staticmethod
def pytorch_half_pixel_func(x_resized, x_scale, length_resized, length_original):
return (x_resized + 0.5) / x_scale - 0.5 if length_resized > 1 else 0.0
@staticmethod
def asymmetric_func(x_resized, x_scale, length_resized, length_original):
return x_resized / x_scale
@staticmethod
def tf_half_pixel_for_nn_func(x_resized, x_scale, length_resized, length_original):
return (x_resized + 0.5) / x_scale
@staticmethod
def align_corners_func(x_resized, x_scale, length_resized, length_original):
return 0 if length_resized == 1 else x_resized * (length_original - 1) / (length_resized - 1)
def get_cubic_coeff(s, a):
abs_s = abs(s)
coeff = np.zeros(4)
@ -211,18 +211,18 @@ Calculations are performed according to the following rules.
coeff[2] = (((-a -2.0) * abs_s+ (2.0 * a + 3.0)) * abs_s - a) * abs_s
coeff[3] = - a * abs_s * abs_s * (abs_s - 1.0)
return coeff
def triangle_coeffs(dz):
return np.maximum(0.0, 1.0 - np.abs(dz))
@unique
class ShapeCalculationMode(Enum):
SIZES = 0
SCALES = 1
class InterpolateCalculation:
def __init__(self, attrs: dict):
self.mode = attrs['mode']
@ -233,38 +233,38 @@ Calculations are performed according to the following rules.
'linear_onnx': self.onnx_linear_interpolation
}[self.mode]
self.attrs = attrs
self.pads_begin = attrs.get('pads_begin', [0])
self.pads_end = attrs.get('pads_end', [0])
self.coordinate_transformation_mode = attrs.get('coordinate_transformation_mode', 'half_pixel')
self.nearest_mode = attrs.get('nearest_mode', 'round_prefer_floor')
self.cube_coeff = attrs.get('cube_coeff', -0.75)
self.antialias = attrs.get('antialias', False)
self.shape_calculation_mode = {
'sizes': ShapeCalculationMode.SIZES,
'scales': ShapeCalculationMode.SCALES
}[attrs['shape_calculation_mode']]
self.get_original_coordinate = self.get_coordinate_transformation_mode()
self.get_nearest_pixel = GetNearestPixel(self.nearest_mode)
def get_coordinate_transformation_mode(self):
return GetOriginalCoordinate(self.coordinate_transformation_mode)
def shape_infer(self, input_data, sizes, scales):
result = input_data.shape + self.pads_begin + self.pads_end
if self.shape_calculation_mode == ShapeCalculationMode.SIZES:
for i, axis in enumerate(self.axes):
result[axis] = sizes[i]
else:
for i, axis in enumerate(self.axes):
result[axis] = math.floor(scales[i] * result[axis])
return result
@staticmethod
def correct_pad(pad, rank):
pad_len = len(pad)
@ -274,17 +274,17 @@ Calculations are performed according to the following rules.
return np.array(pad[: rank - 1]).astype(np.int64)
else:
return np.array(pad, dtype=np.int64)
def __call__(self, input_data, sizes, scales, axes):
rank = input_data.ndim
self.pads_begin = InterpolateCalculation.correct_pad(self.pads_begin, rank)
self.pads_end = InterpolateCalculation.correct_pad(self.pads_end, rank)
self.pads = list(zip(self.pads_begin, self.pads_end))
self.axes = np.array(axes).astype(np.int64)
self.output_shape = self.shape_infer(input_data, sizes, scales)
padded_data = np.pad(input_data, self.pads, 'constant')
if self.shape_calculation_mode == ShapeCalculationMode.SIZES:
num_of_axes = len(self.axes)
self.scales = np.zeros(num_of_axes)
@ -292,18 +292,18 @@ Calculations are performed according to the following rules.
self.scales[i] = self.output_shape[axis] / padded_data.shape[axis]
else:
self.scales = scales
if self.mode == 'nearest':
self.all_scales = np.ones(rank).astype(np.float)
for i, axis in enumerate(self.axes):
self.all_scales[axis] = self.scales[i]
self.input_shape = padded_data.shape
return self.func(padded_data)
def clip_coord(self, coord, axis):
return max(0, min(coord, self.input_shape[axis] - 1))
def cubic_interpolation(self, input_data):
rank = len(self.input_shape)
result = np.zeros(self.output_shape)
@ -328,28 +328,28 @@ Calculations are performed according to the following rules.
summa += coeffs_prod * input_data[tuple(coords_for_sum)]
result[coordinates] = summa
return result
def linear_interpolation(self, input_data):
result = np.zeros(self.output_shape)
num_of_axes = len(self.axes)
is_downsample = False
for scale in self.scales:
is_downsample = is_downsample or (scale < 1)
antialias = is_downsample and self.antialias
a = np.zeros(num_of_axes)
for i, _ in enumerate(self.axes):
a[i] = self.scales[i] if antialias else 1.0
prod_of_a = np.prod(a)
r = np.zeros(num_of_axes).astype(np.int64)
for i, _ in enumerate(self.axes):
r[i] = 2 if self.scales[i] > 1.0 else int(math.ceil(2.0/a[i]))
indices = [tuple(np.array(ind).astype(np.int64) - r) for ind in np.ndindex(tuple(2 * r + 1))]
for coordinates in np.ndindex(tuple(self.output_shape)):
icoords = np.array(coordinates).astype(np.float64)
icoords_r = np.array(coordinates).astype(np.float64)
@ -357,51 +357,51 @@ Calculations are performed according to the following rules.
in_coord = self.get_original_coordinate(coordinates[axis], self.scales[i], self.output_shape[axis], self.input_shape[axis])
icoords[axis] = in_coord
icoords_r[axis] = round(in_coord)
summa = 0.0
wsum = 0.0
for index in indices:
inner_coords = np.array(coordinates)
for i, axis in enumerate(self.axes):
inner_coords[axis] = index[i] + icoords_r[axis]
conditions = [inner_coords[axis] >= 0 and inner_coords[axis] < self.input_shape[axis] for axis in self.axes]
if not all(conditions):
continue
dz = np.zeros(num_of_axes)
for i, axis in enumerate(self.axes):
dz[i] = icoords[axis] - inner_coords[axis]
w = prod_of_a * np.prod(triangle_coeffs(a * dz))
wsum += w
summa += w * input_data[tuple(inner_coords)]
if wsum == 0:
result[coordinates] = 0.0
else:
result[coordinates] = summa / wsum
return result
def onnx_linear_interpolation5D(self, input_data):
rank = len(self.input_shape)
assert rank in [3, 5], "mode 'linear_onnx' supports only 3D or 5D tensors"
assert set(self.axes) == {2, 3, 4} or set(self.axes) == {0, 1, 2}, \
"mode 'linear_onnx' supports only case when axes = {2, 3, 4} or axes = {0, 1, 2}"
result = np.zeros(self.output_shape)
if rank == 3:
reshaped_data = np.reshape(input_data, (1, 1, self.input_shape[0], self.input_shape[1], self.input_shape[2]))
result = np.reshape(result, (1, 1, self.output_shape[0], self.output_shape[1], self.output_shape[2]))
else:
reshaped_data = input_data
input_shape = np.array(reshaped_data.shape).astype(np.int64)
output_shape = np.array(result.shape).astype(np.int64)
batch_size = input_shape[0];
num_channels = input_shape[1];
input_depth = input_shape[2];
@ -410,31 +410,31 @@ Calculations are performed according to the following rules.
output_depth = output_shape[2];
output_height = output_shape[3];
output_width = output_shape[4];
depth_scale = self.scales[0];
height_scale = self.scales[1];
width_scale = self.scales[2];
z_original = np.zeros(output_depth).astype(np.float)
y_original = np.zeros(output_height).astype(np.float)
x_original = np.zeros(output_width).astype(np.float)
in_z1 = np.zeros(output_depth).astype(np.int64)
in_z2 = np.zeros(output_depth).astype(np.int64)
in_y1 = np.zeros(output_height).astype(np.int64)
in_y2 = np.zeros(output_height).astype(np.int64)
in_x1 = np.zeros(output_width).astype(np.int64)
in_x2 = np.zeros(output_width).astype(np.int64)
dz1 = np.zeros(output_depth).astype(np.float)
dz2 = np.zeros(output_depth).astype(np.float)
dy1 = np.zeros(output_height).astype(np.float)
dy2 = np.zeros(output_height).astype(np.float)
dx1 = np.zeros(output_width).astype(np.float)
dx2 = np.zeros(output_width).astype(np.float)
for z in range(0, output_depth):
in_z = self.get_original_coordinate(z, depth_scale, output_depth, input_depth)
z_original[z] = in_z
@ -443,11 +443,11 @@ Calculations are performed according to the following rules.
in_z2[z] = min(in_z1[z] + 1, input_depth - 1)
dz1[z] = abs(in_z - in_z1[z])
dz2[z] = abs(in_z - in_z2[z])
if in_z1[z] == in_z2[z]:
dz1[z] = 0.5
dz2[z] = 0.5
for y in range(0, output_height):
in_y = self.get_original_coordinate(y, height_scale, output_height, input_height)
y_original[y] = in_y
@ -456,19 +456,19 @@ Calculations are performed according to the following rules.
in_y2[y] = min(in_y1[y] + 1, input_height - 1)
dy1[y] = abs(in_y - in_y1[y])
dy2[y] = abs(in_y - in_y2[y])
if in_y1[y] == in_y2[y]:
dy1[y] = 0.5
dy2[y] = 0.5
for x in range(0, output_width):
in_x = self.get_original_coordinate(x, width_scale, output_width, input_width);
x_original[x] = in_x
in_x = max(0.0, min(in_x, input_width - 1));
in_x1[x] = min(in_x, input_width - 1);
in_x2[x] = min(in_x1[x] + 1, input_width - 1);
dx1[x] = abs(in_x - in_x1[x]);
dx2[x] = abs(in_x - in_x2[x]);
if in_x1[x] == in_x2[x]:
@ -487,33 +487,33 @@ Calculations are performed according to the following rules.
x212 = reshaped_data[n, c, in_z2[z], in_y1[y], in_x2[x]]
x122 = reshaped_data[n, c, in_z2[z], in_y2[y], in_x1[x]]
x222 = reshaped_data[n, c, in_z2[z], in_y2[y], in_x2[x]]
temp = dx2[x] * dy2[y] * dz2[z] * x111 + dx1[x] * dy2[y] * dz2[z] * x211
temp += dx2[x] * dy1[y] * dz2[z] * x121 + dx1[x] * dy1[y] * dz2[z] * x221
temp += dx2[x] * dy2[y] * dz1[z] * x112 + dx1[x] * dy2[y] * dz1[z] * x212
temp += dx2[x] * dy1[y] * dz1[z] * x122 + dx1[x] * dy1[y] * dz1[z] * x222
result[n, c, z, y, x] = temp
return np.reshape(result, self.output_shape)
def onnx_linear_interpolation4D(self, input_data):
rank = len(self.input_shape)
assert rank in [2, 4], "mode 'linear_onnx' supports only 2D or 4D tensors"
assert set(self.axes) == {2, 3} or set(self.axes) == {0, 1}, \
"mode 'linear_onnx' supports only case when axes = {2, 3} or axes = {0, 1}"
result = np.zeros(self.output_shape)
if rank == 2:
reshaped_data = np.reshape(input_data, (1, 1, self.input_shape[0], self.input_shape[1]))
result = np.reshape(result, (1, 1, self.output_shape[0], self.output_shape[1]))
else:
reshaped_data = input_data
input_shape = np.array(reshaped_data.shape).astype(np.int64)
output_shape = np.array(result.shape).astype(np.int64)
output_height = output_shape[2]
output_width = output_shape[3]
input_height = input_shape[2]
@ -522,21 +522,21 @@ Calculations are performed according to the following rules.
width_scale = self.scales[1]
batch_size = input_shape[0]
num_channels = input_shape[1]
y_original = np.zeros(output_height).astype(np.float)
x_original = np.zeros(output_width).astype(np.float)
in_y1 = np.zeros(output_height).astype(np.int64)
in_y2 = np.zeros(output_height).astype(np.int64)
in_x1 = np.zeros(output_width).astype(np.int64)
in_x2 = np.zeros(output_width).astype(np.int64)
dy1 = np.zeros(output_height).astype(np.float)
dy2 = np.zeros(output_height).astype(np.float)
dx1 = np.zeros(output_width).astype(np.float)
dx2 = np.zeros(output_width).astype(np.float)
for y in range(0, output_height):
in_y = self.get_original_coordinate(y, height_scale, output_height, input_height)
y_original[y] = in_y
@ -545,25 +545,25 @@ Calculations are performed according to the following rules.
in_y2[y] = min(in_y1[y] + 1, input_height - 1)
dy1[y] = abs(in_y - in_y1[y])
dy2[y] = abs(in_y - in_y2[y])
if in_y1[y] == in_y2[y]:
dy1[y] = 0.5
dy2[y] = 0.5
for x in range(0, output_width):
in_x = self.get_original_coordinate(x, width_scale, output_width, input_width);
x_original[x] = in_x
in_x = max(0.0, min(in_x, input_width - 1));
in_x1[x] = min(in_x, input_width - 1);
in_x2[x] = min(in_x1[x] + 1, input_width - 1);
dx1[x] = abs(in_x - in_x1[x]);
dx2[x] = abs(in_x - in_x2[x]);
if in_x1[x] == in_x2[x]:
dx1[x] = 0.5
dx2[x] = 0.5
for n in range(0, batch_size):
for c in range(0, num_channels):
for y in range(0, output_height):
@ -574,21 +574,21 @@ Calculations are performed according to the following rules.
x22 = reshaped_data[n, c, in_y2[y], in_x2[x]]
temp = dx2[x] * dy2[y] * x11 + dx1[x] * dy2[y] * x21 + dx2[x] * dy1[y] * x12 + dx1[x] * dy1[y] * x22
result[n, c, y, x] = temp
return np.reshape(result, self.output_shape)
def onnx_linear_interpolation(self, input_data):
rank = len(self.input_shape)
assert rank in [2, 3, 4, 5], "mode 'linear_onnx' supports only 2D, 3D, 4D, or 5D tensors"
if rank in [2, 4]:
self.onnx_linear_interpolation4D(input_data)
else:
self.onnx_linear_interpolation5D(input_data)
def nearest_interpolation(self, input_data):
result = np.zeros(self.output_shape)
num_of_axes = len(self.axes)
for coordinates in np.ndindex(tuple(self.output_shape)):
input_coords = np.array(coordinates, dtype=np.int64)
@ -597,7 +597,7 @@ Calculations are performed according to the following rules.
nearest_pixel = self.get_nearest_pixel(in_coord, scale < 1)
input_coords[axis] = max(0, min(nearest_pixel, self.input_shape[axis] - 1))
result[coordinates] = input_data[tuple(input_coords)]
return result
@ -617,13 +617,13 @@ Calculations are performed according to the following rules.
<dim>80</dim>
</port>
<port id="1">
<dim>2</dim>  < !--The values in this input are [24, 160] -->
<dim>2</dim>  <!--The values in this input are [24, 160] -->
</port>
<port id="2">
<dim>2</dim>  < !--The values in this input are [0.5, 2.0] -->
<dim>2</dim>  <!--The values in this input are [0.5, 2.0] -->
</port>
<port id="3">
<dim>2</dim>  < !--The values in this input are [2, 3] (axes). -->
<dim>2</dim>  <!--The values in this input are [2, 3] (axes). -->
</port>
</input>
<output>

View File

@ -5,7 +5,7 @@ NV12toBGR
.. meta::
:description: Learn about NV12toBGR-8 - an image processing operation, which
:description: Learn about NV12toBGR-8 - an image processing operation, which
can be performed to convert an image from NV12 to BGR format.
**Versioned name**: *NV12toBGR-8*
@ -70,13 +70,13 @@ Same as specified for :doc:`NV12toRGB <openvino_docs_ops_image_NV12toRGB_8>` ope
<layer ... type="NV12toBGR">
<input>
<port id="0"> < !-- Y plane -->
<port id="0"> <!-- Y plane -->
<dim>1</dim>
<dim>480</dim>
<dim>640</dim>
<dim>1</dim>
</port>
<port id="1"> < !-- UV plane -->
<port id="1"> <!-- UV plane -->
<dim>1</dim>
<dim>240</dim>
<dim>320</dim>

View File

@ -5,7 +5,7 @@ NV12toRGB
.. meta::
:description: Learn about NV12toRGB-8 - an image processing operation, which
:description: Learn about NV12toRGB-8 - an image processing operation, which
can be performed to convert an image from NV12 to RGB format.
**Versioned name**: *NV12toRGB-8*
@ -102,13 +102,13 @@ Input NV12 image tensor shall have ``NHWC (also known as NYXC)`` layout and can
<layer ... type="NV12toRGB">
<input>
<port id="0"> < !-- Y plane -->
<port id="0"> <!-- Y plane -->
<dim>1</dim>
<dim>480</dim>
<dim>640</dim>
<dim>1</dim>
</port>
<port id="1"> < !-- UV plane -->
<port id="1"> <!-- UV plane -->
<dim>1</dim>
<dim>240</dim>
<dim>320</dim>

View File

@ -0,0 +1,66 @@
.. {#openvino_docs_ops_infrastructure_Assign_6}
Assign
======
.. meta::
:description: Learn about Assign-6 - an infrastructure operation, which
can be performed on a single input tensor to set a value to variable_id.
**Versioned name**: *Assign-6*
**Category**: *Infrastructure*
**Short description**: *Assign* sets an input value to the ``variable_id`` variable.
**Detailed description**:
ReadValue, Assign, and Variable define a coherent mechanism for reading, writing and
storing a memory buffer between inference calls. More details can be found on the
:doc:`StateAPI<openvino_docs_OV_UG_stateful_models_intro>` documentation page.
*Assign* sets an input value to the ``variable_id`` variable. This value will be read
by the *ReadValue* operation on the next inference call if it has not been reset.
The operation checks if the shape and type specified in ``variable_id`` extend (relax)
the shape and type inferred from the 1st input. If not, it returns an error. For example,
if the type in the variable is specified as dynamic, it means that any type for 1st
input is allowed but if it is specified as f32, only f32 type is allowed.
Only one pair of ReadValue and Assign operations is expected for each Variable in the model.
**Attributes**:
* *variable_id*
* **Description**: identifier of the variable to be updated
* **Range of values**: any non-empty string
* **Type**: string
* **Required**: *yes*
**Inputs**
* **1**: ``new_value`` - input tensor of any supported type. **Required.**
**Outputs**
* **1**: tensor with the same shape and type as ``new_value``.
**Example**
.. code-block:: xml
:force:
<layer ... type="Assign" ...>
<data variable_id="lstm_state_1"/>
<input>
<port id="0">
<dim>1</dim>
<dim>3</dim>
<dim>224</dim>
<dim>224</dim>
</port>
</input>
</layer>

View File

@ -0,0 +1,96 @@
.. {#openvino_docs_ops_infrastructure_ReadValue_6}
ReadValue
=========
.. meta::
:description: Learn about ReadValue-6 - an infrastructure operation, which
can be performed on a single input tensor or without input tensors
to return the value of variable_id.
**Versioned name**: *ReadValue-6*
**Category**: *Infrastructure*
**Short description**: *ReadValue* returns value of the ``variable_id`` variable.
**Detailed description**:
*ReadValue*, *Assign*, and *Variable* define a coherent mechanism for reading, writing,
and storing some memory buffer between inference calls. More details can be found on the
:doc:`StateAPI<openvino_docs_OV_UG_stateful_models_intro>` documentation page.
If the 1st input is provided and this is the first inference or reset has been called,
*ReadValue* returns the value from the 1st input.
If the 1st input is not provided and this is the first inference or reset has been called,
*ReadValue* returns the tensor with the ``variable_shape`` and ``variable_type`` and zero values.
In all other cases *ReadValue* returns the value from the corresponding ``variable_id`` variable.
If the 1st input has been provided, the operation checks if ``variable_shape`` and ``variable_type``
extend (relax) the shape and type inferred from the 1st input. If not, it returns an error.
For example, if ``variable_type`` is specified as dynamic, it means that any type for 1st input
is allowed but if it is specified as f32, only f32 type is allowed.
Only one pair of ReadValue and Assign operations is expected for each Variable in the model.
**Attributes**:
* *variable_id*
* **Description**: identifier of the variable to be read.
* **Range of values**: any non-empty string
* **Type**: string
* **Required**: *yes*
* *variable_type*
* **Description**: the type of the variable
* **Range of values**: : u1, u4, u8, u16, u32, u64, i4, i8, i16, i32, i64, f16, f32, boolean, bf16
* **Type**: ``string``
* **Required**: *yes*
* *variable_shape*
* **Description**: the shape of the variable
* **Range of values**: list of integers, empty list is allowed, which means 0D or scalar tensor
* **Type**: ``int[]``
* **Required**: *yes*
**Inputs**
* **1**: ``init_value`` - input tensor whose values are used in the first inference or after a reset call. **Optional.**
**Outputs**
* **1**: tensor with the same shape and type as specified in *variable_type*, *variable_shape*.
**Example**
.. code-block:: xml
:force:
<layer ... type="ReadValue" ...>
<data variable_id="lstm_state_1" variable_type="f32" variable_shape="1,3,224,224"/>
<input>
<port id="0">
<dim>1</dim>
<dim>3</dim>
<dim>224</dim>
<dim>224</dim>
</port>
</input>
<output>
<port id="1">
<dim>1</dim>
<dim>3</dim>
<dim>224</dim>
<dim>224</dim>
</port>
</output>
</layer>

View File

@ -5,7 +5,7 @@ BatchToSpace
.. meta::
:description: Learn about BatchToSpace-2 - a data movement operation,
:description: Learn about BatchToSpace-2 - a data movement operation,
which can be performed on four required input tensors.
**Versioned name**: *BatchToSpace-2*
@ -21,25 +21,25 @@ BatchToSpace
1. Reshape ``data`` input to produce a tensor of shape :math:`[B_1, \dots, B_{N - 1}, \frac{batch}{\left(B_1 \times \dots \times B_{N - 1}\right)}, D_1, D_2, \dots, D_{N - 1}]`
.. math::
x^{\prime} = reshape(data, [B_1, \dots, B_{N - 1}, \frac{batch}{\left(B_1 \times \dots \times B_{N - 1}\right)}, D_1, D_2, \dots, D_{N - 1}])
2. Permute dimensions of :math:`x^{\prime}` to produce a tensor of shape :math:`[\frac{batch}{\left(B_1 \times \dots \times B_{N - 1}\right)}, D_1, B_1, D_2, B_2, \dots, D_{N-1}, B_{N - 1}]`
.. math::
x^{\prime\prime} = transpose(x', [N, N + 1, 0, N + 2, 1, \dots, N + N - 1, N - 1])
3. Reshape :math:`x^{\prime\prime}` to produce a tensor of shape :math:`[\frac{batch}{\left(B_1 \times \dots \times B_{N - 1}\right)}, D_1 \times B_1, D_2 \times B_2, \dots, D_{N - 1} \times B_{N - 1}]`
.. math::
x^{\prime\prime\prime} = reshape(x^{\prime\prime}, [\frac{batch}{\left(B_1 \times \dots \times B_{N - 1}\right)}, D_1 \times B_1, D_2 \times B_2, \dots, D_{N - 1} \times B_{N - 1}])
4. Crop the start and end of spatial dimensions of :math:`x^{\prime\prime\prime}` according to ``crops_begin`` and ``crops_end`` inputs to produce the output :math:`y` of shape:
.. math::
\left[\frac{batch}{\left(B_1 \times \dots \times B_{N - 1}\right)}, crop(D_1 \times B_1, CB_1, CE_1), crop(D_2 \times B_2, CB_2, CE_2), \dots , crop(D_{N - 1} \times B_{N - 1}, CB_{N - 1}, CE_{N - 1})\right]
Where
@ -80,27 +80,27 @@ Example: 2D input tensor ``data``
.. code-block:: xml
:force:
<layer type="BatchToSpace" ...>
<input>
<port id="0"> < !-- data -->
<dim>10</dim> < !-- batch -->
<dim>2</dim> < !-- spatial dimension 1 -->
<port id="0"> <!-- data -->
<dim>10</dim> <!-- batch -->
<dim>2</dim> <!-- spatial dimension 1 -->
</port>
<port id="1"> < !-- block_shape value: [1, 5] -->
<port id="1"> <!-- block_shape value: [1, 5] -->
<dim>2</dim>
</port>
<port id="2"> < !-- crops_begin value: [0, 2] -->
<port id="2"> <!-- crops_begin value: [0, 2] -->
<dim>2</dim>
</port>
<port id="3"> < !-- crops_end value: [0, 0] -->
<port id="3"> <!-- crops_end value: [0, 0] -->
<dim>2</dim>
</port>
</input>
<output>
<port id="3">
<dim>2</dim> < !-- data.shape[0] / (block_shape.shape[0] * block_shape.shape[1]) -->
<dim>8</dim> < !-- data.shape[1] * block_shape.shape[1] - crops_begin[1] - crops_end[1]-->
<dim>2</dim> <!-- data.shape[0] / (block_shape.shape[0] * block_shape.shape[1]) -->
<dim>8</dim> <!-- data.shape[1] * block_shape.shape[1] - crops_begin[1] - crops_end[1]-->
</port>
</output>
</layer>
@ -109,33 +109,33 @@ Example: 5D input tensor ``data``
.. code-block:: xml
:force:
<layer type="BatchToSpace" ...>
<input>
<port id="0"> < !-- data -->
<dim>48</dim> < !-- batch -->
<dim>3</dim> < !-- spatial dimension 1 -->
<dim>3</dim> < !-- spatial dimension 2 -->
<dim>1</dim> < !-- spatial dimension 3 -->
<dim>3</dim> < !-- spatial dimension 4 -->
<port id="0"> <!-- data -->
<dim>48</dim> <!-- batch -->
<dim>3</dim> <!-- spatial dimension 1 -->
<dim>3</dim> <!-- spatial dimension 2 -->
<dim>1</dim> <!-- spatial dimension 3 -->
<dim>3</dim> <!-- spatial dimension 4 -->
</port>
<port id="1"> < !-- block_shape value: [1, 2, 4, 3, 1] -->
<port id="1"> <!-- block_shape value: [1, 2, 4, 3, 1] -->
<dim>5</dim>
</port>
<port id="2"> < !-- crops_begin value: [0, 0, 1, 0, 0] -->
<port id="2"> <!-- crops_begin value: [0, 0, 1, 0, 0] -->
<dim>5</dim>
</port>
<port id="3"> < !-- crops_end value: [0, 0, 1, 0, 0] -->
<port id="3"> <!-- crops_end value: [0, 0, 1, 0, 0] -->
<dim>5</dim>
</port>
</input>
<output>
<port id="3">
<dim>2</dim> < !-- data.shape[0] / (block_shape.shape[0] * block_shape.shape[1] * ... * block_shape.shape[4]) -->
<dim>6</dim> < !-- data.shape[1] * block_shape.shape[1] - crops_begin[1] - crops_end[1]-->
<dim>10</dim> < !-- data.shape[2] * block_shape.shape[2] - crops_begin[2] - crops_end[2] -->
<dim>3</dim> < !-- data.shape[3] * block_shape.shape[3] - crops_begin[3] - crops_end[3] -->
<dim>3</dim> < !-- data.shape[4] * block_shape.shape[4] - crops_begin[4] - crops_end[4] -->
<dim>2</dim> <!-- data.shape[0] / (block_shape.shape[0] * block_shape.shape[1] * ... * block_shape.shape[4]) -->
<dim>6</dim> <!-- data.shape[1] * block_shape.shape[1] - crops_begin[1] - crops_end[1]-->
<dim>10</dim> <!-- data.shape[2] * block_shape.shape[2] - crops_begin[2] - crops_end[2] -->
<dim>3</dim> <!-- data.shape[3] * block_shape.shape[3] - crops_begin[3] - crops_end[3] -->
<dim>3</dim> <!-- data.shape[4] * block_shape.shape[4] - crops_begin[4] - crops_end[4] -->
</port>
</output>
</layer>

View File

@ -5,7 +5,7 @@ Broadcast
.. meta::
:description: Learn about Broadcast-1 - a data movement operation,
:description: Learn about Broadcast-1 - a data movement operation,
which can be performed on two required and one optional input tensor.
**Versioned name**: *Broadcast-1*
@ -53,7 +53,7 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
.. code-block:: xml
:force:
<layer ... type="Broadcast" ...>
<data mode="numpy"/>
<input>
@ -63,9 +63,9 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
<dim>1</dim>
</port>
<port id="1">
<dim>4</dim> < !--The tensor contains 4 elements: [1, 16, 50, 50] -->
<dim>4</dim> <!--The tensor contains 4 elements: [1, 16, 50, 50] -->
</port>
< !-- the 3rd input shouldn't be provided with mode="numpy" -->
<!-- the 3rd input shouldn't be provided with mode="numpy" -->
</input>
<output>
<port id="2">
@ -76,7 +76,7 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
</port>
</output>
</layer>
<layer ... type="Broadcast" ...>
<data mode="explicit"/>
<input>
@ -84,10 +84,10 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
<dim>16</dim>
</port>
<port id="1">
<dim>4</dim> < !--The tensor contains 4 elements: [1, 16, 50, 50] -->
<dim>4</dim> <!--The tensor contains 4 elements: [1, 16, 50, 50] -->
</port>
<port id="1">
<dim>1</dim> < !--The tensor contains 1 elements: [1] -->
<dim>1</dim> <!--The tensor contains 1 elements: [1] -->
</port>
</input>
<output>
@ -99,7 +99,7 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
</port>
</output>
</layer>
<layer ... type="Broadcast" ...>
<data mode="explicit"/>
<input>
@ -108,10 +108,10 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
<dim>50</dim>
</port>
<port id="1">
<dim>4</dim> < !--The tensor contains 4 elements: [1, 50, 50, 16] -->
<dim>4</dim> <!--The tensor contains 4 elements: [1, 50, 50, 16] -->
</port>
<port id="1">
<dim>2</dim> < !--The tensor contains 2 elements: [1, 2] -->
<dim>2</dim> <!--The tensor contains 2 elements: [1, 2] -->
</port>
</input>
<output>

View File

@ -5,7 +5,7 @@ Broadcast
.. meta::
:description: Learn about Broadcast-3 - a data movement operation,
:description: Learn about Broadcast-3 - a data movement operation,
which can be performed on two required and one optional input tensor.
**Versioned name**: *Broadcast-3*
@ -61,7 +61,7 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
.. code-block:: xml
:force:
<layer ... type="Broadcast" ...>
<data mode="numpy"/>
<input>
@ -71,9 +71,9 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
<dim>1</dim>
</port>
<port id="1">
<dim>4</dim> < !--The tensor contains 4 elements: [1, 16, 50, 50] -->
<dim>4</dim> <!--The tensor contains 4 elements: [1, 16, 50, 50] -->
</port>
< !-- the 3rd input shouldn't be provided with mode="numpy" -->
<!-- the 3rd input shouldn't be provided with mode="numpy" -->
</input>
<output>
<port id="2">
@ -84,7 +84,7 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
</port>
</output>
</layer>
<layer ... type="Broadcast" ...>
<data mode="explicit"/>
<input>
@ -92,10 +92,10 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
<dim>16</dim>
</port>
<port id="1">
<dim>4</dim> < !--The tensor contains 4 elements: [1, 16, 50, 50] -->
<dim>4</dim> <!--The tensor contains 4 elements: [1, 16, 50, 50] -->
</port>
<port id="1">
<dim>1</dim> < !--The tensor contains 1 elements: [1] -->
<dim>1</dim> <!--The tensor contains 1 elements: [1] -->
</port>
</input>
<output>
@ -107,7 +107,7 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
</port>
</output>
</layer>
<layer ... type="Broadcast" ...>
<data mode="explicit"/>
<input>
@ -116,10 +116,10 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
<dim>50</dim>
</port>
<port id="1">
<dim>4</dim> < !--The tensor contains 4 elements: [1, 50, 50, 16] -->
<dim>4</dim> <!--The tensor contains 4 elements: [1, 50, 50, 16] -->
</port>
<port id="1">
<dim>2</dim> < !--The tensor contains 2 elements: [1, 2] -->
<dim>2</dim> <!--The tensor contains 2 elements: [1, 2] -->
</port>
</input>
<output>
@ -131,7 +131,7 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
</port>
</output>
</layer>
<layer ... type="Broadcast" ...>
<data mode="bidirectional"/>
<input>
@ -141,9 +141,9 @@ For example, ``axes_mapping = [1]`` enables broadcasting of a tensor with shape
<dim>1</dim>
</port>
<port id="1">
<dim>4</dim> < !--The tensor contains 4 elements: [1, 1, 50, 50] -->
<dim>4</dim> <!--The tensor contains 4 elements: [1, 1, 50, 50] -->
</port>
< !-- the 3rd input shouldn't be provided with mode="bidirectional" -->
<!-- the 3rd input shouldn't be provided with mode="bidirectional" -->
</input>
<output>
<port id="2">

View File

@ -5,7 +5,7 @@ Concat
.. meta::
:description: Learn about Concat-1 - a data movement operation,
:description: Learn about Concat-1 - a data movement operation,
which can be performed on arbitrary number of input tensors.
**Versioned name**: *Concat-1*
@ -39,25 +39,25 @@ Concat
.. code-block:: xml
:force:
<layer id="1" type="Concat">
<data axis="1" />
<input>
<port id="0">
<dim>1</dim>
<dim>8</dim> < !-- axis for concatenation -->
<dim>8</dim> <!-- axis for concatenation -->
<dim>50</dim>
<dim>50</dim>
</port>
<port id="1">
<dim>1</dim>
<dim>16</dim> < !-- axis for concatenation -->
<dim>16</dim> <!-- axis for concatenation -->
<dim>50</dim>
<dim>50</dim>
</port>
<port id="2">
<dim>1</dim>
<dim>32</dim> < !-- axis for concatenation -->
<dim>32</dim> <!-- axis for concatenation -->
<dim>50</dim>
<dim>50</dim>
</port>
@ -65,7 +65,7 @@ Concat
<output>
<port id="0">
<dim>1</dim>
<dim>56</dim> < !-- concatenated axis: 8 + 16 + 32 = 48 -->
<dim>56</dim> <!-- concatenated axis: 8 + 16 + 32 = 48 -->
<dim>50</dim>
<dim>50</dim>
</port>
@ -75,25 +75,25 @@ Concat
.. code-block:: xml
:force:
<layer id="1" type="Concat">
<data axis="-3" />
<input>
<port id="0">
<dim>1</dim>
<dim>8</dim> < !-- axis for concatenation -->
<dim>8</dim> <!-- axis for concatenation -->
<dim>50</dim>
<dim>50</dim>
</port>
<port id="1">
<dim>1</dim>
<dim>16</dim> < !-- axis for concatenation -->
<dim>16</dim> <!-- axis for concatenation -->
<dim>50</dim>
<dim>50</dim>
</port>
<port id="2">
<dim>1</dim>
<dim>32</dim> < !-- axis for concatenation -->
<dim>32</dim> <!-- axis for concatenation -->
<dim>50</dim>
<dim>50</dim>
</port>
@ -101,7 +101,7 @@ Concat
<output>
<port id="0">
<dim>1</dim>
<dim>56</dim> < !-- concatenated axis: 8 + 16 + 32 = 48 -->
<dim>56</dim> <!-- concatenated axis: 8 + 16 + 32 = 48 -->
<dim>50</dim>
<dim>50</dim>
</port>

View File

@ -5,7 +5,7 @@ DepthToSpace
.. meta::
:description: Learn about DepthToSpace-1 - a data movement operation,
:description: Learn about DepthToSpace-1 - a data movement operation,
which can be performed on a single input tensor.
**Versioned name**: *DepthToSpace-1*
@ -21,7 +21,7 @@ DepthToSpace
The operation is equivalent to the following transformation of the input tensor ``data`` with ``K`` spatial dimensions of shape ``[N, C, D1, D2, ..., DK]`` to *Y* output tensor. If ``mode = blocks_first``:
.. code-block:: cpp
x' = reshape(data, [N, block_size, block_size, ..., block_size, C / (block_size ^ K), D1, D2, ..., DK])
x'' = transpose(x', [0, K + 1, K + 2, 1, K + 3, 2, K + 4, 3, ..., K + (K + 1), K])
y = reshape(x'', [N, C / (block_size ^ K), D1 * block_size, D2 * block_size, D3 * block_size, ..., DK * block_size])
@ -29,7 +29,7 @@ The operation is equivalent to the following transformation of the input tensor
If ``mode = depth_first``:
.. code-block:: cpp
x' = reshape(data, [N, C / (block_size ^ K), block_size, block_size, ..., block_size, D1, D2, ..., DK])
x'' = transpose(x', [0, 1, K + 2, 2, K + 3, 3, K + 4, 4, ..., K + (K + 1), K + 1])
y = reshape(x'', [N, C / (block_size ^ K), D1 * block_size, D2 * block_size, D3 * block_size, ..., DK * block_size])
@ -70,7 +70,7 @@ If ``mode = depth_first``:
.. code-block:: xml
:force:
<layer type="DepthToSpace" ...>
<data block_size="2" mode="blocks_first"/>
<input>
@ -83,10 +83,10 @@ If ``mode = depth_first``:
</input>
<output>
<port id="1">
<dim>5</dim> < !-- data.shape[0] -->
<dim>7</dim> < !-- data.shape[1] / (block_size ^ 2) -->
<dim>4</dim> < !-- data.shape[2] * block_size -->
<dim>6</dim> < !-- data.shape[3] * block_size -->
<dim>5</dim> <!-- data.shape[0] -->
<dim>7</dim> <!-- data.shape[1] / (block_size ^ 2) -->
<dim>4</dim> <!-- data.shape[2] * block_size -->
<dim>6</dim> <!-- data.shape[3] * block_size -->
</port>
</output>
</layer>

View File

@ -5,14 +5,14 @@ Gather
.. meta::
:description: Learn about Gather-1 - a data movement operation,
:description: Learn about Gather-1 - a data movement operation,
which can be performed on three required input tensors.
**Versioned name:** *Gather-1*
**Category:** *Data movement*
**Short description:** *Gather* operation takes slices of data in the first input tensor according
**Short description:** *Gather* operation takes slices of data in the first input tensor according
to the indices specified in the second input tensor and axis from the third input.
**Detailed description**
@ -30,13 +30,13 @@ Where ``axis`` is the value from the third input.
* **1**: Tensor with arbitrary data. **Required.**
* **2**: Tensor with indices to gather. The values for indices are in the range ``[0, input1[axis] - 1]``. **Required.**
* **3**: Scalar or 1D tensor *axis* is a dimension index to gather data from. For example, *axis* equal
to 1 means that gathering is performed over the first dimension. Negative value means reverse indexing.
* **3**: Scalar or 1D tensor *axis* is a dimension index to gather data from. For example, *axis* equal
to 1 means that gathering is performed over the first dimension. Negative value means reverse indexing.
Allowed values are from ``[-len(input1.shape), len(input1.shape) - 1]``. **Required.**
**Outputs**
* **1**: The resulting tensor that consists of elements from the first input tensor gathered by indices
* **1**: The resulting tensor that consists of elements from the first input tensor gathered by indices
from the second input tensor. Shape of the tensor is ``[input1.shape[:axis], input2.shape, input1.shape[axis + 1:]]``
**Example**
@ -58,17 +58,17 @@ Where ``axis`` is the value from the third input.
<dim>20</dim>
<dim>28</dim>
</port>
<port id="2"/> < !-- axis = 1 -->
<port id="2"/> <!-- axis = 1 -->
</input>
<output>
<port id="2">
<dim>6</dim> < !-- embedded dimension from the 1st input -->
<dim>15</dim> < !-- embedded dimension from the 2nd input -->
<dim>4</dim> < !-- embedded dimension from the 2nd input -->
<dim>20</dim> < !-- embedded dimension from the 2nd input -->
<dim>28</dim> < !-- embedded dimension from the 2nd input -->
<dim>10</dim> < !-- embedded dimension from the 1st input -->
<dim>24</dim> < !-- embedded dimension from the 1st input -->
<dim>6</dim> <!-- embedded dimension from the 1st input -->
<dim>15</dim> <!-- embedded dimension from the 2nd input -->
<dim>4</dim> <!-- embedded dimension from the 2nd input -->
<dim>20</dim> <!-- embedded dimension from the 2nd input -->
<dim>28</dim> <!-- embedded dimension from the 2nd input -->
<dim>10</dim> <!-- embedded dimension from the 1st input -->
<dim>24</dim> <!-- embedded dimension from the 1st input -->
</port>
</output>
</layer>

View File

@ -5,7 +5,7 @@ Gather
.. meta::
:description: Learn about Gather-7 - a data movement operation,
:description: Learn about Gather-7 - a data movement operation,
which can be performed on three required input tensors.
**Versioned name**: *Gather-7*
@ -29,12 +29,12 @@ the number of batch dimensions. ``N`` and ``M`` are numbers of dimensions of ``d
**Attributes**:
* *batch_dims*
* **Description**: *batch_dims* (also denoted as ``b``) is a leading number of dimensions of ``data``
tensor and ``indices`` representing the batches, and *Gather* starts to gather from the ``b``
dimension. It requires the first ``b`` dimensions in `data` and `indices` tensors to be equal.
* **Description**: *batch_dims* (also denoted as ``b``) is a leading number of dimensions of ``data``
tensor and ``indices`` representing the batches, and *Gather* starts to gather from the ``b``
dimension. It requires the first ``b`` dimensions in `data` and `indices` tensors to be equal.
If ``batch_dims`` is less than zero, the normalized value is used ``batch_dims = indices.rank + batch_dims``.
* **Range of values**: ``[-min(data.rank, indices.rank); min(data.rank, indices.rank)]`` and
* **Range of values**: ``[-min(data.rank, indices.rank); min(data.rank, indices.rank)]`` and
``batch_dims' <= axis'``. Where ``batch_dims'`` and ``axis'`` stand for normalized ``batch_dims`` and ``axis`` values.
* **Type**: *T_AXIS*
* **Default value**: 0
@ -46,7 +46,7 @@ Example 1 with default *batch_dims* value:
batch_dims = 0
axis = 0
indices = [0, 0, 4]
data = [1, 2, 3, 4, 5]
output = [1, 1, 5]
@ -58,15 +58,15 @@ Example 2 with non-default *batch_dims* value:
batch_dims = 1
axis = 1
indices = [[0, 0, 4], <-- this is applied to the first batch
[4, 0, 0]] <-- this is applied to the second batch
indices_shape = (2, 3)
data = [[1, 2, 3, 4, 5], <-- the first batch
[6, 7, 8, 9, 10]] <-- the second batch
data_shape = (2, 5)
output = [[ 1, 1, 5],
[10, 6, 6]]
output_shape = (2, 3)
@ -78,24 +78,24 @@ Example 3 with non-default *batch_dims* value:
batch_dims = 2
axis = 2
indices = [[[0, 0, 4], <-- this is applied to the first batch, index = (0, 0)
[4, 0, 0]], <-- this is applied to the second batch, index = (0, 1)
[[1, 2, 4], <-- this is applied to the third batch, index = (1, 0)
[4, 3, 2]]] <-- this is applied to the fourth batch, index = (1, 1)
indices_shape = (2, 2, 3)
data = [[[1, 2, 3, 4, 5], <-- the first batch, index = (0, 0)
[6, 7, 8, 9, 10]], <-- the second batch, index = (0, 1)
[[11, 12, 13, 14, 15], <-- the third batch, index = (1, 0)
[16, 17, 18, 19, 20]]] <-- the fourth batch, index = (1, 1)
data_shape = (2, 2, 5)
output = [[[ 1, 1, 5],
[10, 6, 6]],
[[12, 13, 15],
[20, 19, 18]]]
output_shape = (2, 2, 3)
@ -106,28 +106,28 @@ Example 4 with *axis* > *batch_dims*:
batch_dims = 1
axis = 2
indices = [[1, 2, 4], <-- this is applied to the first batch
[4, 3, 2]] <-- this is applied to the second batch
indices_shape = (2, 3)
data = [[[[ 1, 2, 3, 4], <-- first batch
[ 5, 6, 7, 8],
[ 9, 10, 11, 12],
[13, 14, 15, 16],
[17, 18, 19, 20]]],
[[[21, 22, 23, 24], <-- second batch
[25, 26, 27, 28],
[29, 30, 31, 32],
[33, 34, 35, 36],
[37, 38, 39, 40]]]]
data_shape = (2, 1, 5, 4)
output = [[[[ 5, 6, 7, 8],
[ 9, 10, 11, 12],
[17, 18, 19, 20]]],
[[[37, 38, 39, 40],
[33, 34, 35, 36],
[29, 30, 31, 32]]]]
@ -140,15 +140,15 @@ Example 5 with negative *batch_dims* value:
batch_dims = -1 <-- normalized value will be indices.rank + batch_dims = 2 - 1 = 1
axis = 1
indices = [[0, 0, 4], <-- this is applied to the first batch
[4, 0, 0]] <-- this is applied to the second batch
indices_shape = (2, 3)
data = [[1, 2, 3, 4, 5], <-- the first batch
[6, 7, 8, 9, 10]] <-- the second batch
data_shape = (2, 5)
output = [[ 1, 1, 5],
[10, 6, 6]]
output_shape = (2, 3)
@ -167,7 +167,7 @@ Example 5 with negative *batch_dims* value:
**Outputs**
* **1**: The resulting tensor of type *T* that consists of elements from ``data`` tensor gathered by ``indices``.
* **1**: The resulting tensor of type *T* that consists of elements from ``data`` tensor gathered by ``indices``.
The shape of the output tensor is ``data.shape[:axis] + indices.shape[batch_dims:] + data.shape[axis + 1:]``
**Types**
@ -193,7 +193,7 @@ Example 5 with negative *batch_dims* value:
<dim>32</dim>
<dim>21</dim>
</port>
<port id="2"/> < !-- axis = 1 -->
<port id="2"/> <!-- axis = 1 -->
</input>
<output>
<port id="2">

Some files were not shown because too many files have changed in this diff Show More