Compare commits

..

111 Commits

Author SHA1 Message Date
Maxim Vafin fb35278525
Change opt-125m-gptq model (#24901)
# Conflicts:
#	tests/model_hub_tests/torch_tests/test_hf_transformers.py

### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-06-10 11:08:09 +02:00
Xuejun Zhai d9ef8441b1
Port 24606 to release 202303 (#24796) 2024-06-03 09:35:49 +04:00
Yuan Hu e9f8604430
port PR 24562 to release/2023/3 (#24670)
### Details:
 - *port PR #24562 to release 2023.3*
 - *add support for InferAPI1 and api_version*

### Tickets:
-
*[issue-24509](https://github.com/openvinotoolkit/openvino/issues/24509)*

---------

Signed-off-by: HU Yuan2 <yuan2.hu@intel.com>
Co-authored-by: Wanglei Shen <wanglei.shen@intel.com>
2024-05-28 07:37:02 +08:00
Xuejun Zhai 78b55038b5
Port PR 24018 to release/2023/3 (#24371)
### Details:
 - Port PR2418 to release 2023.3
 - *...*

### Tickets:
 - *CVS-132745*

Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
2024-05-07 15:11:35 +04:00
Ilya Lavrenov d3cb701018
Added check for conflicting DEB / RPM versions (#24275)
### Details:
 - Added check for conflicting DEB / RPM versions
2024-04-28 01:19:34 +04:00
Ilya Lavrenov 6b3f564fe8
Port openvino-telemetry support in Debian package to 2023.3 (#24257)
### Ported PRs:
- https://github.com/openvinotoolkit/openvino/pull/23917
- https://github.com/openvinotoolkit/openvino/pull/24060
- https://github.com/openvinotoolkit/openvino/pull/23816

---------

Co-authored-by: Mikhail Ryzhov <mikhail.ryzhov@intel.com>
Co-authored-by: Przemyslaw Wysocki <przemyslaw.wysocki@intel.com>
2024-04-28 00:55:35 +04:00
Xiping Yan f32c023c91
[CPU]Fix regression issue. (#23904)
### Details:
 - *Add callback fun param for selectPreferPrimitiveDescriptor*
 - *Update Snippet node selectPreferPrimitiveDescriptor algorithm.*
 - *Add test*

### Tickets:
 - *137307*

---------

Signed-off-by: xipingya <xiping.yan@intel.com>
2024-04-26 10:21:25 +02:00
Maciej Smyk 53ca00bc81
[DOCS] release policy for 2023.3 (#24185)
Added Release Policy article to docs. Updated with changes from the
https://github.com/openvinotoolkit/openvino/pull/24145
2024-04-23 18:19:37 +02:00
Maciej Smyk 0d6fadd2da
[DOCS] Fixing Tensor Names examples in Model Representation for 2023.3 (#24173)
Port from https://github.com/openvinotoolkit/openvino/pull/23482

* Fixed examples of Tensor Names,
* Added an example of ov::element::Type for both Python and C++
2024-04-22 17:20:08 +02:00
Ilya Lavrenov 28c1585d65
Port several PRs from master to 2023.3 LTS (#23461)
### Ported PRs:
- https://github.com/openvinotoolkit/openvino/pull/22928
- https://github.com/openvinotoolkit/openvino/pull/22963
- https://github.com/openvinotoolkit/openvino/pull/23345
- https://github.com/openvinotoolkit/openvino/pull/23460

---------

Co-authored-by: Andrey Babushkin <andrey.babushkin@intel.com>
2024-04-16 18:23:32 +04:00
Maksim Kutakov 15b0f99132
[CPU] fix gen_pattern::AttrAny::equal_to (#23689)
### Details:
Backport https://github.com/openvinotoolkit/openvino/pull/21890 from
master

### Tickets:
 - CVS-133621

Co-authored-by: Tingqian Li <tingqian.li@intel.com>
2024-04-16 12:35:46 +02:00
Maksim Kutakov e6fbca158a
[CPU] Fix binary PReLU spare register search (#23910)
### Details:
Fix the algorithm of assigning an auxiliary SIMD register in the binary
PReLU post op JIT injector.

OneDNN fork PR: https://github.com/openvinotoolkit/oneDNN/pull/240

### Tickets:
 - CVS-137492
2024-04-16 12:35:35 +02:00
Karol Blaszczak 89e0a839d0
[DOCS] old fix for yolo article 23.3 (#23933) 2024-04-10 12:13:49 +02:00
Tatiana Savina feaeca5191
[DOCS] Port Add MS Windows version for NPU to 23.3 (#23872)
### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-04-04 14:53:13 +02:00
Steve Yoo ddef44e96d
[GPU] Skip concat_input_order (#23787)
### Details:
- *Skip concat_input_order for an element which lies outside of the
buffer range*

### Tickets:
 - *135790*
2024-04-02 20:56:35 -07:00
Sungeun Kim c4ada32cc8
[GPU] add reorder between mvn and conv. (#23670) (#23765)
- add reorder to avoid inserting padding to mvn before ocl conv.
- mvn does not support padding.

### Tickets:
 - 136385
2024-04-03 10:01:34 +09:00
Sebastian Golebiewski 769fa5764e
[DOCS] Update code examples with imports and save_model for for 23.3 (#23684)
Updating code examples with imports for ``convert_model`` and adding an
example of how to save a converted model. This PR addresses the 120254
ticket.
Porting: https://github.com/openvinotoolkit/openvino/pull/23654
2024-03-26 13:42:30 +01:00
Andrzej Kopytko 687c648120
[DOCS] update banner for 2023.3 (#23656)
### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-03-25 13:44:49 +01:00
Andrzej Kopytko beff6958a9
[Docs] restore banner information (#23623)
### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-03-22 09:36:06 +01:00
Andrzej Kopytko 385782df37
[DOCS] Added off on click for 2023.3 (#23467)
Port for 
https://github.com/openvinotoolkit/openvino/pull/23411
2024-03-14 15:16:15 +01:00
tadamczx 655a72607f
[DOCS] Python API version fix (#23403)
CMakeLists.txt update, to be able to build right version of Python API.
2024-03-13 07:05:06 +01:00
Andrzej Kopytko beb158f6f0
Docs remove canonical tag from 2023.3 (#23375)
### Details:
 - *item1*
 - *...*

### Tickets:
 - *ticket-id*
2024-03-12 09:43:03 +01:00
Maciej Smyk 0c9ad9ede5
[DOCS] Fix of home tab title for 2023.3 (#23367)
Jira: 133084

* Fixed the home page tab title name.
* Fixed the missing line in banner
2024-03-11 11:51:35 +01:00
Wang, Yang 735a3ba80a
Update port name checking logic to 2023.3 (#23339)
Port from https://github.com/openvinotoolkit/openvino/pull/23081 to
2023.3
### Details:
 - *item1*
 - *...*

### Tickets:
 - 122932
2024-03-08 13:12:19 +04:00
Anastasia Kuporosova c4034018fd
[PyOV] Cherry-picked convert ExecutionMode enum to any (#23296) 2024-03-07 11:02:31 +04:00
Chen Peter 004a6e9cf2
Map configuration to API 2.0 properties (#23161)
### Details:
- *Create table to map some of the Inference Engine configurations to
API 2.0 C++/Python properties*
- *Provide configurations/properties header files for other
configurations/properties*

### Tickets:
 - *CVS-94044*

---------

Signed-off-by: Peter Chen <peter.chen@intel.com>
2024-03-01 11:25:44 +04:00
River Li 0671bf3161
Port PR-23100 for Windows unicode issue (#23120)
### Details:
 - solve windows unicode inconsistent issue
 - add more test case

### Tickets:
 - Closes https://github.com/openvinotoolkit/openvino/issues/23072
 - CVS-128116
2024-02-28 11:08:13 +04:00
Pawel Raasz d02cd0c143
[core] Restore OV exception create function (#23014)
### Details:
- Restore `create` member function in OV exception classes to be
compatible with 2023.2 API.

### Tickets:
 - [CVS-133319](https://jira.devtools.intel.com/browse/CVS-133319)
2024-02-26 13:21:44 +04:00
Karol Blaszczak 7fe8b03cba
[DOCS] benchmark fix table (#23062) 2024-02-23 20:19:17 +01:00
Andrzej Kopytko 381a550f74
DOCS Udpate banchmark files to 2023.3 (#23040)
Port from https://github.com/openvinotoolkit/openvino/pull/23029 to
2023.3
2024-02-23 10:07:37 +01:00
Andrzej Kopytko 51055ef9e1
[DOCS] upgrade chart js to v4 (#23020)
port to https://github.com/openvinotoolkit/openvino/pull/23008
2024-02-22 13:41:53 +01:00
River Li 6758735364
[Core] port PR-22525 and PR-22804 (#22554)
### Details:
 - Port: https://github.com/openvinotoolkit/openvino/pull/22525
 - Port: https://github.com/openvinotoolkit/openvino/pull/22804

### Tickets:
 - CVS-131048

---------

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
Co-authored-by: Vitaliy Urusovskij <vitaliy.urusovskij@intel.com>
2024-02-19 14:24:31 +04:00
Ilya Lavrenov 376428b59a
Ported to 2023.3 from master (#22899)
### Ported from master:
- https://github.com/openvinotoolkit/openvino/pull/22816
- https://github.com/openvinotoolkit/openvino/pull/22847
- https://github.com/openvinotoolkit/openvino/pull/22866

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Maksim Kutakov <maksim.kutakov@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-19 11:18:24 +04:00
Sebastian Golebiewski ee39aaaa84
[DOCS] Updating Interactive Tutorials for 23.3 (#22862)
Updating Interactive Tutorials - adding new notebooks. Porting:
https://github.com/openvinotoolkit/openvino/pull/22815
2024-02-15 15:35:33 +01:00
Ilya Lavrenov c17db5989c
Ported several PRs from master to 2023.3 LTS (#22843)
### Ported:
- https://github.com/openvinotoolkit/openvino/pull/22606
- https://github.com/openvinotoolkit/openvino/pull/22734
- https://github.com/openvinotoolkit/openvino/pull/22776
- https://github.com/openvinotoolkit/openvino/pull/22820
- https://github.com/openvinotoolkit/openvino/pull/22827

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Andrei Kashchikhin <andrey.kashchikhin@intel.com>
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-15 11:30:04 +04:00
Vishniakov Nikolai b03c36fee3
[OV JS] Port of Fix npm dependencies (#22830)
Port of:
https://github.com/openvinotoolkit/openvino/pull/22810

To fix this issue:
https://github.com/openvinotoolkit/openvino/security/dependabot/514
2024-02-14 17:39:18 +01:00
Sebastian Golebiewski 4bf8a469dc
[DOCS] Removing the note about GPU support for static models only (#22803)
Removing the note about GPU support for static models only, as GPU now
supports both type of shapes. Porting:
https://github.com/openvinotoolkit/openvino/pull/22802
2024-02-13 08:20:45 +01:00
Karol Blaszczak bed27aa64d
[DOCS] tiny fix for ovms linking (#22755) 2024-02-09 12:29:57 +01:00
Anastasia Kuporosova 493539be12
remove flake8-commas requirement (#22611)
### Details:
 - cherry-picked https://github.com/openvinotoolkit/openvino/pull/22546

### Tickets:
 - *ticket-id*
2024-02-09 12:10:03 +04:00
Karol Blaszczak 1886fc6d31
[DOCS] review for 22356 and 22410 (#22466)
relates to:
https://github.com/openvinotoolkit/openvino/pull/22410
https://github.com/openvinotoolkit/openvino/pull/22356

---------

Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
2024-02-08 18:56:53 +01:00
Pavel Esir 77a0b916ba
Keep compressed constants produced by FW #22095 (#22096)
### Details:
 - keep compressed constants produced by FW
 - cherry pick of #22095

### Tickets:
 - CVS-119139
2024-02-08 07:16:44 +04:00
Maciej Smyk f33953bb0c
[DOCS] Fix for xml comments + ScatterUpdate Inputs for 23.3 (#22624)
Port from https://github.com/openvinotoolkit/openvino/pull/22623

- Fixed xml comments throughout the docs.
- Fixed ScatterUpdate Inputs paragraph.
2024-02-06 14:06:53 +01:00
River Li af87a9c5e4
[CAPI] Port PR-22579 (#22596)
https://github.com/openvinotoolkit/openvino/pull/22579
2024-02-04 06:15:46 +01:00
Andrzej Kopytko 6683952c0a
[DOCS] Benchmark Latency as default - port (#22622)
change from master
2024-02-02 12:24:59 +01:00
Alexander Suvorov 28e80c22cf
[DOCS] Update selector tool 23.3 (#22605) 2024-02-01 18:57:34 +01:00
Ilya Lavrenov 7a0f7481cc
Ported several PRs from master to 2023.3 (#22528)
* Bump actions/cache from 3 to 4 (#22224)

Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Disable proxy for GPU plugin (#22235)

* Disable proxy for GPU

* Fix

* Ported conda-forge patch to work with latest protobuf (#22277)

* Bump sccache version (#22282)

* [GHA] Set SCCACHE_IGNORE_SERVER_IO_ERROR (#22346)

* [GHA] Change default port for sccache server to avoid collisions (#22364)

* Change Windows CC workflow but based on last successful WinCC commit (#22469)

* changed linux cc build timeout (#22561)

* Bump mozilla-actions/sccache-action from 0.0.3 to 0.0.4 (#22552)

Bumps [mozilla-actions/sccache-action](https://github.com/mozilla-actions/sccache-action) from 0.0.3 to 0.0.4.
- [Release notes](https://github.com/mozilla-actions/sccache-action/releases)
- [Commits](https://github.com/mozilla-actions/sccache-action/compare/v0.0.3...v0.0.4)

---
updated-dependencies:
- dependency-name: mozilla-actions/sccache-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [CI] [GHA] Download `ninja` from GitHub instead of installing through `choco` in Windows workflows (#22570)

* download ninja from github

* add ninja to path

* add note

* add retries

* [CI] [GHA] Add retries for APT, update deadsnakes action (#22515)

* update setup_python version, add retries for apt in Linux workflows

* account for sudo

* use sudo via sh

* bump to v5 (#22505)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Oleg Pipikin <oleg.pipikin@intel.com>
Co-authored-by: Andrey Babushkin <andrey.babushkin@intel.com>
Co-authored-by: Alina Kladieva <alina.kladieva@intel.com>
Co-authored-by: Mikhail Ryzhov <mikhail.ryzhov@intel.com>
Co-authored-by: Andrei Kashchikhin <andrey.kashchikhin@intel.com>
2024-02-01 17:37:27 +04:00
Sebastian Golebiewski 91cde333cb
[DOCS] Port docs patches for 23.3 (#22590)
* Porting 21679 - fixing typo

* Porting 22571 - specify number of jobs in build
2024-02-01 14:02:35 +04:00
Maciej Smyk 4cf6f57fb5
comment fix (#22557) 2024-01-31 14:02:50 +01:00
Maciej Smyk ffb5270986
[DOCS] ref fix + benchmark fix + ov:state_network fix (#22529) 2024-01-30 10:38:47 +01:00
Zlobin Vladimir ec4f36b015
Fix grammar (#22457) 2024-01-30 01:21:31 +04:00
Sebastian Golebiewski d9faf192a6
Removing NPU plugin specific properties (#22512) 2024-01-29 15:17:04 +01:00
Karol Blaszczak f52de9434b
[DOCS] conformance tables update (#22459) 2024-01-29 13:45:44 +01:00
Sebastian Golebiewski 9cd1afbe96
Updating reference to Inference Modes (#22502) 2024-01-29 13:18:03 +01:00
Sebastian Golebiewski 935533429f
Fix Ubuntu version in NPU article (#22499) 2024-01-29 13:01:17 +01:00
Sebastian Golebiewski af64de8add
[DOCS] Updating interactive tutorials for 23.3 (#22495)
* Updating Interactive Tutorials

* Updating Tutorials
2024-01-29 11:03:24 +01:00
Ivan Tikhonov 39fc555364
Stateful Models Documentation Update (#22410)
* Squashed commits: Updating Stateful Models Documentation

* Apply suggestions from code review

Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>

* update illustrations

* Fix links; change illustrations format to svg; apply review comments

* add missing files

* Fix missprints; apply review comments; fix links

* fix missprints

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

* fix illustrations

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/stateful_models_intro/ways_to_get_stateful_model.rst

* Fix the remaining links, add a correct illustration for makestateful transformation, refactoring

---------

Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
2024-01-29 13:35:05 +04:00
Karol Blaszczak 48d5d4b79c
[DOCS] excell link restructure (#22463) 2024-01-26 16:55:27 +01:00
Tatiana Savina 8ea3f717cb
[DOCS] Fix nightly link (#22431)
* fix nightly link

* change link
2024-01-25 17:06:24 +01:00
Karol Blaszczak b1c8402eaa
[DOCS] fix link (#22404) 2024-01-25 14:03:19 +01:00
Sebastian Golebiewski 44d661a0ea
[DOCS] Fixing reference to PyTorch conversion for 23.3 (#22393)
* Fixing reference to PyTorch conversion
* Fixing formatting in Release Notes
* Removing Openvino Supported Models article
2024-01-24 16:19:19 +01:00
Sergey Lyalin ad379d7dd5
Updated Optimize and Deploy Generative AI Models documentation article (#22356)
* Updated Optimize and Deploy Generative AI Models article with stateful models information and new examples/resources

* Apply suggestions from code review

Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>

* Apply suggestions from code review

Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>

---------

Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
2024-01-24 18:20:11 +04:00
Maciej Smyk c4d1b11ee8
[DOCS] NPU table fix for 23.3 2024-01-24 14:44:06 +01:00
Karol Blaszczak 9645660166
[DOCS] NPU further tweaks (#22385) 2024-01-24 13:31:59 +01:00
Maciej Smyk 2d6b7e2b80
Update NPU.rst (#22377) 2024-01-24 11:21:27 +01:00
Andrzej Kopytko 0390d68ddb
[DOCS] quick fix graph.js to 23.3 2024-01-24 11:17:38 +01:00
Karol Blaszczak 489cc9304d
[DOCS] final benchmark data + conf version (#22375)
* [DOCS] final benchmark data
* conf.py update 23.3
2024-01-24 11:09:17 +01:00
Karol Blaszczak 900d7f8048
[DOCS] NPU update (#22373) 2024-01-24 09:44:14 +01:00
Karol Blaszczak 0245affbb4
[DOCS] releasenotes 23.3 update (#22362) 2024-01-24 08:34:59 +01:00
Sebastian Golebiewski 7721fa7603
[DOCS] Porting Benchmark Changes to 23.3 (#22357)
* change benchmarks to 23.3

* Updating benchmarks - porting 22353

* Porting 22353

---------

Co-authored-by: Kopytko, AndrzejX <andrzejx.kopytko@intel.com>
2024-01-23 17:52:59 +01:00
Tatiana Savina 1207ec4277
[DOCS] Docs review (#22340)
* add changes

* remove article
2024-01-23 14:32:31 +01:00
Andrzej Kopytko d9d4c9dacd
Main changes without valid data (#22065) (#22099) 2024-01-23 14:22:04 +01:00
Andrzej Kopytko a9d502b315
[Docs] benchmarks precisions port 2023 3 (#22319)
* [DOCS] update

* added rst update
2024-01-23 13:35:55 +01:00
Maciej Smyk 57a9c41a9a
[DOCS] Removing broken links from Model Representation article for 23.3 (#22344)
* Update model_representation.rst

* Update model_representation.rst
2024-01-23 13:00:12 +01:00
Tatiana Savina dd0a833d68
update ov version (#22342) 2024-01-23 12:02:15 +01:00
Vishniakov Nikolai d5c91c758c
Fix wstring join_paths (#22271) (#22305)
* Fix wstring join_paths

* Fix codestyle
2024-01-23 03:10:12 +04:00
Mikhail Ryzhov ecbb1387ab
increased timeout (#22306) 2024-01-22 22:04:40 +04:00
Andrzej Kopytko 45b501bb6f
[Docs] graph.js bf16 prec port to 2023.3 (#22317)
* Main changes without valid data

* fix
2024-01-22 16:08:27 +01:00
Alexander Kozlov aeda8aae2e
[DOCS] Updated weight compression documentation (#22063)
authored by alexander.kozlov@intel.com
2024-01-22 12:52:52 +01:00
Tatiana Savina b58d9cede7
add save model info (#22267) (#22269) 2024-01-19 14:49:17 +01:00
Ilya Lavrenov d19307ba13
Ported changes from master (#22193)
* [CI] [GHA] Introduce JS API as a part of the existing workflows (#21898)

* add js api to linux

* try inside the ov repo

* use rel path

* use a separate job for js api

* correct command formatting

* add missing var

* use spacing

* mv js building

* add node installing

* add to windows

* check pwsh and cmd running npm

* add smart CI conditions; disable for win

* use node version as env var

* extract js job into a separate workflow, add to other *nix

* fix input name

* Activate js bindings tests for arm64

* upload ov js package

* correct formatting

* add missing syntax

---------

Co-authored-by: Vishniakov Nikolai <nikolai.vishniakov@intel.com>

* Cmake Python build option flags should be added to the command in step #3 not step #4. I fixed the typo (#21993)

* [CI] [GHA] [JS API] Remove explicit default values settings in Linux ARM64 `cmake` (#22019)

* rm explicit default values settings

* Activate mac arm64 js api check

* Specify test run

---------

Co-authored-by: Vishniakov Nikolai <nikolai.vishniakov@intel.com>

* [OV JS] Activate validation for mac x86 (#22035)

* Extend validation for mac x86

* Remove extra params

* fixed broken doc links (#22088)

Co-authored-by: Przemyslaw Wysocki <przemyslaw.wysocki@intel.com>

* [GHA] Update MO deps (#22130)

* [GHA] Update MO deps

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>

* Update .github/components.yml

---------

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>

* Avoid DOWNLOAD_EXTRACT_TIMESTAMP warning (#22135)

* Avoid DOWNLOAD_EXTRACT_TIMESTAMP warning

* Change applying policy condition

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>

---------

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>

* Fixed API validator search (#22136)

* [OV JS] Conditional enabling of JS API (#22139)

* Disable js api building for vcpkg

* Disable JS API by default

* Add disable JS API conditions in features.cmake

* Update cmake/features.cmake

* Update src/bindings/js/CMakeLists.txt

---------

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>

* Fixed GHSA-h5c8-rqwp-cp95 (#22159)

* [PyOV][SAMPLES] Fix bugbear issue B038 (#22183)

* Fixed compilation on GHA CI

* Decrease number of workers for ONNX Model tests to prevent OOM kills (#22243)

* Decrease number of workers for ONNX Model tests to prevent OOM kills

* Try to use "-n auto" also

---------

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
Co-authored-by: Andrei Kashchikhin <andrey.kashchikhin@intel.com>
Co-authored-by: Vishniakov Nikolai <nikolai.vishniakov@intel.com>
Co-authored-by: fredrickomondi <omondifredrick@gmail.com>
Co-authored-by: Santhosh Mamidisetti <92091342+SANTHOSH-MAMIDISETTI@users.noreply.github.com>
Co-authored-by: Przemyslaw Wysocki <przemyslaw.wysocki@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
Co-authored-by: Andrey Babushkin <andrey.babushkin@intel.com>
2024-01-19 16:46:19 +04:00
Sebastian Golebiewski 187e262c92
[DOCS] Updating templates for Binder and Google Colab buttons for 23.3 (#22238)
* Update for Binder and Google Colab buttons

* change image source
2024-01-19 07:55:20 +01:00
Alexander Suvorov aeff428378
[DOCS] Update 2023.3.0 Selector Tool (#22218)
* Update selector tool
2024-01-17 19:48:54 +01:00
Sergey Lyalin 3dd41ff8c3
String Tensors Basic Documentation (#22097)
* Basic documentation of how to use string tensors

* Fix typo: str_data -> bytes_data

* More clarification about UTF-8 and represenation capabilities

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/string_tensors.rst

Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Applied code review suggestion about 'floating-point'

* Update docs/articles_en/openvino_workflow/running_inference_with_openvino/string_tensors.rst

* Fixed a snippet tab rendering and items formatting in Additional Reasources section

---------

Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
2024-01-17 19:02:16 +04:00
Sebastian Golebiewski 10cae14374
Port 21661 Merge Samples Articles Language Versions (#22187) 2024-01-17 13:27:12 +04:00
Sebastian Golebiewski af4b99e7cf
Port 22006 Updating options for torch.compile (#22188) 2024-01-16 16:51:20 +01:00
Xuejun Zhai 7b782956ab
[CAPI][TEST] fix cancel api test with infer finished before cancel call (#22175)
* [CAPI][TEST] fix cancel api test with infer finished before cancel call

Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>

* fix error

Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>

---------

Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
2024-01-16 13:17:13 +00:00
Karol Blaszczak 1ef753a590
[DOCS] torchvision and optimization section (#22142)
port: https://github.com/openvinotoolkit/openvino/pull/22043
2024-01-15 08:20:18 +01:00
Karol Blaszczak e64ec3ff5a
[DOCS] fix npu and install 23.3 (#22125) 2024-01-12 17:51:51 +01:00
Sebastian Golebiewski 258dde9f52
Fix images for notebooks (#22131) 2024-01-12 14:18:51 +01:00
River Li 7c37e0916c
[CAPI] avoid random timeout issue in test cases (#22086) 2024-01-11 14:34:14 +04:00
Ilya Lavrenov b981887d6f
Compatibility with newer OpenCL versions (#22051) 2024-01-10 07:31:17 +00:00
Maciej Smyk 47c49e7e84
[DOCS] Hyperlink 23.3 update for 23.3 (#21909)
* link update

* link update

* link update

* Update dev_api_references.rst

* Update dev_api_references.rst

* Update apidoc.py

* notebooks links

* notebook link update

* filename update

* link update

* link update api
2024-01-08 14:37:37 +01:00
Luo Cheng ceeafaf64f
[CPU] Support group beam search (#21983)
* support group beam search

* support dyn batch without set_state

* apply review comments

* strides may be incorrect when batch changed

---------

Co-authored-by: Yu Xu <yu.xu@intel.com>
2024-01-08 16:42:05 +04:00
River Li 49231c0196
[CAPI] fixed ov_infer_request_wait_for return value issue (#22014) 2024-01-08 16:41:21 +04:00
Ilya Lavrenov 38ee887911
Ported changes from master to 2023.3 (#22012)
* [JS OV] Specify JS API GitHub config (#21785)

* Extend CODEOWNERS by js api responsibles

* Extend smart ci by js component

* Fix codeowners

* Fix component definition

* Update components.yml

* Update components.yml

---------

Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>

* [GHA] Fix smart ci for docs changes in linux arm64 (#21798)

Update linux_arm64.yml

---------

Co-authored-by: Vishniakov Nikolai <nikolai.vishniakov@intel.com>
Co-authored-by: Alina Kladieva <alina.kladieva@intel.com>
2024-01-08 16:33:58 +04:00
Maciej Smyk 429b89f0e4
Debian9 update to Ubuntu 18.04 (#21935) 2024-01-08 15:39:37 +04:00
Vitaliy Urusovskij 1df3008870
Fix `INFER_NOT_STARTED` infer status code (#21939)
* Fix `INFER_NOT_STARTED` infer status code

* Update `returnDeviceBusyOnSetBlobAfterAsyncInfer` test
to check `INFER_NOT_STARTED` status only

* Update `wait` tests with `INFER_NOT_STARTED` check
2024-01-08 15:35:20 +04:00
Maxim Vafin 7835368489
[DOCS] Update supported ops by PT FE (#21961) 2024-01-08 11:11:16 +01:00
Maxim Vafin cc8fadf187
[DOCS] Describe dict support (#21923)
* [DOCS] Describe dict support

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

---------

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
2024-01-08 11:11:05 +01:00
Pavel Durandin a741d5a8ba
Fix minus axis case for kv cache primitive buffer fusing (#21958)
Co-authored-by: Taylor Lee <taylor.lee@intel.com>
2024-01-05 09:22:42 +09:00
Alina Kladieva ca31765d1a
Bump OMZ submodule (#21943)
unlimit pyparsing to fix CVS-128637
2024-01-04 09:51:29 +01:00
Roman Lyamin 112a16a056
[GPU] Small fix for dynamic_shape_gather_opts pass (#21807) (#21882) 2023-12-28 14:48:38 +04:00
Roman Lyamin 7d948bf596
[GPU] Add set_arguments() call in case of dynamic dependencies of static inst (#21664) (#21884)
Co-authored-by: Sergey Shlyapnikov <sergey.shlyapnikov@intel.com>
2023-12-28 14:48:13 +04:00
Maciej Smyk 4e15cc0b7f
Port from 21815 (#21839) 2023-12-27 16:01:33 +01:00
Wilson Seok e1bd2a7b55
Revert "[GPU] Fix accuracy problems brought by permute_f_y kernel (#20241)" (#21830) (#21876)
This reverts commit 6719133af2.

Co-authored-by: Pavel Durandin <pavel.durandin@intel.com>
2023-12-27 18:31:41 +04:00
Surya Siddharth Pemmaraju 8a5e6b6738
Fixed a bug in input validation for torch.compile options (#21835)
* Fixed bug in input validation for torch.compile options

* Added default device if device is None

* Addressed PR comments
2023-12-22 14:26:08 +04:00
Sebastian Golebiewski 81b5895d24
Move Troubleshooting Reshape article to legacy section (#21837) 2023-12-22 11:15:01 +01:00
Maciej Smyk e37569ffd5
[DOCS] Removal of unused images + fixing image directives for 23.3 (#21818)
* removal of unused images

* removal of duplicate image files

* removal of unused files + fix for image directive

* removal of unused images

* removal of unused images + fix for protecting_model.rst

* overview document graph update
2023-12-21 13:14:37 +01:00
Karol Blaszczak 07fc4d0813
[DOCS] NPU update 23.3 (#21812) 2023-12-21 09:24:52 +00:00
Tatiana Savina 7ef8faecf0
fix note and img (#21797) 2023-12-21 08:47:51 +01:00
Ilya Lavrenov 53960d6018
Switched GHA branches to 2023.3 (#21793) 2023-12-20 19:22:51 +04:00
19656 changed files with 906708 additions and 688033 deletions

View File

@ -0,0 +1,6 @@
TransposeOpTest.NHWC2NCHW
TransposeOpTest.NCHW2NHWC
TransposeOpTest.TwoDim_int16
GatherOpTest.Gather_axis1_indices2d_int16
SoftmaxOperator.ThreeDimsAxis1
SoftmaxOperator.ThreeDimsAxis0

View File

@ -0,0 +1 @@
rel-1.14.0

14
.ci/pot/Jenkinsfile vendored Normal file
View File

@ -0,0 +1,14 @@
#!groovy
properties([
parameters([
string(defaultValue: '',
description: 'Pipeline shared library version (branch/tag/commit). Determined automatically if empty',
name: 'library_version')
])
])
loadOpenVinoLibrary {
potEntrypoint(this)
}

View File

@ -1,3 +0,0 @@
*
!install_build_dependencies.sh
!scripts/install_dependencies/install_openvino_dependencies.sh

14
.gitattributes vendored
View File

@ -65,5 +65,15 @@
*.vsdx filter=lfs diff=lfs merge=lfs -text
*.bmp filter=lfs diff=lfs merge=lfs -text
*.svg filter=lfs diff=lfs merge=lfs -text
.github/scripts/workflow_rerun/tests/data/log_archive_with_error.zip filter=lfs diff=lfs merge=lfs -text
.github/scripts/workflow_rerun/tests/data/log_archive_wo_error.zip filter=lfs diff=lfs merge=lfs -text
#POT attributes
tools/pot/tests/data/test_cases_refs/* filter=lfs diff=lfs merge=lfs -text
tools/pot/tests/data/models/*/* filter=lfs diff=lfs merge=lfs -text
tools/pot/tests/data/reference_models/* filter=lfs diff=lfs merge=lfs -text
tools/pot/tests/data/video/* filter=lfs diff=lfs merge=lfs -text
tools/pot/tests/data/reference_fake_quantize_conf/* filter=lfs diff=lfs merge=lfs -text
/tools/pot/tests/** -pot_package
/tools/pot/tools/auxilary/** -pot_package
/tools/pot/tools/run_series_experiments.py -pot_package
/tools/pot/.pylintrc -pot_package
/tools/pot/README_dev.md -pot_package

17
.github/CODEOWNERS vendored
View File

@ -29,14 +29,9 @@
/src/bindings/python/ @openvinotoolkit/openvino-ie-python-api-maintainers
/src/bindings/c/ @openvinotoolkit/openvino-c-api-maintainers
/src/bindings/js/ @openvinotoolkit/openvino-js-api-maintainers
/src/common/*transformations/ @openvinotoolkit/openvino-ie-transformations-maintainers
/src/core/ @openvinotoolkit/openvino-ngraph-maintainers
# OpenVINO Transformations
/src/common/*transformations/ @openvinotoolkit/openvino-ie-transformations-maintainers
/src/core/src/pass/ @openvinotoolkit/openvino-ie-transformations-maintainers
/src/core/src/pattern/ @openvinotoolkit/openvino-ie-transformations-maintainers
/src/core/include/openvino/pass/ @openvinotoolkit/openvino-ie-transformations-maintainers
# OpenVINO Samples:
/samples/c/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-c-api-maintainers
/samples/cpp/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-maintainers
@ -65,8 +60,8 @@
/src/tests/**/gpu/ @openvinotoolkit/openvino-ie-gpu-maintainers
/thirdparty/ocl/ @openvinotoolkit/openvino-ie-gpu-maintainers @openvinotoolkit/openvino-ie-gpu-developers
# OpenVINO NPU:
/src/plugins/intel_npu/ @openvinotoolkit/openvino-npu-maintainers @openvinotoolkit/openvino-npu-developers
# OpenVINO GNA:
/src/plugins/intel_gna/ @openvinotoolkit/openvino-ie-gna-maintainers
# OpenVINO Auto (MULTI) plugin:
/src/plugins/auto/ @openvinotoolkit/openvino-ie-auto-multi-maintainers
@ -101,7 +96,7 @@
/tests/layer_tests/tensorflow_tests @openvinotoolkit/openvino-tf-frontend-maintainers
/tests/layer_tests/jax_tests @openvinotoolkit/openvino-tf-frontend-maintainers
/tests/model_hub_tests @openvinotoolkit/openvino-tf-frontend-maintainers
/tests/model_hub_tests/pytorch @openvinotoolkit/openvino-pytorch-frontend-maintainers
/tests/model_hub_tests/torch_tests @openvinotoolkit/openvino-pytorch-frontend-maintainers
# Tools:
/tools/ @openvinotoolkit/openvino-tools-maintainers
@ -110,7 +105,8 @@
/tools/openvino_dev/ @openvinotoolkit/openvino-tools-maintainers @openvinotoolkit/openvino-ie-python-api-maintainers
/tools/mo/ @openvinotoolkit/openvino-mo-maintainers
/tools/ovc/ @openvinotoolkit/openvino-mo-maintainers
/thirdparty/open_model_zoo/ @openvinotoolkit/omz-maintainers
/tools/pot/ @openvinotoolkit/openvino-pot-maintainers
/thirdparty/open_model_zoo/ @openvinotoolkit/omz-maintainers @openvinotoolkit/openvino-pot-maintainers
# Documentation
/docs/ @openvinotoolkit/openvino-docs-maintainers
@ -124,6 +120,7 @@
/docs/snippets/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-maintainers
/docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino_contrib-arm_plugin-maintainers
/docs/OV_Runtime_UG/supported_plugins/CPU.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-cpu-maintainers
/docs/OV_Runtime_UG/supported_plugins/GNA.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-gna-maintainers
/docs/OV_Runtime_UG/supported_plugins/GPU*.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-gpu-maintainers
# Configuration management

View File

@ -41,7 +41,9 @@ body:
options:
- CPU
- GPU
- NPU
- GNA
- NCS2 (Intel Movidius)
- HDDL
- AUTO
- HETERO
- BATCH

View File

@ -72,7 +72,7 @@ body:
- type: textarea
id: build_script
attributes:
label: Build script or step-by-step to reproduce
label: Build scrip or step-by-step to reproduce
description: How can we reproduce your issue?
placeholder: Please provide detailed instructions on how to reproduce the issue
validations:

View File

@ -42,8 +42,6 @@ body:
discussions, guides.
value: |
- [Contribution guide - start here!](https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING.md)
- [Intel DevHub Discord channel](https://discord.gg/7pVRxUwdWG) - engage in discussions, ask questions and talk to OpenVINO developers
- [How to link your Pull Request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#manually-linking-a-pull-request-to-an-issue-using-the-pull-request-sidebar)
validations:
required: true

View File

@ -1,4 +0,0 @@
lib/
dist/
node_modules/
coverage/

View File

@ -1 +0,0 @@
20.6.0

View File

@ -1,3 +0,0 @@
dist/
node_modules/
coverage/

View File

@ -1,16 +0,0 @@
{
"printWidth": 80,
"tabWidth": 2,
"useTabs": false,
"semi": true,
"singleQuote": true,
"quoteProps": "as-needed",
"jsxSingleQuote": false,
"trailingComma": "none",
"bracketSpacing": true,
"bracketSameLine": true,
"arrowParens": "avoid",
"proseWrap": "always",
"htmlWhitespaceSensitivity": "css",
"endOfLine": "lf"
}

View File

@ -1,286 +0,0 @@
/**
* Unit tests for the action's main functionality, src/cleanupImpl.js
*/
const core = require('@actions/core');
const path = require('path');
const os = require('os');
const fs = require('fs');
const cleanupImpl = require('../src/cleanupImpl');
// Mock the GitHub Actions core library
const getInputMock = jest.spyOn(core, 'getInput').mockImplementation();
const setFailedMock = jest.spyOn(core, 'setFailed').mockImplementation();
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-cleanup-'));
const cacheRemotePath = path.join(tempDir, 'cache_remote');
const cacheFiles = ['cache_1.cache', 'cache_2.cache', 'cache_3.cache'];
const minAccessTime = 7 * 24 * 60 * 60 * 1000; // 1 week
// Mock the action's main function
const runMock = jest.spyOn(cleanupImpl, 'cleanUp');
describe('cleanup', () => {
beforeEach(() => {
jest.clearAllMocks();
// Set up file system before each test
fs.mkdirSync(cacheRemotePath, { recursive: true });
const fileSizeInBytes = 1024 * 1024 * 1024; // 1 GB
const buffer = Buffer.alloc(fileSizeInBytes);
let id = 1;
for (const cache of cacheFiles) {
const cachePath = path.join(cacheRemotePath, cache);
fs.writeFileSync(cachePath, buffer);
fs.utimesSync(
cachePath,
new Date(Date.now() - minAccessTime * id++),
new Date()
);
}
});
// Clean up mock file system after each test
afterEach(() => {
fs.rmSync(tempDir, { recursive: true });
});
it('Cleanup old files using restore key', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 1;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
// cache2 and cache3 should be removed
for (const cache of cacheFiles.slice(1, 2)) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(false);
}
// check that file1 exists
expect(fs.existsSync(path.join(cacheRemotePath, cacheFiles[0]))).toBe(true);
});
it('Cleanup old files using key', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'cache-size':
return 1;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
// cache2 and cache3 should be removed
for (const cache of cacheFiles.slice(1, 2)) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(false);
}
// check that file1 exists
expect(fs.existsSync(path.join(cacheRemotePath, cacheFiles[0]))).toBe(true);
});
it('Skip cleanup old files by size', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 5;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
for (const cache of cacheFiles) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(true);
}
});
it('Skip cleanup old files by atime', async () => {
for (const cache of cacheFiles) {
const cachePath = path.join(cacheRemotePath, cache);
fs.utimesSync(
cachePath,
new Date(Date.now() - minAccessTime / 2),
new Date()
);
}
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 1;
case 'max-cache-size':
return 5;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
for (const cache of cacheFiles) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(true);
}
});
it('Cleanup recently used files by max cache limit', async () => {
for (const cache of cacheFiles) {
const cachePath = path.join(cacheRemotePath, cache);
fs.utimesSync(
cachePath,
new Date(Date.now() - minAccessTime / 2),
new Date()
);
}
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 1;
case 'max-cache-size':
return 2;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
// cache2 and cache3 should be removed
for (const cache of cacheFiles.slice(1, 2)) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(false);
}
// check that file1 exists
expect(fs.existsSync(path.join(cacheRemotePath, cacheFiles[0]))).toBe(true);
});
it('Test unexpected behaviour', async () => {
// Set folder permissions to read-only
fs.chmodSync(cacheRemotePath, '555');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 2;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
expect(setFailedMock).not.toHaveBeenCalled();
fs.chmodSync(cacheRemotePath, '777');
});
it('Cleanup absent directory', async () => {
const cachePathAbsent = path.join(tempDir, 'cache_remote_absent');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cachePathAbsent;
case 'key':
return 'cache';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 2;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
expect(setFailedMock).not.toHaveBeenCalled();
});
it('Cleanup directory with subdirectory', async () => {
const cacheSubPath = path.join(cacheRemotePath, 'cache_subdir.cache');
fs.mkdirSync(cacheSubPath, { recursive: true });
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'restore-keys':
return 'cache';
case 'cache-size':
return 1;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
// cache2 and cache3 should be removed
for (const cache of cacheFiles.slice(1, 2)) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(false);
}
// check that file1 exists
expect(fs.existsSync(path.join(cacheRemotePath, cacheFiles[0]))).toBe(true);
// check that sub directory exists
expect(fs.existsSync(cacheSubPath)).toBe(true);
});
});

View File

@ -1,266 +0,0 @@
/**
* Unit tests for the action's main functionality, src/restoreImpl.js
*/
const core = require('@actions/core');
const path = require('path');
const os = require('os');
const fs = require('fs');
const tar = require('tar');
const restoreImpl = require('../src/restoreImpl');
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-restore-'));
const cacheLocalPath = path.join(tempDir, 'cache_local');
const cacheRemotePath = path.join(tempDir, 'cache_remote');
const cacheTmpPath = path.join(tempDir, 'cache_tmp');
const cacheFiles = ['cache_1.cache', 'cache_2.cache', 'cache_3.cache'];
const testFiles = ['file1.txt', 'file2.txt', 'file3.txt'];
// Mock the GitHub Actions core library
const getInputMock = jest.spyOn(core, 'getInput').mockImplementation();
const setOutputMock = jest.spyOn(core, 'setOutput').mockImplementation();
const setFailedMock = jest.spyOn(core, 'setFailed').mockImplementation();
// Clean up mock file system after each test
afterEach(() => {
fs.rmSync(tempDir, { recursive: true });
});
// Mock the action's main function
const runMock = jest.spyOn(restoreImpl, 'restore');
describe('restore', () => {
beforeEach(() => {
jest.clearAllMocks();
// Set up mock file system before each test
fs.mkdirSync(cacheLocalPath, { recursive: true });
fs.mkdirSync(cacheRemotePath, { recursive: true });
fs.mkdirSync(cacheTmpPath, { recursive: true });
// Define test files
const file1Path = path.join(cacheTmpPath, testFiles[0]);
fs.writeFileSync(file1Path, 'File 1 contents');
tar.c(
{
gzip: true,
file: cacheFiles[0],
cwd: cacheTmpPath,
sync: true
},
['.']
);
fs.renameSync(cacheFiles[0], path.join(cacheRemotePath, cacheFiles[0]));
fs.utimesSync(
path.join(cacheRemotePath, cacheFiles[0]),
new Date(Date.now() - 1500),
new Date(Date.now() - 1500)
);
//
const file2Path = path.join(cacheTmpPath, testFiles[1]);
fs.writeFileSync(file2Path, 'File 2 contents');
tar.c(
{
gzip: true,
file: cacheFiles[1],
cwd: cacheTmpPath,
sync: true
},
['.']
);
fs.renameSync(cacheFiles[1], path.join(cacheRemotePath, cacheFiles[1]));
fs.utimesSync(
path.join(cacheRemotePath, cacheFiles[1]),
new Date(Date.now() - 1000),
new Date(Date.now() - 1000)
);
//
const file3Path = path.join(cacheTmpPath, testFiles[2]);
fs.writeFileSync(file3Path, 'File 3 contents');
tar.c(
{
gzip: true,
file: cacheFiles[2],
cwd: cacheTmpPath,
sync: true
},
['.']
);
fs.renameSync(cacheFiles[2], path.join(cacheRemotePath, cacheFiles[2]));
fs.utimesSync(
path.join(cacheRemotePath, cacheFiles[2]),
new Date(Date.now() - 500),
new Date(Date.now() - 500)
);
});
it('gets the cache file', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await restoreImpl.restore();
expect(runMock).toHaveReturned();
// Verify that all of the core library functions were called correctly
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
cacheFiles[2]
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
let id = 1;
for (const filename of testFiles) {
const filePath = path.join(cacheLocalPath, filename);
expect(fs.existsSync(filePath)).toBe(true);
const fileContent = fs.readFileSync(filePath, 'utf8');
expect(fileContent).toBe(`File ${id++} contents`);
}
});
it('gets the updated cache file', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
// updated cache 2 access time
fs.utimesSync(
path.join(cacheRemotePath, cacheFiles[1]),
new Date(Date.now() - 100),
new Date(Date.now() - 100)
);
await restoreImpl.restore();
expect(runMock).toHaveReturned();
// Verify that all of the core library functions were called correctly
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
cacheFiles[1]
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
let id = 1;
for (const filename of testFiles.slice(0, 1)) {
const filePath = path.join(cacheLocalPath, filename);
expect(fs.existsSync(filePath)).toBe(true);
const fileContent = fs.readFileSync(filePath, 'utf8');
expect(fileContent).toBe(`File ${id++} contents`);
}
// check that file3 is absent
expect(fs.existsSync(path.join(cacheLocalPath, testFiles[2]))).toBe(false);
});
it('gets the cache file and extract to absent dir', async () => {
const cacheAbsentPath = path.join(tempDir, 'cache_absent');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'path':
return cacheAbsentPath;
default:
return '';
}
});
await restoreImpl.restore();
expect(runMock).toHaveReturned();
// Verify that all of the core library functions were called correctly
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
cacheFiles[2]
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
let id = 1;
for (const filename of testFiles) {
const filePath = path.join(cacheAbsentPath, filename);
expect(fs.existsSync(filePath)).toBe(true);
const fileContent = fs.readFileSync(filePath, 'utf8');
expect(fileContent).toBe(`File ${id++} contents`);
}
});
it('Test when no cache found', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await restoreImpl.restore();
expect(runMock).toHaveReturned();
// Verify that all of the core library functions were called correctly
expect(setOutputMock).toHaveBeenNthCalledWith(1, 'cache-file', '');
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', false);
for (const filename of testFiles) {
const filePath = path.join(cacheLocalPath, filename);
expect(fs.existsSync(filePath)).toBe(false);
}
expect(setFailedMock).not.toHaveBeenCalled();
});
it('Unsupported cache found', async () => {
const cacheFakePath = path.join(cacheRemotePath, 'cache-fake.cache');
fs.writeFileSync(cacheFakePath, 'Fake content');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await restoreImpl.restore();
expect(runMock).toHaveReturned();
expect(setOutputMock).not.toHaveBeenCalled();
expect(setFailedMock).not.toHaveBeenCalled();
});
});

View File

@ -1,190 +0,0 @@
/**
* Unit tests for the action's main functionality, src/saveImpl.js
*/
const core = require('@actions/core');
const path = require('path');
const os = require('os');
const fs = require('fs');
const saveImpl = require('../src/saveImpl');
// Mock the GitHub Actions core library
const getInputMock = jest.spyOn(core, 'getInput').mockImplementation();
const setOutputMock = jest.spyOn(core, 'setOutput').mockImplementation();
const setFailedMock = jest.spyOn(core, 'setFailed').mockImplementation();
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-'));
const cacheLocalPath = path.join(tempDir, 'cache_local');
const cacheRemotePath = path.join(tempDir, 'cache_remote');
// Clean up mock file system after each test
afterEach(() => {
fs.rmSync(tempDir, { recursive: true });
});
// Mock the action's main function
const runMock = jest.spyOn(saveImpl, 'save');
describe('save', () => {
beforeEach(() => {
jest.clearAllMocks();
// Set up mock file system before each test
// Create cache pathes
fs.mkdirSync(cacheLocalPath, { recursive: true });
fs.mkdirSync(cacheRemotePath, { recursive: true });
// Define test files
const file1Path = path.join(cacheLocalPath, 'file1.txt');
const file2Path = path.join(cacheLocalPath, 'file2.txt');
// Write content to the test files
fs.writeFileSync(file1Path, 'File 1 contents');
fs.writeFileSync(file2Path, 'File 2 contents');
});
it('Cache files pass', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
'cache.cache'
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
expect(fs.existsSync(`${cacheRemotePath}/cache.cache`)).toBe(true); // Check if the tar file was created
});
it('Cache files: absent local cache dir', async () => {
// Set the action's inputs as return values from core.getInput()
const cachePathAbsent = path.join(tempDir, 'cache_local_absent');
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'path':
return cachePathAbsent;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(fs.existsSync(`${cacheRemotePath}/cache.cache`)).toBe(false); // Check that the tar file was not created
});
it('Cache files: absent remote cache dir', async () => {
// Set the action's inputs as return values from core.getInput()
const cacheRemotePathAbsent = path.join(tempDir, 'cache_remote_absent');
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePathAbsent;
case 'key':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
'cache.cache'
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
expect(fs.existsSync(`${cacheRemotePathAbsent}/cache.cache`)).toBe(true); // Check if the tar file was created
});
it('Cache files: key is not set', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).not.toHaveBeenCalled();
expect(setFailedMock).not.toHaveBeenCalled();
});
it('Cache files: file already exists', async () => {
const cacheFakePath = path.join(cacheRemotePath, 'cache-fake.cache');
fs.writeFileSync(cacheFakePath, 'Fake content');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache-fake';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).not.toHaveBeenCalled();
expect(setFailedMock).not.toHaveBeenCalled();
});
it('Test unexpected behaviour', async () => {
// Set folder permissions to read-only
fs.chmodSync(cacheRemotePath, '555');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).not.toHaveBeenCalled();
expect(setFailedMock).toHaveBeenCalled();
});
});

View File

@ -1,60 +0,0 @@
name: 'Cache'
description: 'Cache to the local file system'
author: 'Mikhail Ryzhov'
inputs:
cache-path:
description: 'A local path where to search and save caches'
required: true
path:
description:
'A list of files, directories, and wildcard patterns to cache and restore'
required: true
key:
description: 'An explicit key for restoring and saving the cache'
required: true
restore-keys:
description:
'An ordered list of keys to use for restoring stale cache if no cache hit
occurred for key.'
default: ''
required: false
save-always:
description:
'Run the post step to save the cache even if another step before fails'
default: false
required: false
cleanup-always:
description:
'Run the post cleanup step to remove old cache files even if another step
before fails'
default: false
required: false
cache-size:
description:
'Allowed cache storage size in Gb. Least recently used caches will be
automatically evicted to limit the total cache storage. The cache size can
exceed this value, but it is limited by "max-cache-size" value'
default: 10
required: false
max-cache-size:
description:
'Maximum cache storage size in Gb. All cache files will be automatically
evicted to limit the total cache storage'
default: 50
required: false
outputs:
cache-hit:
description:
'A boolean value to indicate an exact match was found for the primary key'
cache-file:
description: 'Found cache file'
runs:
using: node20
main: 'dist/restore/index.js'
post: 'dist/save/index.js'
post-if:
'success() || github.event.inputs.save-always ||
github.event.inputs.cleanup-always'

View File

@ -1,37 +0,0 @@
name: 'Cleanup Cache'
description: 'Remove old cache artifacts'
author: 'OpenVINO Developers'
inputs:
cache-path:
description: 'A local path where to search and save caches'
required: true
key:
description: 'An explicit key for restoring and saving the cache'
required: true
restore-keys:
description:
'An ordered list of keys to use for restoring stale cache if no cache hit
occurred for key.'
default: ''
required: false
cache-size:
description:
'Allowed cache storage size in Gb. Least recently used caches will be
automatically evicted to limit the total cache storage. The cache size can
exceed this value, but it is limited by "max-cache-size" value'
default: 10
required: false
max-cache-size:
description:
'Maximum cache storage size in Gb. All cache files will be automatically
evicted to limit the total cache storage'
default: 50
required: false
runs:
using: 'node20'
main: '../dist/cleanup-only/index.js'
branding:
icon: 'archive'
color: 'gray-dark'

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@ -1,50 +0,0 @@
env:
commonjs: true
es6: true
jest: true
node: true
globals:
Atomics: readonly
SharedArrayBuffer: readonly
ignorePatterns:
- '!.*'
- '**/node_modules/.*'
- '**/dist/.*'
- '**/coverage/.*'
- '*.json'
parser: '@babel/eslint-parser'
parserOptions:
ecmaVersion: 2023
sourceType: module
requireConfigFile: false
babelOptions:
babelrc: false
configFile: false
presets:
- jest
plugins:
- jest
extends:
- eslint:recommended
- plugin:github/recommended
- plugin:jest/recommended
rules:
{
'camelcase': 'warn',
'eslint-comments/no-use': 'off',
'eslint-comments/no-unused-disable': 'off',
'i18n-text/no-en': 'off',
'import/no-commonjs': 'off',
'import/no-namespace': 'off',
'no-console': 'off',
'no-unused-vars': 'warn',
'prettier/prettier': 'error',
'semi': 'error'
}

View File

@ -1,18 +0,0 @@
# Unordered list style
MD004:
style: dash
# Ordered list item prefix
MD029:
style: one
# Spaces after list markers
MD030:
ul_single: 1
ol_single: 1
ul_multi: 1
ol_multi: 1
# Code block style
MD046:
style: fenced

View File

@ -1,10 +0,0 @@
rules:
document-end: disable
document-start:
level: warning
present: false
line-length:
level: warning
max: 80
allow-non-breakable-words: true
allow-non-breakable-inline-mappings: true

10054
.github/actions/cache/package-lock.json generated vendored

File diff suppressed because it is too large Load Diff

View File

@ -1,73 +0,0 @@
{
"name": "compressed-cache-action",
"description": "action to save/restore cache as tarballs",
"version": "0.0.1",
"author": "OpenVINO Developers",
"private": true,
"keywords": [
"GitHub",
"Actions",
"JavaScript"
],
"engines": {
"node": ">=20"
},
"main": "dist/restore/index.js",
"scripts": {
"bundle": "npm run format:write && npm run package",
"ci-test": "npx jest",
"coverage": "npx jest -- --coverage",
"format:write": "npx prettier --write .",
"format:check": "npx prettier --check .",
"lint": "npx eslint . -c ./linters/.eslintrc.yml ",
"package": "npx ncc build src/restore.js -o dist/restore && npx ncc build src/save.js -o dist/save && npx ncc build src/restore.js -o dist/restore-only && npx ncc build src/save.js -o dist/save-only && npx ncc build src/cleanup.js -o dist/cleanup-only",
"package:watch": "npm run package -- --watch",
"test": "npx jest",
"all": "npm run format:write && npm run lint && npm run test && npm run coverage && npm run package"
},
"eslintConfig": {
"extends": "./linters/.eslintrc.yml"
},
"jest": {
"verbose": true,
"clearMocks": true,
"testEnvironment": "node",
"moduleFileExtensions": [
"js"
],
"testMatch": [
"**/*.test.js"
],
"testPathIgnorePatterns": [
"/node_modules/",
"/dist/"
],
"coverageReporters": [
"json-summary",
"text",
"lcov"
],
"collectCoverage": true,
"collectCoverageFrom": [
"./src/**"
]
},
"dependencies": {
"@actions/core": "^1.10.1",
"fs": "^0.0.1-security",
"tar": "^6.2.0"
},
"devDependencies": {
"@babel/core": "^7.24.0",
"@babel/eslint-parser": "^7.23.10",
"@babel/preset-env": "^7.24.0",
"@vercel/ncc": "^0.38.1",
"babel-preset-jest": "^29.6.3",
"eslint": "^8.57.0",
"eslint-plugin-github": "^4.10.2",
"eslint-plugin-jest": "^27.9.0",
"fs": "0.0.1-security",
"jest": "^29.7.0",
"prettier": "^3.2.5"
}
}

View File

@ -1,35 +0,0 @@
name: 'Restore Cache'
description: 'Restore Cache artifacts to improve workflow execution time'
author: 'OpenVINO Developers'
inputs:
cache-path:
description: 'A local path where to search and save caches'
required: true
path:
description:
'A list of files, directories, and wildcard patterns to cache and restore'
required: true
key:
description: 'An explicit key for restoring and saving the cache'
required: true
restore-keys:
description:
'An ordered list of keys to use for restoring stale cache if no cache hit
occurred for key.'
default: ''
required: false
outputs:
cache-hit:
description:
'A boolean value to indicate an exact match was found for the primary key'
cache-file:
description: 'Found cache file'
runs:
using: 'node20'
main: '../dist/restore-only/index.js'
branding:
icon: 'archive'
color: 'gray-dark'

View File

@ -1,22 +0,0 @@
name: 'Save a cache'
description: 'Cache to the local file system'
author: 'OpenVINO Developers'
inputs:
cache-path:
description: 'A local path where to search and save caches'
required: true
path:
description:
'A list of files, directories, and wildcard patterns to cache and restore'
required: true
key:
description: 'An explicit key for restoring and saving the cache'
required: true
runs:
using: 'node20'
main: '../dist/save-only/index.js'
branding:
icon: 'archive'
color: 'gray-dark'

View File

@ -1,3 +0,0 @@
const { cleanUp } = require('./cleanupImpl');
cleanUp();

View File

@ -1,85 +0,0 @@
const core = require('@actions/core');
const fs = require('fs/promises');
const path = require('path');
const {
getSortedCacheFiles,
humanReadableFileSize,
calculateTotalSize
} = require('./utils');
// Function to remove old files if their combined size exceeds the allowed size
async function cleanUp() {
try {
const cacheRemotePath = core.getInput('cache-path', { required: true });
const key = core.getInput('key', { required: true });
const keysRestore = core
.getInput('restore-keys', { required: false })
.split('\n')
.map(s => s.replace(/^!\s+/, '!').trim())
.filter(x => x !== '');
const cacheSize = core.getInput('cache-size', { required: false });
const cacheMaxSize = core.getInput('max-cache-size', { required: false });
// Minimum time peroid in milliseconds when the files was not useds
const minAccessTime = 7 * 24 * 60 * 60 * 1000; // 1 week
const currentDate = new Date();
const minAccessDateAgo = new Date(currentDate - minAccessTime);
core.debug(`cache-path: ${cacheRemotePath}`);
core.debug(`key: ${key}`);
core.debug(`restore-keys: ${keysRestore}`);
core.debug(`cache-size: ${cacheSize}`);
core.debug(`max-cache-size: ${cacheMaxSize}`);
let keyPattern = key;
if (keysRestore && keysRestore.length) {
keyPattern = keysRestore.join('|');
}
const files = await getSortedCacheFiles(cacheRemotePath, keyPattern);
const minCacheSizeInBytes = cacheSize * 1024 * 1024 * 1024;
const maxCacheSizeInBytes = cacheMaxSize * 1024 * 1024 * 1024;
let totalSize = await calculateTotalSize(cacheRemotePath, files);
if (totalSize <= minCacheSizeInBytes) {
core.info(
`The cache storage size ${humanReadableFileSize(totalSize)} less then the allowed size ${humanReadableFileSize(minCacheSizeInBytes)}`
);
return;
}
core.info(
`The cache storage size ${humanReadableFileSize(totalSize)} exceeds allowed size ${humanReadableFileSize(minCacheSizeInBytes)}`
);
for (const file of files.reverse()) {
const filePath = path.join(cacheRemotePath, file);
const fileStats = await fs.stat(filePath);
// skipping recently used files if total cache size less then maxCacheSizeInBytes
if (
!fileStats.isFile() ||
(fileStats.atime >= minAccessDateAgo && totalSize < maxCacheSizeInBytes)
) {
core.info(`Skipping: ${filePath}`);
continue;
}
core.info(`Removing file: ${filePath}`);
await fs.unlink(filePath);
core.info(`${filePath} removed successfully`);
totalSize -= fileStats.size;
// Exit if total size is within limit
if (totalSize <= minCacheSizeInBytes) {
core.info('Old cache files removed successfully');
return;
}
}
} catch (error) {
core.error(`Error removing old cache files: ${error.message}`);
}
}
module.exports = {
cleanUp
};

View File

@ -1,6 +0,0 @@
/**
* The entrypoint for the action.
*/
const { restore } = require('./restoreImpl');
restore();

View File

@ -1,84 +0,0 @@
const core = require('@actions/core');
const fs = require('fs/promises');
const path = require('path');
const tar = require('tar');
const os = require('os');
const {
getSortedCacheFiles,
humanReadableFileSize,
checkFileExists
} = require('./utils');
/**
* The main function for the action.
* @returns {Promise<void>} Resolves when the action is complete.
*/
async function restore() {
try {
const cacheRemotePath = core.getInput('cache-path', { required: true });
const cacheLocalPath = core.getInput('path', { required: true });
const key = core.getInput('key', { required: true });
const keysRestore = core
.getInput('restore-keys', { required: false })
.split('\n')
.map(s => s.replace(/^!\s+/, '!').trim())
.filter(x => x !== '');
core.debug(`cache-path: ${cacheRemotePath}`);
core.debug(`path: ${cacheLocalPath}`);
core.debug(`key: ${key}`);
core.debug(`restore-keys: ${keysRestore}`);
let keyPattern = key;
if (keysRestore && keysRestore.length) {
keyPattern = keysRestore.join('|');
}
core.info(`Looking for ${keyPattern} in ${cacheRemotePath}`);
const files = await getSortedCacheFiles(cacheRemotePath, keyPattern);
if (files.length) {
const cacheFile = files[0];
const cachePath = path.join(cacheRemotePath, cacheFile);
const cacheStat = await fs.stat(cachePath);
core.info(
`Found cache file: ${cacheFile}, size: ${humanReadableFileSize(cacheStat.size)}`
);
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'cache-'));
// copy file to temp dir
await fs.copyFile(cachePath, path.join(tempDir, cacheFile));
core.info(`${cacheFile} was copied to ${tempDir}/${cacheFile}`);
// extract
if (!(await checkFileExists(cacheLocalPath))) {
await fs.mkdir(cacheLocalPath);
}
core.info(`Extracting ${cacheFile} to ${cacheLocalPath}`);
tar.x({
file: path.join(tempDir, cacheFile),
cwd: cacheLocalPath,
sync: true
});
core.info(`Cache extracted to ${cacheLocalPath}`);
core.setOutput('cache-file', cacheFile);
core.setOutput('cache-hit', true);
} else {
core.warning(
`Could not found any suitable cache files in ${cacheRemotePath} with key ${key}`
);
core.setOutput('cache-file', '');
core.setOutput('cache-hit', false);
}
} catch (error) {
// do not fail action if cache could not be restored
core.error(error.message);
}
}
module.exports = {
restore
};

View File

@ -1,14 +0,0 @@
const core = require('@actions/core');
const { save } = require('./saveImpl');
const { cleanUp } = require('./cleanupImpl');
const saveAlways = core.getInput('save-always', { required: false });
const cleanUpAlways = core.getInput('cleanup-always', { required: false });
if (saveAlways === 'true') {
save();
}
if (cleanUpAlways === 'true') {
cleanUp();
}

View File

@ -1,71 +0,0 @@
const core = require('@actions/core');
const tar = require('tar');
const fs = require('fs/promises');
const path = require('path');
const { humanReadableFileSize, checkFileExists } = require('./utils');
/**
* The main function for the action.
* @returns {Promise<void>} Resolves when the action is complete.
*/
async function save() {
try {
const cacheRemotePath = core.getInput('cache-path', { required: true });
const toCachePath = core.getInput('path', { required: true });
const key = core.getInput('key', { required: true });
core.debug(`cache-path: ${cacheRemotePath}`);
core.debug(`path: ${toCachePath}`);
core.debug(`key: ${key}`);
if (!key) {
core.warning(`Key ${key} is not specified.`);
return;
}
const tarName = `${key}.cache`;
const tarPath = path.join(cacheRemotePath, tarName);
const tarNameTmp = `${key}.tmp`;
const tarPathTmp = path.join(cacheRemotePath, tarNameTmp);
if (await checkFileExists(tarPath)) {
core.warning(`Cache file ${tarName} already exists`);
return;
}
core.info(`Preparing cache archive ${tarName}`);
tar.c(
{
gzip: true,
file: tarName,
cwd: toCachePath,
sync: true
},
['.']
);
const tarStat = await fs.stat(tarName);
core.info(
`Created cache tarball: ${tarName}, size: ${humanReadableFileSize(tarStat.size)}`
);
// remote cache directory may not be created yet
if (!(await checkFileExists(cacheRemotePath))) {
await fs.mkdir(cacheRemotePath, { recursive: true });
}
core.info('Copying cache...');
await fs.copyFile(tarName, tarPathTmp);
// After copying is done, rename file
await fs.rename(tarPathTmp, tarPath);
core.info(`${tarName} copied to ${tarPath}`);
core.setOutput('cache-file', tarName);
core.setOutput('cache-hit', true);
} catch (error) {
core.setFailed(error.message);
}
}
module.exports = {
save
};

View File

@ -1,72 +0,0 @@
const core = require('@actions/core');
const fs = require('fs');
const path = require('path');
async function getSortedCacheFiles(cachePath, key = '') {
if (!(await checkFileExists(cachePath))) {
core.warning(`${cachePath} doesn't exist`);
return [];
}
const cachePattern = new RegExp(`^((${key}).*[.]cache)$`);
const filesSorted = (await fs.promises.readdir(cachePath))
.filter(fileName => cachePattern.test(fileName))
.map(fileName => ({
name: fileName,
time: fs.statSync(path.join(cachePath, fileName)).atimeMs
}))
.sort((a, b) => b.time - a.time)
.map(file => file.name);
core.debug(
filesSorted.map(fileName => ({
name: fileName,
atime: fs.statSync(path.join(cachePath, fileName)).atimeMs
}))
);
return filesSorted;
}
function humanReadableFileSize(sizeInBytes) {
const units = ['B', 'KB', 'MB', 'GB', 'TB'];
let id = 0;
while (sizeInBytes >= 1024 && id < units.length - 1) {
sizeInBytes /= 1024;
id++;
}
return `${sizeInBytes.toFixed(2)} ${units[id]}`;
}
// Function to calculate the total size of files in bytes
async function calculateTotalSize(dir, files) {
let totalSize = 0;
for (const file of files) {
const filePath = path.join(dir, file);
const fileStats = await fs.promises.stat(filePath);
if (fileStats.isFile()) {
totalSize += fileStats.size;
}
}
return totalSize;
}
async function checkFileExists(filePath) {
try {
await fs.promises.access(filePath);
return true;
} catch (error) {
return false;
}
}
module.exports = {
getSortedCacheFiles,
humanReadableFileSize,
calculateTotalSize,
checkFileExists
};

View File

@ -1,72 +0,0 @@
name: 'Handle Docker images'
description: 'Builds, tags and pushes a given Docker image when needed'
inputs:
images:
description: 'Image names (registry name + namespace + base name)'
required: true
registry:
description: 'Docker registry'
required: true
dockerfiles_root_dir:
description: 'Path to dockerfiles root dir relative to repository root'
required: true
push:
description: 'Push built images to registry'
required: false
default: 'true'
changed_components:
description: 'Components changed by a pull request'
required: true
outputs:
images:
description: "Images to use in workflow"
value: ${{ steps.handle_images.outputs.images }}
runs:
using: 'composite'
steps:
- name: Checkout head
uses: actions/checkout@v4
- name: Checkout base
uses: actions/checkout@v4
with:
ref: ${{ github.base_ref || github.event.merge_group.base_ref }}
sparse-checkout: ${{ inputs.dockerfiles_root_dir }}/docker_tag
path: base
- name: Install Python dependencies
uses: py-actions/py-dependency-install@v4
with:
path: "${{ github.action_path }}/requirements.txt"
update-setuptools: "false"
update-wheel: "false"
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v3
- name: Handle docker images
id: handle_images
shell: bash
run: |
images=$(echo "${{ inputs.images }}" | tr '\n' ',' | sed 's/,*$//')
pr="${{ github.event.pull_request.number }}"
python3 .github/actions/handle_docker/get_images_to_build.py \
-d "${{ inputs.dockerfiles_root_dir }}" \
-r "${{ inputs.registry }}" \
--images "$images" \
--head_tag_file "${{ inputs.dockerfiles_root_dir }}/docker_tag" \
--base_tag_file "base/${{ inputs.dockerfiles_root_dir }}/docker_tag" \
--docker_env_changed "${{ fromJSON(inputs.changed_components).docker_env }}" \
--dockerfiles_changed "${{ fromJSON(inputs.changed_components).dockerfiles }}" \
--docker_builder "${{ steps.buildx.outputs.name}}" \
--repo "${{ github.repository }}" \
--ref_name "${{ github.ref_name }}" \
$([[ -n $pr ]] && echo "--pr $pr" || echo '-s ${{ github.sha }}') \
$([[ -n "${{ inputs.push }}" ]] && echo "--push" || echo '')
env:
GITHUB_TOKEN: ${{ github.token }}

View File

@ -1,129 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import argparse
import json
import re
import sys
from distutils.util import strtobool
from helpers import *
from images_api import *
def parse_args():
parser = argparse.ArgumentParser(description='Returns list of Docker images to build for a given workflow')
parser.add_argument('-i', '--images', required=True, help='Comma-separated docker images')
parser.add_argument('-d', '--dockerfiles_root', required=True, help='Path to dockerfiles')
parser.add_argument('-r', '--registry', required=True, help='Docker registry name')
parser.add_argument('-s', '--commit', required=False, help='Commit SHA. If not set, --pr is used')
parser.add_argument('-b', '--docker_builder', required=False, help='Docker buildx builder name')
parser.add_argument('--pr', type=int, required=False, help='PR number, if event is pull_request')
parser.add_argument('--head_tag_file', default='.github/dockerfiles/docker_tag', help='Head docker tag file path')
parser.add_argument('--base_tag_file', default=None, required=False, help='Base docker tag file path')
parser.add_argument('--ref_name', required=False, default='', help='GitHub ref name')
parser.add_argument('--repo', default='openvinotoolkit/openvino', help='GitHub repository')
parser.add_argument('--docker_env_changed', type=lambda x: bool(strtobool(x)), default=True,
help='Whether PR changes docker env')
parser.add_argument('--dockerfiles_changed', type=lambda x: bool(strtobool(x)), default=True,
help='Whether PR changes dockerfiles')
parser.add_argument('--action_path', default='.github/actions/handle_docker', help='Path to this GitHub action')
parser.add_argument('--push', action='store_true', required=False, help='Whether to push images to registry')
parser.add_argument('--dry_run', action='store_true', required=False, help='Dry run')
args = parser.parse_args()
return args
def main():
init_logger()
logger = logging.getLogger(__name__)
args = parse_args()
for arg, value in sorted(vars(args).items()):
logger.info(f"Argument {arg}: {value}")
head_tag = Path(args.head_tag_file).read_text().strip()
base_tag_exists = args.base_tag_file and Path(args.base_tag_file).exists()
base_tag = Path(args.base_tag_file).read_text().strip() if base_tag_exists else None
all_dockerfiles = Path(args.dockerfiles_root).rglob('**/*/Dockerfile')
images = ImagesHandler(args.dry_run)
for image in all_dockerfiles:
images.add_from_dockerfile(image, args.dockerfiles_root, args.registry, head_tag, base_tag)
requested_images = set(args.images.split(','))
skip_workflow = False
missing_only = False
merge_queue_target_branch = next(iter(re.findall(f'^gh-readonly-queue/(.*)/', args.ref_name)), None)
if args.pr:
environment_affected = args.docker_env_changed or args.dockerfiles_changed
if environment_affected:
expected_tag = f'pr-{args.pr}'
if head_tag != expected_tag:
logger.error(f"Please update docker tag in {args.head_tag_file} to {expected_tag}")
sys.exit(1)
elif merge_queue_target_branch:
environment_affected = head_tag != base_tag
if environment_affected:
logger.info(f"Environment is affected by PR(s) in merge group")
else:
environment_affected = False
if environment_affected:
changeset = get_changeset(args.repo, args.pr, merge_queue_target_branch, args.commit)
changed_dockerfiles = [p for p in changeset if p.startswith(args.dockerfiles_root) and p.endswith('Dockerfile')]
if args.docker_env_changed:
logger.info(f"Common docker environment is modified, will build all requested images")
changed_images = requested_images
else:
logger.info(f"Common docker environment is not modified, will build only changed and missing images")
changed_images = set([name_from_dockerfile(d, args.dockerfiles_root) for d in changed_dockerfiles])
unchanged_images = requested_images - changed_images
unchanged_with_no_base = images.get_missing(unchanged_images, base=True)
if unchanged_with_no_base:
logger.info("The following images were unchanged, but will be built anyway since the base for them "
f"is missing in registry: {unchanged_with_no_base}")
images_to_tag = unchanged_images.difference(unchanged_with_no_base)
images_to_build = requested_images.intersection(changed_images).union(unchanged_with_no_base)
only_dockerfiles_changed = len(changeset) == len(changed_dockerfiles)
if only_dockerfiles_changed and not images_to_build:
skip_workflow = True
else:
logger.info(f"Environment is not affected, will build only missing images, if any")
images_to_build = requested_images
images_to_tag = []
missing_only = True
if not images_to_build:
logger.info(f"No images to build, will return the list of pre-built images with a new tag")
built_images = images.build(images_to_build, missing_only, args.push, args.docker_builder)
if not built_images:
logger.info(f"No images were built, a new tag will be applied to a pre-built base image if needed")
# When a custom builder is used, it allows to push the image automatically once built. Otherwise, pushing manually
if args.push and not args.docker_builder:
images.push(images_to_build, missing_only)
if environment_affected and base_tag:
images.tag(images_to_tag)
images_output = images_to_output(images.get(requested_images))
set_github_output("images", json.dumps(images_output))
if skip_workflow:
logger.info(f"Docker image changes are irrelevant for current workflow, workflow may be skipped")
set_github_output("skip_workflow", str(skip_workflow))
main()

View File

@ -1,75 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import logging
import os
import subprocess
from ghapi.all import GhApi
from pathlib import Path
def init_logger():
logging.basicConfig(level=logging.INFO,
format='%(asctime)s %(name)-15s %(levelname)-8s %(message)s',
datefmt='%m-%d-%Y %H:%M:%S')
def set_github_output(name: str, value: str, github_output_var_name: str = 'GITHUB_OUTPUT'):
"""Sets output variable for a GitHub Action"""
logger = logging.getLogger(__name__)
# In an environment variable "GITHUB_OUTPUT" GHA stores path to a file to write outputs to
with open(os.environ.get(github_output_var_name), 'a+') as file:
logger.info(f"Add {name}={value} to {github_output_var_name}")
print(f'{name}={value}', file=file)
def images_to_output(images: list):
images_output = {}
for image in images:
image_name, os_name = image.name.split('/', 1)
if image_name not in images_output:
images_output[image_name] = {}
images_output[image_name][os_name] = image.ref()
return images_output
def get_changeset(repo: str, pr: str, target_branch: str, commit_sha: str):
"""Returns changeset either from PR or commit"""
owner, repository = repo.split('/')
gh_api = GhApi(owner=owner, repo=repository, token=os.getenv("GITHUB_TOKEN"))
if pr:
changed_files = gh_api.pulls.list_files(pr)
elif target_branch:
target_branch_head_commit = gh_api.repos.get_branch(target_branch).commit.sha
changed_files = gh_api.repos.compare_commits(f'{target_branch_head_commit}...{commit_sha}').get('files', [])
else:
raise ValueError(f'Either "pr" or "target_branch" parameter must be non-empty')
return set([f.filename for f in changed_files])
def run(cmd: str, dry_run: bool = False, fail_on_error: bool = True):
logger = logging.getLogger('run')
logger.info(cmd)
if dry_run:
return 0, ''
with subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) as proc:
for line in proc.stdout:
logger.info(line.strip())
proc.communicate()
if proc.returncode != 0:
msg = f"Command '{cmd}' returned non-zero exit status {proc.returncode}"
if fail_on_error:
raise RuntimeError(msg)
logger.warning(msg)
return proc.returncode
def name_from_dockerfile(dockerfile: str | Path, dockerfiles_root: str | Path) -> str:
image_name = str(Path(dockerfile).relative_to(dockerfiles_root).parent.as_posix())
return image_name

View File

@ -1,137 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import logging
import os
import subprocess
from pathlib import Path
from typing import Iterable
from helpers import run, name_from_dockerfile
class Image:
def __init__(self, name: str, dockerfile: Path, registry: str):
self.logger = logging.getLogger(self.__class__.__name__)
self.name = name
self.dockerfile = dockerfile
self.registry = registry
self.tag = 'latest'
self.base_tag = None
def __str__(self):
return self.name
def __eq__(self, img):
return img.name == self.name if img else False
def with_tag(self, tag: str):
self.tag = tag
return self
def with_base_tag(self, tag: str):
self.base_tag = tag
return self
def ref(self):
return f"{self.registry}/{self.name}:{self.tag}"
def base_ref(self):
if not self.base_tag:
return None
return f"{self.registry}/{self.name}:{self.base_tag}"
def push(self, dry: bool = False):
cmd = f"docker push {self.ref()} "
run(cmd, dry)
def build(self, dry: bool = False, push: bool = True, docker_builder: str = None, import_cache: bool = True,
export_cache: bool = True):
cache_cmd = ""
if import_cache:
cache_cmd += f"--cache-from type=registry,ref={self.ref()}-cache "
if self.base_tag:
cache_cmd += f"--cache-from type=registry,ref={self.base_ref()}-cache "
if export_cache:
cache_cmd += f"--cache-to type=registry,ref={self.ref()}-cache,mode=max "
build_cmd = f"docker buildx build --builder={docker_builder}" if docker_builder else "docker build"
push_cmd = f"--push" if push else ""
cmd = f"{build_cmd} " \
f"--file {self.dockerfile} " \
f"--tag {self.ref()} " \
f"{cache_cmd} " \
f"{push_cmd} " \
"."
run(cmd, dry)
def tag_base(self, dry: bool = False):
if not self.base_tag:
raise AttributeError("Tag for base image is not specified")
cmd = f"docker buildx imagetools create -t {self.ref()} {self.base_ref()}"
run(cmd, dry)
def is_missing(self, dry: bool = False, base: bool = False) -> bool:
image = self.base_ref() if base else self.ref()
if base and not image:
self.logger.warning(f"Base ref for image {self.ref()} is missing")
return True
cmd = f"docker manifest inspect {image}"
is_missing = False
self.logger.info(cmd)
if not dry:
try:
subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True, text=True)
except subprocess.CalledProcessError:
self.logger.warning(f"{image} is missing in registry")
is_missing = True
return is_missing
# Making it a class, so it's a little easier to switch to a tree structure for building inherited images if we want
class ImagesHandler:
def __init__(self, dry_run: bool = False):
self.logger = logging.getLogger(self.__class__.__name__)
self.images = dict()
self.dry_run = dry_run
def add_from_dockerfile(self, dockerfile: str | Path, dockerfiles_root: str | Path, registry: str, tag: str,
base_tag: str = None):
image_name = name_from_dockerfile(dockerfile, dockerfiles_root)
image = Image(image_name, Path(dockerfile), registry).with_tag(tag).with_base_tag(base_tag)
self.add(image)
def add(self, image: Image):
self.images[image.name] = image
def get(self, image_names: Iterable = None) -> list:
images = [self.images[name] for name in image_names] if image_names is not None else self.images.values()
return images
def get_missing(self, image_names: Iterable = None, base: bool = False) -> list:
missing_images = [image.name for image in self.get(image_names) if image.is_missing(self.dry_run, base)]
return missing_images
def build(self, image_names: Iterable = None, missing_only: bool = False, push: bool = True, builder: str = None,
import_cache: bool = True, export_cache: bool = True):
to_build = self.get(self.get_missing(image_names)) if missing_only else self.get(image_names)
for image in to_build:
image.build(self.dry_run, push, builder, import_cache, export_cache)
return to_build
def push(self, image_names: Iterable = None, missing_only: bool = False):
to_push = self.get(self.get_missing(image_names)) if missing_only else self.get(image_names)
for image in to_push:
image.push(self.dry_run)
def tag(self, image_names: Iterable = None):
for image in self.get(image_names):
image.tag_base(self.dry_run)

View File

@ -1 +0,0 @@
ghapi~=1.0.5

View File

@ -23,19 +23,7 @@ runs:
using: 'composite'
steps:
- name: Check if Python is already installed (Linux)
if: ${{ runner.os == 'Linux' }}
shell: bash
id: check_python
run: |
PYTHON_INSTALLED=$(python${{ inputs.version }} -V) || true
if [[ $PYTHON_INSTALLED ]]; then
echo "installed=true" >> $GITHUB_OUTPUT
else
echo "installed=false" >> $GITHUB_OUTPUT
fi
- if: ${{ runner.os == 'Linux' && inputs.self-hosted-runner == 'true' && steps.check_python.outputs.installed == 'false' }}
- if: ${{ runner.os == 'Linux' && inputs.self-hosted-runner == 'true' }}
name: Install 'actions/setup-python@v4' dependencies
shell: bash
run: apt-get update && apt-get install -y ca-certificates software-properties-common gpg-agent tzdata
@ -43,18 +31,20 @@ runs:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
TZ: "Europe/London" # to prevent tzdata from waiting user input
- if: ${{ runner.os == 'Linux' && runner.arch == 'ARM64' && steps.check_python.outputs.installed == 'false' }}
- if: ${{ runner.os == 'Linux' && runner.arch == 'ARM64' }}
name: Setup sudo and python3
shell: bash
run: apt-get update && apt-get install -y sudo python3 # Needed for the deadsnakes action
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
- if: ${{ runner.os == 'Linux' && runner.arch == 'ARM64' && steps.check_python.outputs.installed == 'false' }}
- if: ${{ runner.os == 'Linux' && runner.arch == 'ARM64' }}
name: Setup Python ${{ inputs.version }}
uses: akashchi/deadsnakes-action@92417281055a5878a0450f240a5b95883eb2d7e2
with:
python-version: ${{ inputs.version }}
- if: ${{ runner.os == 'macOS' || runner.os == 'Windows' || (runner.os == 'Linux' && runner.arch != 'ARM64' && steps.check_python.outputs.installed == 'false' ) }}
- if: ${{ runner.os == 'macOS' || runner.os == 'Windows' || (runner.os == 'Linux' && runner.arch != 'ARM64') }}
name: Setup Python ${{ inputs.version }}
uses: actions/setup-python@v5
with:

View File

@ -13,9 +13,6 @@ inputs:
commit_sha:
description: "GitHub commit hash. Used if no PR number is set"
required: false
ref_name:
description: "GitHub ref name"
required: false
component_pattern:
description: "Pattern to extract component name from PR label. If not set, any label is considered a component name"
required: false
@ -51,9 +48,6 @@ outputs:
affected_components:
description: "Affected components to run validation for and their validation scope"
value: ${{ steps.smart_ci.outputs.affected_components }}
changed_components:
description: "Actually changed components (for push events everything is marked as changed)"
value: ${{ steps.smart_ci.outputs.changed_components }}
skip_workflow:
description: "Whether the workflow should be run with Smart CI rules applied or skipped completely"
value: ${{ steps.smart_ci.outputs.skip_workflow }}
@ -94,7 +88,6 @@ runs:
python ${{ github.action_path }}/smart_ci.py \
$([[ -n "${{ inputs.pr }}" ]] && echo '--pr ${{ inputs.pr }}' || echo '-s ${{ inputs.commit_sha }}') \
-r ${{ inputs.repository }} \
-f "${{ inputs.ref_name }}" \
-p "${{ inputs.component_pattern }}" \
-c "${{ inputs.components_config }}" \
-m "${{ inputs.components_config_schema }}" \

View File

@ -1,6 +1,3 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import os
import re
import argparse
@ -112,26 +109,11 @@ def get_changed_component_names(pr, all_possible_components: set, component_patt
return components
def get_changeset(gh_api, pr, target_branch, commit_sha):
"""Returns changeset either from PR or commit"""
if pr:
return gh_api.pulls.list_files(pr)
if target_branch:
target_branch_head_commit = gh_api.repos.get_branch(target_branch).commit.sha
# In merge-queue branch all commits between head of target branch and head of current branch (commit_sha)
# contain changes added to queue earlier to be validated together. Getting all of them + changes from
# commit_sha below
changed_files = gh_api.repos.compare_commits(f'{target_branch_head_commit}...{commit_sha}').get('files', [])
return changed_files
raise ValueError(f'Either "pr" or "target_branch" parameter must be non-empty')
def parse_args():
parser = argparse.ArgumentParser(description='Returns product components changed in a given PR or commit')
parser.add_argument('--pr', type=int, required=False, help='PR number. If not set, --commit is used')
parser.add_argument('-s', '--commit-sha', required=False, help='Commit SHA. If not set, --pr is used')
parser.add_argument('-r', '--repo', help='GitHub repository')
parser.add_argument('-f', '--ref_name', required=False, help='GitHub ref name')
parser.add_argument('-p', '--pattern', default=None, help='Pattern to extract component name from PR label. '
'If not set, any label is considered a component name')
parser.add_argument('-c', '--components-config', default='.github/components.yml',
@ -190,33 +172,24 @@ def main():
component_name = component_name_from_label(label, args.pattern)
all_possible_components.add(component_name if component_name else label)
run_full_scope = False
no_match_files_changed = False
# For now, we don't want to apply smart ci rules for post-commits
is_postcommit = not pr
merge_queue_prefix = 'gh-readonly-queue/'
is_merge_queue = args.ref_name.startswith(merge_queue_prefix)
merge_queue_target_branch = re.findall(f'^{merge_queue_prefix}(.*)/', args.ref_name)[0] if is_merge_queue else None
if is_merge_queue:
logger.info(f"The run is a merge-queue run, executing full validation scope for all components, if "
f"not all queued changes match patterns in 'skip-when-only-listed-files-changed'")
run_full_scope = True
elif is_postcommit:
if is_postcommit:
logger.info(f"The run is a post-commit run, executing full validation scope for all components")
run_full_scope = True
else:
no_match_files_changed = 'no-match-files' in [label.name for label in pr.labels]
if no_match_files_changed:
logger.info(f"There are changed files that don't match any pattern in labeler config, "
f"executing full validation scope for all components")
run_full_scope = True
run_full_scope = is_postcommit or no_match_files_changed
# In post-commits - validate all components regardless of changeset
# In pre-commits - validate only changed components with their dependencies
all_defined_components = components_config.keys()
changed_by_pr = get_changed_component_names(pr, all_possible_components, args.pattern) if pr else None
changed_component_names = set(all_defined_components) if run_full_scope else changed_by_pr
changed_component_names = set(all_defined_components) if run_full_scope else \
get_changed_component_names(pr, all_possible_components, args.pattern)
logger.info(f"changed_component_names: {changed_component_names}")
@ -224,7 +197,7 @@ def main():
affected_components = cfg.get_affected_components(changed_component_names)
skip_workflow = False
if is_merge_queue or (args.pr and not run_full_scope):
if args.pr and not run_full_scope:
if args.skip_when_only_listed_labels_set:
excepted_labels = set(args.skip_when_only_listed_labels_set.split(','))
excepted_labels_only = changed_component_names - excepted_labels == set()
@ -232,7 +205,7 @@ def main():
if not skip_workflow and args.skip_when_only_listed_files_changed:
# To avoid spending extra API requests running step below only if necessary
changed_files = get_changeset(gh_api, args.pr, merge_queue_target_branch, args.commit_sha)
changed_files = gh_api.pulls.list_files(args.pr)
patterns = set(args.skip_when_only_listed_files_changed.split(','))
matched_files_only = all(any(fnmatch(f.filename, pattern) for pattern in patterns) for f in changed_files)
@ -247,12 +220,6 @@ def main():
affected_components_output = {name: {s: True for s in scope} for name, scope in affected_components.items()}
set_github_output("affected_components", json.dumps(affected_components_output))
# Components actually changed by a pull request are marked as True (if event is PR;
# otherwise all components considered changed).
changed_components_output = {name: True if not pr or name in changed_by_pr else False
for name in all_possible_components}
set_github_output("changed_components", json.dumps(changed_components_output))
if __name__ == '__main__':
main()

View File

@ -1,6 +1,3 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import logging
import sys
import unittest

View File

@ -13,10 +13,12 @@ LP_transformations:
- TFL_FE
- ONNX_FE
- PDPD_FE
- POT
preprocessing:
revalidate:
- inference
- GNA
- C_API
- Python_API
@ -27,13 +29,13 @@ CPU:
revalidate:
- C_API
- Python_API
- JS_API
- samples
- ONNX_RT
- PyTorch_FE
- TF_FE
- ONNX_FE
build:
- AUTO
- HETERO
- AUTO_BATCH
- TEMPLATE
@ -41,39 +43,53 @@ CPU:
GPU:
build:
- AUTO
- HETERO
- AUTO_BATCH
- TEMPLATE
- IR_FE
- PROXY
NPU:
revalidate: []
build: []
GNA:
build:
- HETERO
- AUTO_BATCH
- TEMPLATE
- IR_FE
HETERO:
revalidate:
- CPU
- GPU
- GNA
- HETERO
- AUTO_BATCH
- TEMPLATE
- C_API
- Python_API
build:
- TEMPLATE
- IR_FE
AUTO_BATCH:
revalidate:
- CPU
- GPU
- GNA
- HETERO
- AUTO_BATCH
- TEMPLATE
- C_API
- Python_API
build:
- TEMPLATE
- IR_FE
TEMPLATE:
revalidate:
- CPU
- GPU
- GNA
- HETERO
- AUTO_BATCH
- TEMPLATE
- AUTO
- C_API
- Python_API
@ -83,10 +99,11 @@ TEMPLATE:
AUTO:
revalidate:
- TEMPLATE
- AUTO
- C_API
- Python_API
build:
- TEMPLATE
- IR_FE
PROXY:
@ -99,6 +116,7 @@ IR_FE:
revalidate:
- C_API
- Python_API
- JS_API
- samples
build:
- CPU
@ -106,7 +124,6 @@ IR_FE:
ONNX_FE:
revalidate:
- MO
- ONNX_RT
build:
- CPU
- Python_API
@ -124,7 +141,6 @@ TF_FE:
build:
- CPU
- Python_API
- TOKENIZERS
TFL_FE:
revalidate:
@ -139,12 +155,10 @@ PyTorch_FE:
build:
- CPU
- Python_API
- TOKENIZERS
C_API:
build:
- CPU
- GPU
- HETERO
- AUTO_BATCH
- AUTO
@ -154,6 +168,7 @@ Python_API:
revalidate:
- samples
- MO
- POT
- tools
build:
- CPU
@ -169,6 +184,8 @@ Python_API:
- PyTorch_FE
JS_API:
revalidate:
- samples
build:
- CPU
- IR_FE
@ -186,6 +203,7 @@ IE_Tests:
revalidate:
- CPU
- GPU
- GNA
- HETERO
- AUTO_BATCH
- TEMPLATE
@ -201,6 +219,11 @@ MO:
build:
- Python_API
POT:
build:
- CPU
- Python_API
tools:
build:
- CPU
@ -221,7 +244,3 @@ NVIDIA:
ONNX_RT:
revalidate: []
build: []
TOKENIZERS:
revalidate: []
build: []

View File

@ -61,6 +61,23 @@ updates:
dependency-type: "production"
versioning-strategy: increase-if-necessary
# POT requirements
- package-ecosystem: pip
directory: "/tools/pot"
schedule:
interval: "daily"
time: "09:00"
timezone: "Asia/Dubai"
open-pull-requests-limit: 3
assignees:
- "AlexKoff88"
- "KodiaqQ"
- "jiwaszki"
- "p-wysocki"
- "akuporos"
- "rkazants"
versioning-strategy: increase-if-necessary
#
# Python Samples
#

View File

@ -10,13 +10,9 @@ allow-licenses:
- 'BlueOak-1.0.0'
- '0BSD'
- 'Python-2.0'
- 'LGPL-3.0'
fail-on-scopes:
- 'runtime'
- 'development'
- 'unknown'
license-check: true
vulnerability-check: true
allow-dependencies-licenses:
- 'pkg:pypi/PyGithub@2.2.0'
- 'pkg:pypi/psycopg2-binary'

View File

@ -1 +0,0 @@
pr-24878

View File

@ -1,73 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
libhdf5-dev \
# For Java API
default-jdk \
# Compiler
gcc-10 \
g++-10 \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set gcc-10 as a default compiler
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 30 && \
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 30
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-aarch64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,72 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
# For Java API
default-jdk \
# Compiler \
gcc-10 \
g++-10 \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set gcc-10 as a default compiler
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 30 && \
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 30
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,89 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/nvidia/cuda:11.8.0-runtime-ubuntu20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
wget \
git \
ca-certificates \
gpg-agent \
tzdata \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
# For Java API
default-jdk \
# Compiler \
gcc-10 \
g++-10 \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set gcc-10 as a default compiler
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 30 && \
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 30
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Install CUDA
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin && \
mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \
apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub && \
add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
RUN apt update && apt install -y \
libcudnn8=8.9.4.*-1+cuda11.8 \
libcudnn8-dev=8.9.4.*-1+cuda11.8 \
libcudnn8-samples=8.9.4.*-1+cuda11.8 \
cuda-runtime-11-8 \
cuda-11-8 \
libcutensor1=1.6.1.5-1 \
libcutensor-dev=1.6.1.5-1 \
cuda-drivers=520.61.05-1 && \
rm -rf /var/lib/apt/lists/*
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,73 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:22.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
libtbb2 \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
# For Java API
default-jdk \
# Compiler \
gcc-10 \
g++-10 \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set gcc-10 as a default compiler
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 30 && \
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 30
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,72 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:22.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
libtbb2 \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
# For Java API
default-jdk \
# Compiler \
clang \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set clang as a default compiler
RUN update-alternatives --install /usr/bin/cc cc /usr/bin/clang 100 && \
update-alternatives --install /usr/bin/c++ c++ /usr/bin/clang++ 100
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,51 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Python
python3.11-dev \
python3.11-venv \
python3.11-distutils \
libhdf5-dev \
&& \
rm -rf /var/lib/apt/lists/*
# Install openvino dependencies
ADD scripts/install_dependencies/install_openvino_dependencies.sh /install_openvino_dependencies.sh
RUN chmod +x /install_openvino_dependencies.sh && \
/install_openvino_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}
ENV PIP_INSTALL_PATH=/venv/lib/python3.11/site-packages

View File

@ -1,52 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Python
python3.11-dev \
python3.11-venv \
python3.11-distutils \
libhdf5-dev \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}
ENV PIP_INSTALL_PATH=/venv/lib/python3.11/site-packages

View File

@ -1,52 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:22.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Python
python3.11-dev \
python3.11-venv \
python3.11-distutils \
libhdf5-dev \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}
ENV PIP_INSTALL_PATH=/venv/lib/python3.11/site-packages

View File

@ -6,6 +6,7 @@
"openvino-ci",
"openvino-pushbot",
"workbench-ci-bot",
"openvino-pot-ci",
"sysicvvpux",
"ote-ci-bot"
],
@ -23,6 +24,7 @@
"openvino-docs-maintainers": "category: docs",
"openvino-ie-maintainers": "category: inference",
"openvino-ie-cpu-maintainers": "category: CPU",
"openvino-ie-gna-maintainers": "category: GNA",
"openvino-ie-gpu-maintainers": "category: GPU",
"openvino-ie-lpt-maintainers": "category: LP transformations",
"openvino-ie-transformations-maintainers": "category: transformations",
@ -41,6 +43,7 @@
"openvino-scripts-maintainers": "category: build",
"openvino-tests-maintainers": "category: IE Tests",
"openvino-tools-maintainers": "category: tools",
"openvino-pot-maintainers": "category: POT",
"openvino-configuration-mgmt": "category: dependency_changes",
"openvino-samples-maintainers": "category: samples",
"openvino-c-api-maintainers": "category: C API"

26
.github/labeler.yml vendored
View File

@ -67,13 +67,13 @@
- 'src/frontends/common/include/openvino/frontend/extension.hpp'
- 'src/frontends/common/include/openvino/frontend/extension/**/*'
'category: GNA':
- 'src/plugins/intel_gna/**/*'
'category: GPU':
- 'src/plugins/intel_gpu/**/*'
- 'thirdparty/ocl/**/*'
'category: NPU':
- 'src/plugins/intel_npu/**/*'
'category: HETERO':
- 'src/plugins/hetero/**/*'
@ -118,6 +118,9 @@
- 'src/frontends/paddle/**/*'
- 'tests/layer_tests/py_frontend_tests/test_paddle_frontend.py'
'category: POT':
- 'tools/pot/**/*'
'category: preprocessing':
- 'src/common/preprocessing/**/*'
@ -145,7 +148,7 @@
- 'tests/layer_tests/tensorflow2_keras_tests/**/*'
- 'tests/layer_tests/jax_tests/**/*'
- any: ['tests/model_hub_tests/**',
'!tests/model_hub_tests/pytorch/**/*']
'!tests/model_hub_tests/torch_tests/**/*']
'category: TFL FE':
- 'src/frontends/tensorflow_lite/**/*'
@ -159,29 +162,18 @@
- 'tests/layer_tests/py_frontend_tests/test_torch_decoder.py'
- 'tests/layer_tests/py_frontend_tests/test_torch_frontend.py'
- any: ['tests/model_hub_tests/**',
'!tests/model_hub_tests/tensorflow/**/*']
'!tests/model_hub_tests/tf_hub_tests/**/*']
'category: tools':
- any: ['tools/**',
'!tools/pot/**/*',
'!tools/mo/**/*',
'!tools/ovc/**/*']
'category: transformations':
- 'src/common/transformations/**/*'
- 'src/common/offline_transformations/**/*'
- 'src/core/pattern/**/*'
- 'src/core/pass/**/*'
- 'src/core/include/openvino/pass/**/*'
'category: licensing':
- 'licensing/**/*'
- 'LICENSE'
'category: docker_env':
- '.dockerignore'
- 'install_build_dependencies.sh'
- '**/install_openvino_dependencies.sh'
- '.github/actions/handle_docker/**/*'
'category: dockerfiles':
- '.github/dockerfiles/**/*'

View File

@ -1,225 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
#!/usr/bin/env python3
from github import Github
from psycopg2 import sql
import os
import logging
import psycopg2
import dateutil
import argparse
def init_logger():
LOGLEVEL = os.environ.get('LOGLEVEL', 'INFO').upper()
logging.basicConfig(level=LOGLEVEL,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d-%Y %H:%M:%S')
def make_parser():
parser = argparse.ArgumentParser()
parser.add_argument('-r', '--repository-name', type=str, required=True,
help='Repository name in OWNER/REPOSITORY format')
parser.add_argument('--run-id', type=str, required=True,
help='Workflow Run ID')
return parser
def create_db_tables(conn, cur):
cur.execute('''CREATE TABLE IF NOT EXISTS workflow_runs(
id SERIAL PRIMARY KEY,
run_id BIGINT,
html_url TEXT,
name VARCHAR(255),
run_started_at TIMESTAMP,
created_at TIMESTAMP,
updated_at TIMESTAMP,
triggering_actor_login VARCHAR(255),
conclusion VARCHAR(25),
event VARCHAR(50),
run_attempt INT,
repository_full_name VARCHAR(255),
head_repository_full_name VARCHAR(255),
head_branch VARCHAR(255),
status VARCHAR(25),
display_title TEXT,
path TEXT,
total_duration_seconds INT
);
''')
cur.execute('''CREATE TABLE IF NOT EXISTS workflow_jobs(
id SERIAL PRIMARY KEY,
job_id BIGINT,
parent_run_id BIGINT,
html_url TEXT,
name VARCHAR(255),
created_at TIMESTAMP,
started_at TIMESTAMP,
completed_at TIMESTAMP,
queued_duration_seconds INT,
duration_seconds INT,
runner_name VARCHAR(255),
status VARCHAR(25),
conclusion VARCHAR(25),
head_branch VARCHAR(255),
run_attempt INT,
workflow_name TEXT
);
''')
cur.execute('''CREATE TABLE IF NOT EXISTS workflow_steps(
id SERIAL PRIMARY KEY,
parent_job_id BIGINT,
name VARCHAR(255),
conclusion VARCHAR(25),
number INT,
started_at TIMESTAMP,
completed_at TIMESTAMP,
duration_seconds INT
);
''')
conn.commit()
def main():
init_logger()
parser = make_parser()
args = parser.parse_args()
logger = logging.getLogger(__name__)
github_token = os.environ.get('GITHUB_TOKEN')
if not github_token:
raise ValueError('GITHUB_TOKEN environment variable is not set!')
run_id = args.run_id
repo_name = args.repository_name
# this should be specified in runner's env
db_username = os.environ.get('PGUSER')
db_password = os.environ.get('PGPASSWORD')
db_host = os.environ.get('PGHOST')
db_database = os.environ.get('PGDATABASE')
db_port = os.environ.get('PGPORT')
conn = psycopg2.connect(host=db_host,
port=db_port,
user=db_username,
password=db_password,
database=db_database)
# Create tables
cur = conn.cursor()
create_db_tables(conn, cur)
# Get the data
g = Github(github_token)
repo = g.get_repo(repo_name)
run = repo.get_workflow_run(int(run_id))
logger.info('Processing run ID %s - %s, URL: %s', run_id, run.name, run.html_url)
if run.status != 'completed':
logger.error('Run %s is not completed! Only completed runs should be in the database', run_id)
raise SystemExit(1)
# We rely on the following assumptions:
# - The workflow run is completed. When run.status != 'completed' we should not add it to the database
# theoretically the second attempt can be triggerred right after the completion of the first one
# or while the runner which executes this script is deploying
#
# - Job's queued duration equals "job.started_at - job.created_at" if started_at > created_at.
# Otherwise the job should not be added to the database
total_duration_seconds = round(run.timing().run_duration_ms / 1000)
workflow_data_query = sql.SQL('''INSERT INTO workflow_runs(
run_id, html_url, name,
run_started_at, created_at, updated_at, triggering_actor_login, conclusion,
event, run_attempt, repository_full_name, head_repository_full_name,
head_branch, status, display_title, path, total_duration_seconds)
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);
''')
workflow_query_args = (run_id, run.html_url, run.name,
run.run_started_at, run.created_at, run.updated_at,
run.raw_data['triggering_actor']['login'],
run.conclusion, run.event,
run.run_attempt, run.raw_data['repository']['full_name'],
run.raw_data['head_repository']['full_name'],
run.head_branch, run.status,
run.display_title, run.path, total_duration_seconds)
logger.debug('Workflow run query: %s', cur.mogrify(workflow_data_query, workflow_query_args))
cur.execute(workflow_data_query, workflow_query_args)
for job in run.jobs():
job_id = job.id
logger.info('Processing job %s, URL: %s', job.name, job.html_url)
queued_duration_seconds = 0
duration_seconds = 0
job_created_at_date = dateutil.parser.parse(job.raw_data['created_at'])
if job_created_at_date > job.started_at:
logger.warning('Skipping job %s of run %s - most likely a stub \
job created after workflow restart', job.name, run_id)
continue
queued_duration_timedelta = job.started_at - job_created_at_date
queued_duration_seconds = round(queued_duration_timedelta.total_seconds())
duration_timedelta = job.completed_at - job.started_at
duration_seconds = round(duration_timedelta.total_seconds())
job_data_query = sql.SQL('''
INSERT INTO workflow_jobs(
job_id, parent_run_id, html_url, name,
created_at, started_at, completed_at,
queued_duration_seconds, duration_seconds,
runner_name, status, conclusion, head_branch,
run_attempt, workflow_name
)
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);
''')
job_query_args = (job_id, run_id, job.html_url, job.name,
job.raw_data['created_at'], job.started_at, job.completed_at,
queued_duration_seconds, duration_seconds,
job.raw_data['runner_name'], job.status, job.conclusion,
job.raw_data['head_branch'], job.raw_data['run_attempt'], job.raw_data['workflow_name'])
logger.debug('Job query: %s', cur.mogrify(job_data_query, job_query_args))
cur.execute(job_data_query, job_query_args)
conn.commit()
for step in job.steps:
logger.info('Processing step %s', step.name)
# It seems like a GitHub bug, but
# sometimes there're steps "In progress"
# despite the workflow being completed
step_duration_seconds = 0
if step.status != 'completed':
logger.warning('Step %s isn\'t completed. Setting duration to zero.', step.name)
step_duration_seconds = 0
else:
step_duration_seconds_timedelta = step.completed_at - step.started_at
step_duration_seconds = round(step_duration_seconds_timedelta.total_seconds())
step_data_query = sql.SQL('''
INSERT INTO workflow_steps(
parent_job_id, name, conclusion,
number, started_at, completed_at,
duration_seconds)
VALUES (%s, %s, %s, %s, %s, %s, %s);
''')
step_query_args = (
job_id, step.name, step.conclusion,
step.number, step.started_at, step.completed_at,
step_duration_seconds)
logger.debug('Step query: %s', cur.mogrify(step_data_query, step_query_args))
cur.execute(step_data_query, step_query_args)
conn.commit()
cur.close()
conn.close()
g.close()
if __name__ == "__main__":
main()

View File

@ -1,75 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import argparse
import logging
import os
from enum import Enum
from github import Github, Auth
class ExternalPRLabels(str, Enum):
ExternalPR = 'ExternalPR'
ExternalIntelPR = 'ExternalIntelPR'
def get_arguments() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument(
'-r',
'--repository-name',
type=str,
required=True,
help='Repository name in the OWNER/REPOSITORY format',
)
parser.add_argument(
'--pr-number', type=int, required=True, help='PR number to label'
)
return parser.parse_args()
def init_logger():
logging.basicConfig(
level=os.environ.get('LOGLEVEL', 'INFO').upper(),
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d-%Y %H:%M:%S',
)
if __name__ == '__main__':
init_logger()
LOGGER = logging.getLogger('labeller')
args = get_arguments()
pr_number = args.pr_number
repository_name = args.repository_name
github = Github(auth=Auth.Token(token=os.environ.get('GITHUB_TOKEN')))
gh_repo = github.get_repo(full_name_or_id=repository_name)
pr = gh_repo.get_pull(number=pr_number)
LOGGER.info(f'CONTEXT: PR #{pr_number}. USER: {pr.user.login}. ALL PR LABELS: {list(pr.get_labels())}')
if not gh_repo.has_in_collaborators(pr.user.login):
LOGGER.info(f'THE {pr.user.login} IS NOT A COLLABORATOR')
for label in pr.get_labels():
if label.name in (ExternalPRLabels.ExternalPR, ExternalPRLabels.ExternalIntelPR):
LOGGER.info(f'THE PR ALREADY HAS THE "{label.name}" LABEL')
break
else:
is_intel_user = bool(pr.user.email and pr.user.email.endswith('@intel.com'))
label_to_add: str = ExternalPRLabels.ExternalIntelPR.name if is_intel_user else ExternalPRLabels.ExternalPR.name
pr.add_to_labels(label_to_add)
LOGGER.info(f'THE "{label_to_add}" LABEL WAS ADDED TO THE PR')
else:
LOGGER.info(
f'THE {pr.user.login} IS A COLLABORATOR, NO NEED TO ADD THE "External" LABEL'
)
github.close()

View File

@ -1 +0,0 @@
python-dateutil==2.9.0.post0

View File

@ -1,28 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import argparse
from pathlib import Path
def get_arguments() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument('-r', '--repository-name',
type=str,
required=True,
help='Repository name in the OWNER/REPOSITORY format')
parser.add_argument('--run-id',
type=int,
required=True,
help='Workflow Run ID')
parser.add_argument('--errors-to-look-for-file',
type=Path,
required=False,
help='.json file with the errors to look for in logs',
default=Path(__file__).resolve().parent.joinpath('errors_to_look_for.json'))
parser.add_argument('--dry-run',
required=False,
action='store_true',
help='Whether to run in dry mode and not actually retrigger the pipeline'
' and only collect and analyze logs')
return parser.parse_args()

View File

@ -1,20 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import logging
import os
GITHUB_TOKEN = os.environ.get('GITHUB_TOKEN')
def init_logger():
LOGLEVEL = os.environ.get('LOGLEVEL', 'INFO').upper()
logging.basicConfig(level=LOGLEVEL,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d-%Y %H:%M:%S')
init_logger()
LOGGER = logging.getLogger('rerunner')

View File

@ -1,62 +0,0 @@
[
{
"error_text": "This is a problem related to network connectivity",
"ticket": 135929
},
{
"error_text": "Unable to make request",
"ticket": 135715
},
{
"error_text": "GnuTLS recv error",
"ticket": 131918
},
{
"error_text": "Connection was reset",
"ticket": 131818
},
{
"error_text": "Failed to connect to github.com",
"ticket": 131657
},
{
"error_text": "Could not resolve host: github.com",
"ticket": 131546
},
{
"error_text": "retrieving gpg key timed out",
"ticket": 131538
},
{
"error_text": "Retry limit has been reached for chunk",
"ticket": 131537
},
{
"error_text": "fatal error: downloading",
"ticket": 131424
},
{
"error_text": "Failure when receiving data from the peer",
"ticket": 137121
},
{
"error_text": "Network is unreachable",
"ticket": 130955
},
{
"error_text": "connection timed out",
"ticket": 130955
},
{
"error_text": "The requested URL returned error: 500",
"ticket": 139384
},
{
"error_text": "Unable to fetch some archives",
"ticket": 130965
},
{
"error_text": "status_string: \"Timeout was reached\"",
"ticket": 142653
}
]

View File

@ -1,137 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import json
import re
import tempfile
from pathlib import Path
from typing import TypedDict
from zipfile import ZipFile
from workflow_rerun.constants import LOGGER
class LogFile(TypedDict):
file_name: str
path: Path
class ErrorData(TypedDict):
error_text: str
ticket: int
class LogAnalyzer:
def __init__(self,
path_to_log_archive: Path,
path_to_errors_file: Path) -> None:
self._path_to_log_archive = path_to_log_archive
self._path_to_errors_file = path_to_errors_file
self._errors_to_look_for: list[ErrorData] = []
self._collect_errors_to_look_for()
self._log_dir = tempfile.TemporaryDirectory().name
self._log_files: list[LogFile] = []
self._collect_log_files()
all_txt_log_files_pretty = '\n'.join(map(lambda item: str(item['path']), self._log_files))
LOGGER.info(f'ALL .txt LOG FILES: \n{all_txt_log_files_pretty}')
self.found_matching_error = False
self.found_error_ticket = None
def _collect_errors_to_look_for(self) -> None:
with open(file=self._path_to_errors_file,
mode='r',
encoding='utf-8') as errors_file:
errors_data = json.load(errors_file)
for error_data in errors_data:
self._errors_to_look_for.append(
ErrorData(error_text=error_data['error_text'],
ticket=error_data['ticket'])
)
def _collect_log_files(self) -> None:
"""
Collects the .txt log files from the log archive
The GitHub Actions pipeline logs archive should have the following structure:
> Job_name_0
> step_name_0.txt
> step_name_1.txt
...
> Job_name_1
> step_name_0.txt
> step_name_1.txt
...
> Job_name_2
...
...
We need to only analyze the `*.txt` files
"""
with ZipFile(file=self._path_to_log_archive,
mode='r') as zip_file:
zip_file.extractall(self._log_dir)
for _file in Path(self._log_dir).iterdir():
if _file.is_dir():
for log_file in _file.iterdir():
self._log_files.append(LogFile(file_name=log_file.name,
path=log_file.resolve()))
def _is_error_in_log(self,
error_to_look_for: str,
log_file_path: Path) -> bool:
"""
Searches for the error in the provided log
"""
error_to_look_for = self._clean_up_string(error_to_look_for)
with open(file=log_file_path,
mode='r',
encoding='utf-8') as log_file:
for line in log_file:
if error_to_look_for in self._clean_up_string(line):
return True
return False
@staticmethod
def _clean_up_string(string: str) -> str:
"""
Replaces special characters with spaces in the string, strips it from leading and following spaces,
and lowers it
for "Could not resolve host: github.com" returns "could not resolve host github com"
This cleanup is applied to both errors to look for and logs themselves for matching
"""
return re.sub(r'[^A-Za-z0-9]+', ' ', string).lower().strip()
def analyze(self) -> None:
"""
Iterates over the known errors and tries to find them in the collected log files
"""
for error in self._errors_to_look_for:
LOGGER.info(f'LOOKING FOR "{error["error_text"]}" ERROR...')
for log_file in self._log_files:
if self._is_error_in_log(error_to_look_for=error['error_text'],
log_file_path=log_file['path']):
LOGGER.info(f'FOUND "{error["error_text"]}" ERROR IN {log_file["path"]}. TICKET: {error["ticket"]}')
self.found_matching_error = True
self.found_error_ticket = error['ticket']
return
if __name__ == '__main__':
# Usage example
log_analyzer = LogAnalyzer(path_to_log_archive=Path('/tmp/logs/log.zip'),
path_to_errors_file=Path('/tmp/errors_to_look_for.json'))
log_analyzer.analyze()
if log_analyzer.found_matching_error:
print('found matching error, see logs above')

View File

@ -1,24 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
from pathlib import Path
import requests
from github.WorkflowRun import WorkflowRun
from workflow_rerun.constants import GITHUB_TOKEN, LOGGER
def collect_logs_for_run(run: WorkflowRun,
log_archive_path: Path) -> Path:
"""
Collects log archive for a pipeline
"""
with open(file=log_archive_path,
mode='wb') as log_archive:
LOGGER.info(f'STARTED LOG COLLECTION FOR {run.id} IN {log_archive_path}')
# PyGitHub does not expose the "/repos/{owner}/{repo}/actions/runs/{run_id}/logs" endpoint so we have to use requests
log_archive.write(requests.get(url=run.logs_url,
headers={'Authorization': f'Bearer {GITHUB_TOKEN}'}).content)
LOGGER.info(f'COLLECTED LOGS FOR {run.id} IN {log_archive_path}')
return log_archive_path

View File

@ -1,71 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import os
import sys
import tempfile
from pathlib import Path
from github import Github, Auth
from workflow_rerun.argument_parser import get_arguments
from workflow_rerun.constants import GITHUB_TOKEN, LOGGER
from workflow_rerun.log_analyzer import LogAnalyzer
from workflow_rerun.log_collector import collect_logs_for_run
if __name__ == '__main__':
args = get_arguments()
run_id = args.run_id
repository_name = args.repository_name
errors_file = args.errors_to_look_for_file
is_dry_run = args.dry_run
if is_dry_run:
LOGGER.info('RUNNING IN DRY RUN MODE. IF ERROR WILL BE FOUND, WILL NOT RETRIGGER')
github = Github(auth=Auth.Token(token=GITHUB_TOKEN))
gh_repo = github.get_repo(full_name_or_id=repository_name)
run = gh_repo.get_workflow_run(id_=run_id)
LOGGER.info(f'CHECKING IF RERUN IS NEEDED FOR {run.html_url} RUN IN {repository_name}.')
# Check if the run has already been retriggered
# we do not want to fall into a loop with retriggers
if run.run_attempt > 1:
LOGGER.info(f'THERE ARE {run.run_attempt} ATTEMPTS ALREADY. NOT CHECKING LOGS AND NOT RETRIGGERING. EXITING')
sys.exit(0)
log_archive_path = Path(tempfile.NamedTemporaryFile(suffix='.zip').name)
collect_logs_for_run(
run=run,
log_archive_path=log_archive_path,
)
log_analyzer = LogAnalyzer(
path_to_log_archive=log_archive_path,
path_to_errors_file=errors_file
)
log_analyzer.analyze()
if log_analyzer.found_matching_error:
LOGGER.info(f'FOUND MATCHING ERROR, RETRIGGERING {run.html_url}')
if is_dry_run:
LOGGER.info(f'RUNNING IN DRY RUN MODE, NOT RETRIGGERING, EXITING')
sys.exit(0)
status = run.rerun()
if status:
LOGGER.info(f'RUN RETRIGGERED SUCCESSFULLY: {run.html_url}')
else:
LOGGER.info(f'RUN WAS NOT RETRIGGERED, SEE ABOVE')
# Needed to run a step after for statistics
with open(file=os.environ['GITHUB_ENV'],
mode='a') as fh:
fh.write('PIPELINE_RETRIGGERED=true\n')
fh.write(f'FOUND_ERROR_TICKET={log_analyzer.found_error_ticket}\n')
# "status" is True (which is 1) if everything is ok, False (which is 0) otherwise
sys.exit(not status)
else:
LOGGER.info(f'NO ERROR WAS FOUND, NOT RETRIGGERING')

View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:894d636bcf156a7f3fae09f3c1d61df6b3db89117a917a3079995805c29115b3
size 89247

View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3f094a737d7ea40dba8d3fb13493275cae243d08e5f1dabce90c316c951a6ac2
size 52047

View File

@ -1,55 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""
Integration tests
"""
import unittest
from pathlib import Path
from github import Github, Auth
import os
import tempfile
from workflow_rerun.log_analyzer import LogAnalyzer
from workflow_rerun.log_collector import collect_logs_for_run
class IntegrationTest(unittest.TestCase):
"""
A class for testing integration between LogAnalyzer and log_collection
"""
def setUp(self) -> None:
print(f'\nIn test: "{self._testMethodName}"', flush=True)
self._cwd = Path(__file__).parent
self.errors_to_look_for_file = self._cwd.parent.joinpath(
'errors_to_look_for.json'
)
self.github = Github(auth=Auth.Token(token=os.environ.get('GITHUB_TOKEN')))
self.gh_repo = self.github.get_repo(full_name_or_id='openvinotoolkit/openvino')
# Even if we use "failure" for status we cannot guarantee logs containing any of the known error
# So these tests use the logs of the most recent successfull pipeline
self.wf_run = self.gh_repo.get_workflow_runs(status='success')[0]
print(f'Workflow run for testing: {self.wf_run}', flush=True)
def test_log_collection_and_analysis(self) -> None:
"""
Ensure logs collected by collect_logs_for_run are analyzed by LogAnalyzer
"""
log_archive_path = Path(tempfile.NamedTemporaryFile(suffix='.zip').name)
collect_logs_for_run(run=self.wf_run,
log_archive_path=log_archive_path)
analyzer = LogAnalyzer(
path_to_log_archive=log_archive_path,
path_to_errors_file=self.errors_to_look_for_file,
)
analyzer.analyze()
self.assertFalse(analyzer.found_matching_error)
def tearDown(self) -> None:
self.github.close()

View File

@ -1,104 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""
LogAnalyzer tests
"""
import unittest
from pathlib import Path
from workflow_rerun.log_analyzer import LogAnalyzer
class LogAnalyzerTest(unittest.TestCase):
"""
A class for testing LogAnalyzer
"""
def setUp(self) -> None:
print(f'\nIn test: "{self._testMethodName}"', flush=True)
self._cwd = Path(__file__).parent
self.log_archive_with_error = self._cwd.joinpath("data").joinpath(
'log_archive_with_error.zip'
)
self.log_archive_wo_error = self._cwd.joinpath("data").joinpath(
'log_archive_wo_error.zip'
)
self.errors_to_look_for_file = self._cwd.parent.joinpath(
'errors_to_look_for.json'
)
def test_log_analyzer_instantiation(self) -> None:
"""
Ensure LogAnalyzer is instantiated correctly.
"""
analyzer = LogAnalyzer(
path_to_log_archive=self.log_archive_wo_error,
path_to_errors_file=self.errors_to_look_for_file,
)
self.assertTrue(
hasattr(analyzer, '_errors_to_look_for'),
'Analyzer should have _errors_to_look_for',
)
self.assertTrue(
hasattr(analyzer, '_log_files'), 'Analyzer should have _log_files'
)
for error_data in analyzer._errors_to_look_for:
self.assertTrue(
error_data['error_text'], 'Each error_data should have text'
)
self.assertTrue(error_data['ticket'], 'Each error_data should have ticket')
for log_file in analyzer._log_files:
self.assertTrue(
log_file['file_name'], 'Each log_file should have file_name'
)
self.assertTrue(log_file['path'], 'Each log_file should have path')
def test_string_cleanup(self) -> None:
"""
Ensure log cleanup function returns correct results
"""
analyzer = LogAnalyzer(
path_to_log_archive=self.log_archive_wo_error,
path_to_errors_file=self.errors_to_look_for_file,
)
data = (
'Connection was reset',
'Failed to connect to github.com',
'Could not resolve host: github.com',
)
expected = (
'connection was reset',
'failed to connect to github com',
'could not resolve host github com',
)
for input_str, expected_str in zip(data, expected):
self.assertEqual(analyzer._clean_up_string(string=input_str), expected_str)
def test_analyzer_with_error(self) -> None:
"""
Ensure LogAnalyzer can find an error
"""
analyzer = LogAnalyzer(
path_to_log_archive=self.log_archive_with_error,
path_to_errors_file=self.errors_to_look_for_file,
)
analyzer.analyze()
self.assertTrue(analyzer.found_matching_error)
def test_analyzer_wo_error(self) -> None:
"""
Ensure LogAnalyzer does not find an error in the log files w/o errors
"""
analyzer = LogAnalyzer(
path_to_log_archive=self.log_archive_wo_error,
path_to_errors_file=self.errors_to_look_for_file,
)
analyzer.analyze()
self.assertFalse(analyzer.found_matching_error)

View File

@ -1,41 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""
log collector tests
"""
import os
import unittest
import tempfile
from pathlib import Path
from github import Github, Auth
from workflow_rerun.log_collector import collect_logs_for_run
class LogCollectorTest(unittest.TestCase):
"""
A class for testing log collection
"""
def setUp(self) -> None:
print(f'\nIn test: "{self._testMethodName}"', flush=True)
self._cwd = Path(__file__).parent
self.github = Github(auth=Auth.Token(token=os.environ.get('GITHUB_TOKEN')))
self.gh_repo = self.github.get_repo(full_name_or_id='openvinotoolkit/openvino')
# Use the logs of the most recent successfull pipeline
self.wf_run = self.gh_repo.get_workflow_runs(status='success')[0]
print(f'Workflow run for testing: {self.wf_run}', flush=True)
def test_log_collection(self) -> None:
"""
Ensure log collection is working
"""
log_archive_path = Path(tempfile.NamedTemporaryFile(suffix='.zip').name)
collect_logs_for_run(run=self.wf_run, log_archive_path=log_archive_path)
self.assertTrue(Path(log_archive_path).exists())
def tearDown(self) -> None:
self.github.close()

View File

@ -2,10 +2,9 @@ name: Android ARM64 with vcpkg
on:
workflow_dispatch:
pull_request:
merge_group:
push:
branches:
# - master
- master
- 'releases/**'
concurrency:
@ -13,8 +12,6 @@ concurrency:
group: ${{ github.event_name == 'push' && github.run_id || github.ref }}-android-arm64-vcpkg
cancel-in-progress: true
permissions: read-all
jobs:
Smart_CI:
runs-on: ubuntu-latest
@ -23,7 +20,7 @@ jobs:
skip_workflow: "${{ steps.smart_ci.outputs.skip_workflow }}"
steps:
- name: checkout action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
sparse-checkout: .github/actions/smart-ci
@ -34,7 +31,6 @@ jobs:
repository: ${{ github.repository }}
pr: ${{ github.event.number }}
commit_sha: ${{ github.sha }}
ref_name: ${{ github.ref_name }}
component_pattern: "category: (.*)"
repo_token: ${{ secrets.GITHUB_TOKEN }}
skip_when_only_listed_labels_set: 'docs'
@ -59,8 +55,6 @@ jobs:
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
SCCACHE_ERROR_LOG: /__w/openvino/sccache_log.txt
SCCACHE_LOG: warn
OPENVINO_REPO: '/__w/openvino/openvino/openvino'
VCPKG_ROOT: '/__w/openvino/openvino/vcpkg'
BUILD_DIR: '/__w/openvino/openvino/build'
@ -77,7 +71,7 @@ jobs:
run: apt-get update && apt-get install --assume-yes --no-install-recommends git ca-certificates
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
path: 'openvino'
@ -92,7 +86,7 @@ jobs:
popd
- name: Clone vcpkg
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
repository: 'microsoft/vcpkg'
# Keep in sync with <root>/vcpkg.json <builtin-baseline>
@ -117,13 +111,13 @@ jobs:
apt --assume-yes install ccache scons ninja-build build-essential python3-pip
# vcpkg requires cmake 3.19 or later
python3 -m pip install -U pip cmake~=3.28.0
python3 -m pip install -U pip cmake
# vcpkg's tool dependencies
apt --assume-yes install curl zip unzip tar
# vcpkg 'python3' port dependencies
apt --assume-yes install autoconf libtool autoconf-archive
# vcpkg tree of dependencies require extra packages
apt --assume-yes install pkgconf linux-libc-dev
apt --assume-yes install pkg-config linux-libc-dev
# Install Android SDK, NDK and Tools
apt -y --no-install-recommends install unzip wget default-jdk
@ -132,7 +126,7 @@ jobs:
echo "yes" | ./cmdline-tools/bin/sdkmanager --sdk_root=${ANDROID_TOOLS} --install "ndk-bundle" "platform-tools" "platforms;android-${{ env.ANDROID_SDK_VERSION }}"
- name: Install sccache
uses: mozilla-actions/sccache-action@2e7f9ec7921547d4b46598398ca573513895d0bd # v0.0.4
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.7.5"
@ -180,17 +174,6 @@ jobs:
- name: Show ccache stats
run: ${SCCACHE_PATH} --show-stats
#
# Upload build logs
#
- name: Upload build logs
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: build_logs
path: ${{ env.SCCACHE_ERROR_LOG }}
if-no-files-found: 'ignore'
Overall_Status:
name: ci/gha_overall_status_android
needs: [Smart_CI, Build]

View File

@ -6,8 +6,6 @@ on:
- created
- edited
permissions: read-all
jobs:
take-issue:
name: Take issue
@ -17,7 +15,7 @@ jobs:
timeout-minutes: 10
steps:
- name: take an issue
uses: bdougie/take-action@1439165ac45a7461c2d89a59952cd7d941964b87 # v1.6.1
uses: bdougie/take-action@v1.6.1
with:
message: Thank you for looking into this issue! Please let us know if you have any questions or require any help.
issueCurrentlyAssignedMessage: Thanks for being interested in this issue. It looks like this ticket is already assigned to a contributor. Please communicate with the assigned contributor to confirm the status of the issue.

View File

@ -1,7 +1,6 @@
name: Documentation
on:
pull_request:
merge_group:
env:
DOXY_VER: '1.9.6'
@ -10,25 +9,23 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
permissions: read-all
jobs:
Build_Doc:
runs-on: ubuntu-20.04
steps:
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
submodules: 'true'
lfs: 'true'
- name: Install apt-get dependencies
uses: awalsh128/cache-apt-pkgs-action@a6c3917cc929dd0345bfb2d3feaf9101823370ad # v1.4.2
uses: awalsh128/cache-apt-pkgs-action@v1.4.1
with:
packages: graphviz texlive liblua5.2-0 libclang1-9 libclang-cpp9
version: 3.0
- uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
- uses: actions/setup-python@v5
id: cp310
with:
python-version: '3.10'
@ -41,7 +38,6 @@ jobs:
run: |
python3 -m pip install -r docs/requirements.txt
(cd docs/openvino_sphinx_theme && python3 setup.py install)
python3 -m pip install docs/openvino_custom_sphinx_sitemap
- name: Download and install doxygen
run: |
@ -53,12 +49,12 @@ jobs:
- name: Build docs
run: |
rm -rf build && mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=ON
cmake .. -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=ON -DENABLE_CPP_API=ON
cmake --build . --target sphinx_docs
- name: Cache documentation
id: cache_sphinx_docs
uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2
uses: actions/cache@v4
with:
path: build/docs/_build/.doctrees
key: sphinx-docs-cache
@ -71,31 +67,39 @@ jobs:
PR_NUMBER=$(echo $GITHUB_REF | awk 'BEGIN { FS = "/" } ; { print $3 }')
echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_ENV
- name: 'Upload doxygen.log'
uses: actions/upload-artifact@v3
with:
name: doxygen_build_log_${{ env.PR_NUMBER }}.log
path: build/docs/doxygen.log
- name: 'Upload sphinx.log'
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: sphinx_build_log_${{ env.PR_NUMBER }}.log
path: build/docs/sphinx.log
- name: 'Upload docs html'
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_docs_html_${{ env.PR_NUMBER }}.zip
path: build/docs/openvino_docs_html.zip
- name: Run Pytest
run: |
pytest --sphinx="./build/docs/sphinx.log" \
pytest --doxygen="./build/docs/doxygen.log" \
--sphinx="./build/docs/sphinx.log" \
--suppress-warnings="./docs/scripts/tests/suppress_warnings.txt" \
--confcutdir="./docs/scripts/tests/" \
--html="./build/docs/_artifacts/doc-generation.html" \
--doxygen-strip="$(pwd)" \
--sphinx-strip="$(pwd)/build/docs/sphinx_source" \
--xfail="./docs/scripts/tests/xfail.txt" \
--self-contained-html ./docs/scripts/tests/test_docs.py
- name: 'Upload test results'
if: failure()
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_docs_pytest
path: build/docs/_artifacts/

View File

@ -1,14 +1,12 @@
name: PR Commits
on: [pull_request]
permissions: read-all
jobs:
Checks:
runs-on: ubuntu-22.04
steps:
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
- name: Install dependencies
run: python3 -m pip install -r ./.github/github_org_control/requirements.txt

View File

@ -1,57 +0,0 @@
name: Cleanup caches
on:
workflow_dispatch:
schedule:
# at 00:00 on the 1st day of every month
- cron: '0 0 1 * *'
permissions: read-all
jobs:
Cleanup_PIP:
runs-on: aks-linux-2-cores-8gb
container:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount:/mount
env:
PIP_CACHE_PATH: /mount/caches/pip
steps:
- name: Pre-Collecting Cache Info
run: |
echo "Cache info: "
du -h -d2 ${PIP_CACHE_PATH}
- name: Cleanup cache
run: |
echo "Delete cache files if they have not been used in over 30 days"
[ ! -z "${PIP_CACHE_PATH}" ] && find ${PIP_CACHE_PATH} ! -type d -atime +30 -delete
- name: Post-Collecting Cache Info
run: |
echo "Cache info: "
du -h -d2 ${PIP_CACHE_PATH}
Cleanup_CCACHE:
runs-on: aks-linux-2-cores-8gb
container:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount:/mount
env:
CCACHE_PATH: /mount/caches/ccache/ubuntu22_riscv64
steps:
- name: Checkout cach action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
sparse-checkout: .github/actions/cache
- name: Cleanup ccache
uses: ./.github/actions/cache/cleanup
with:
cache-size: 10
max-cache-size: 50
cache-path: ${{ env.CCACHE_PATH }}
key: '.'

32
.github/workflows/cleanup_pip_cache.yml vendored Normal file
View File

@ -0,0 +1,32 @@
name: Cleanup PIP caches
on:
workflow_dispatch:
schedule:
# at 00:00 on the 1st day of every month
- cron: '0 0 1 * *'
jobs:
Cleanup_PIP_Caches:
runs-on: aks-linux-2-cores-8gb
container:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount:/mount
env:
PIP_CACHE_PATH: /mount/caches/pip
steps:
- name: Pre-Collecting Cache Info
run: |
echo "Cache info: "
du -h -d2 ${PIP_CACHE_PATH}
- name: Cleanup cache
run: |
echo "Delete cache files if they have not been used in over 30 days"
[ ! -z "${PIP_CACHE_PATH}" ] && find ${PIP_CACHE_PATH} ! -type d -atime +30 -delete
- name: Post-Collecting Cache Info
run: |
echo "Cache info: "
du -h -d2 ${PIP_CACHE_PATH}

View File

@ -16,8 +16,6 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions: read-all
jobs:
Build:
strategy:
@ -27,23 +25,23 @@ jobs:
runs-on: ${{ matrix.os }}
steps:
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
submodules: 'true'
- name: Install OpenCL
uses: awalsh128/cache-apt-pkgs-action@a6c3917cc929dd0345bfb2d3feaf9101823370ad # v1.4.2
uses: awalsh128/cache-apt-pkgs-action@v1.4.1
if: runner.os == 'Linux'
with:
packages: ocl-icd-opencl-dev opencl-headers
version: 3.0
- name: Install scons
if: runner.os == 'macOS'
run: brew install scons
- name: CMake configure
run: cmake -DCMAKE_BUILD_TYPE=Release -DTHREADING=SEQ -B build
- name: Get number of CPU cores
uses: SimenB/github-actions-cpu-cores@v2
id: cpu-cores
- name: Build snippets
run: cmake --build build --target openvino_docs_snippets --parallel
run: cmake --build build --target ie_docs_snippets --parallel ${{ steps.cpu-cores.outputs.count }}

View File

@ -5,22 +5,27 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions: read-all
jobs:
clang-format:
runs-on: ubuntu-22.04
runs-on: ubuntu-20.04
permissions:
pull-requests: write
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@v4
with:
submodules: 'true'
- name: Install clang-format-15
- name: Install clang-format-9
run: |
sudo apt update
sudo apt --assume-yes install clang-format-15
sudo apt --assume-yes install clang-format-9
- name: Install dependencies
run: |
python3 -m pip install --upgrade pip
python3 -m pip install -r ./src/bindings/python/requirements.txt
# Add for -DENABLE_PYTHON=ON, no cython
python3 -m pip install -r ./src/bindings/python/src/compatibility/openvino/requirements-dev.txt
# Run cmake with -DENABLE_PROFILING_ITT=ON -DSELECTIVE_BUILD=COLLECT in order to enable codestyle check for ITT collector
- name: CMake configure
@ -31,7 +36,7 @@ jobs:
- name: suggester / clang-format
if: startsWith(github.event_name, 'pull_request')
uses: reviewdog/action-suggester@9e1cd88b79ba3c0023c94e44accd72344f032093 # v1.13.0
uses: reviewdog/action-suggester@v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
level: warning
@ -42,7 +47,7 @@ jobs:
permissions:
pull-requests: write
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@v4
with:
submodules: 'true'
@ -60,7 +65,7 @@ jobs:
# always provide suggestions even for skipped scripts in ov_shellcheck tagret
- name: ShellCheck action
if: always()
uses: reviewdog/action-shellcheck@6e3a862f231c6895fbd335b70adef8f9243d5762 # v1.21.0
uses: reviewdog/action-shellcheck@v1
with:
level: style
reporter: github-pr-review
@ -73,7 +78,7 @@ jobs:
NamingConventionCheck:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@v4
with:
submodules: 'true'

View File

@ -5,8 +5,6 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions: read-all
jobs:
Coverage:
runs-on: ${{ matrix.config.os }}
@ -18,19 +16,19 @@ jobs:
steps:
- name: Setup python
uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
uses: actions/setup-python@v5
with:
python-version: '3.10.10'
architecture: 'x64'
- name: Setup ccache
uses: hendrikmuhs/ccache-action@c92f40bee50034e84c763e33b317c77adaa81c92 # v1.2.13
uses: hendrikmuhs/ccache-action@v1.2
with:
max-size: 50G
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
submodules: 'true'
@ -42,6 +40,9 @@ jobs:
python3 -m pip install --upgrade pip
python3 -m pip install -r ${{ github.workspace }}/src/bindings/python/wheel/requirements-dev.txt
python3 -m pip install -r ${{ github.workspace }}/src/bindings/python/requirements.txt
# For running Python API tests
python3 -m pip install -r ${{ github.workspace }}/src/bindings/python/src/compatibility/openvino/requirements-dev.txt
# For running Paddle frontend unit tests
python3 -m pip install -r ${{ github.workspace }}/src/frontends/paddle/tests/requirements.txt
# For running ONNX frontend unit tests
@ -56,8 +57,12 @@ jobs:
python3 -m pip install -r ${{ github.workspace }}/tools/mo/requirements_tf2.txt
python3 -m pip install -r ${{ github.workspace }}/tools/mo/requirements_dev.txt
- name: Get number of CPU cores
uses: SimenB/github-actions-cpu-cores@v2
id: cpu-cores
- name: Build OpenVINO with CMake
uses: ashutoshvarma/action-cmake-build@ade188313bc7eaa6f14349569a64d8bc716342ff # master
uses: ashutoshvarma/action-cmake-build@master
with:
build-dir: ${{ github.workspace }}/build
cc: ${{ matrix.config.cc }}
@ -79,6 +84,7 @@ jobs:
-DCMAKE_CXX_LINKER_LAUNCHER=ccache
-DENABLE_SYSTEM_SNAPPY=ON
build-type: Release
parallel: ${{ steps.cpu-cores.outputs.count }}
- name: Install wheel packages
run: cmake -DCOMPONENT=python_wheels -DCMAKE_INSTALL_PREFIX=${{ github.workspace }}/install_pkg -P '${{ github.workspace }}/build/cmake_install.cmake'
@ -99,7 +105,7 @@ jobs:
run: ${{ github.workspace }}/bin/intel64/Release/ov_proxy_plugin_tests
- name: Run OV Hetero Func tests
run: ${{ github.workspace }}/bin/intel64/Release/ov_hetero_func_tests --gtest_filter="*smoke*"
run: ${{ github.workspace }}/bin/intel64/Release/ov_hetero_func_tests
- name: Run IR frontend tests
run: ${{ github.workspace }}/bin/intel64/Release/ov_ir_frontend_tests
@ -114,7 +120,7 @@ jobs:
run: ${{ github.workspace }}/bin/intel64/Release/ov_tensorflow_frontend_tests --gtest_filter=-*IE_GPU*
- name: Build coverage with CMake
uses: ashutoshvarma/action-cmake-build@ade188313bc7eaa6f14349569a64d8bc716342ff # master
uses: ashutoshvarma/action-cmake-build@master
with:
build-dir: ${{ github.workspace }}/coverage
cc: ${{ matrix.config.cc }}
@ -126,6 +132,7 @@ jobs:
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
-DCMAKE_C_LINKER_LAUNCHER=ccache
-DCMAKE_CXX_LINKER_LAUNCHER=ccache
parallel: ${{ steps.cpu-cores.outputs.count }}
- name: Print info
@ -137,6 +144,6 @@ jobs:
lcov --capture --directory ${{ github.workspace }}/. --output-file coverage.info
genhtml coverage.info --output-directory coverage-report
- name: Collect coverage
uses: codecov/codecov-action@125fc84a9a348dbcf27191600683ec096ec9021c # v4.4.1
uses: codecov/codecov-action@v4
with:
verbose: true

View File

@ -14,8 +14,6 @@ concurrency:
group: ${{ github.event_name == 'push' && github.run_id || github.ref }}-linux-coverity
cancel-in-progress: true
permissions: read-all
env:
PIP_CACHE_PATH: /mount/caches/pip/linux
PYTHON_VERSION: '3.11'
@ -29,14 +27,22 @@ jobs:
runs-on: aks-linux-16-cores-32gb
container:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
CMAKE_BUILD_TYPE: 'Release'
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
OPENVINO_CONTRIB_REPO: /__w/openvino/openvino/openvino_contrib
BUILD_DIR: /__w/openvino/openvino/openvino_build
SCCACHE_AZURE_KEY_PREFIX: coverity_ubuntu20_x86_64
COVERITY_TOOL_DIR: /__w/openvino/openvino/coverity_tool
steps:
@ -46,19 +52,19 @@ jobs:
apt-get install --assume-yes --no-install-recommends git ca-certificates
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
path: ${{ env.OPENVINO_REPO }}
submodules: 'true'
ref: ${{ inputs.openvinoRef }}
- name: Clone OpenVINO Contrib
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
repository: 'openvinotoolkit/openvino_contrib'
path: ${{ env.OPENVINO_CONTRIB_REPO }}
submodules: 'true'
ref: 'master'
ref: 'releases/2023/3'
#
# Dependencies
@ -70,6 +76,11 @@ jobs:
# default-jdk - Java API
apt install --assume-yes --no-install-recommends default-jdk
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.7.5"
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: ./openvino/.github/actions/setup_python
with:
@ -88,11 +99,20 @@ jobs:
-G "${{ env.CMAKE_GENERATOR }}" \
-DENABLE_CPPLINT=OFF \
-DENABLE_STRICT_DEPENDENCIES=OFF \
-DENABLE_SYSTEM_TBB=ON \
-DENABLE_SYSTEM_OPENCL=ON \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-DENABLE_FASTER_BUILD=OFF \
-DCPACK_GENERATOR=TGZ \
-DBUILD_nvidia_plugin=OFF \
-DOPENVINO_EXTRA_MODULES=${OPENVINO_CONTRIB_REPO}/modules \
-DCMAKE_CXX_COMPILER_LAUNCHER=${{ env.CMAKE_CXX_COMPILER_LAUNCHER }} \
-DCMAKE_C_COMPILER_LAUNCHER=${{ env.CMAKE_C_COMPILER_LAUNCHER }} \
-S ${OPENVINO_REPO} \
-B ${BUILD_DIR}
- name: Clean sccache stats
run: ${SCCACHE_PATH} --zero-stats
- name: Install Coverity tool
run: |
rm -rf ${COVERITY_TOOL_DIR} && mkdir -p ${COVERITY_TOOL_DIR}
@ -102,7 +122,12 @@ jobs:
popd
- name: Cmake build - OpenVINO with Coverity
run: ${COVERITY_TOOL_DIR}/cov-analysis*/bin/cov-build --dir ${BUILD_DIR}/cov-int cmake --build ${BUILD_DIR} --parallel --config ${{ env.CMAKE_BUILD_TYPE }}
run: |
${COVERITY_TOOL_DIR}/cov-analysis*/bin/cov-build --dir ${BUILD_DIR}/cov-int \
cmake --build ${BUILD_DIR} --parallel --config ${{ env.CMAKE_BUILD_TYPE }}
- name: Show sccache stats
run: ${SCCACHE_PATH} --show-stats
- name: Pack Artefacts
run: |
@ -112,44 +137,12 @@ jobs:
- name: Submit artefacts
run: |
apt-get update && apt-get install -y curl jq
apt-get update && apt-get install -y curl
pushd ${BUILD_DIR}
curl -X POST -d token=${{ secrets.COVERITY_TOKEN }} \
-d email=${{ secrets.COVERITY_USER }} \
-d file_name="openvino.tgz" \
-d version="${{ github.sha }}" \
-d description="https://github.com/openvinotoolkit/openvino/runs/${{ github.run_number }}" \
https://scan.coverity.com/projects/21921/builds/init | tee response
upload_url=$(jq -r '.url' response)
build_id=$(jq -r '.build_id' response)
curl -X PUT \
--header 'Content-Type: application/json' \
--upload-file openvino.tgz \
$upload_url
curl -X PUT \
-d token=${{ secrets.COVERITY_TOKEN }} \
https://scan.coverity.com/projects/21921/builds/$build_id/enqueue
curl --form token=${{ secrets.COVERITY_TOKEN }} \
--form email=${{ secrets.COVERITY_USER }} \
--form file=@openvino.tgz \
--form version="${{ github.sha }}" \
--form description="https://github.com/openvinotoolkit/openvino/runs/${{ github.run_number }}" \
https://scan.coverity.com/builds?project=openvino
popd
- name: Show Coverity configure logs
continue-on-error: true
run: ${COVERITY_TOOL_DIR}/cov-analysis*/bin/cov-configure -c ${COVERITY_TOOL_DIR}/cov-analysis-linux64-2023.6.2/config/coverity_config.xml -lscc text
- name: Upload Coverity build log
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: coverity_logs
path: ${{ env.BUILD_DIR }}/cov-int/build-log.txt
if-no-files-found: 'error'
- name: Upload Coverity build archive
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: coverity_archive
path: ${{ env.BUILD_DIR }}/openvino.tgz
if-no-files-found: 'error'

View File

@ -1,18 +1,17 @@
name: 'Dependency Review'
on: [pull_request, merge_group]
on: [pull_request]
permissions: read-all
permissions:
contents: read
jobs:
dependency-review:
runs-on: ubuntu-latest
steps:
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
- name: Dependency Review
uses: actions/dependency-review-action@72eb03d02c7872a771aacd928f3123ac62ad6d3a # v4.3.3
uses: actions/dependency-review-action@v3
with:
config-file: './.github/dependency_review.yml'
base-ref: ${{ github.pull_request.base.sha || github.event.merge_group.base_ref }}
head-ref: ${{ github.pull_request.head.sha || github.ref }}

View File

@ -2,10 +2,9 @@ name: Fedora (RHEL), Python 3.9
on:
workflow_dispatch:
pull_request:
merge_group:
push:
branches:
# - master
- master
- 'releases/**'
concurrency:
@ -13,8 +12,6 @@ concurrency:
group: ${{ github.event_name == 'push' && github.run_id || github.ref }}-fedora33
cancel-in-progress: true
permissions: read-all
jobs:
Smart_CI:
runs-on: ubuntu-latest
@ -23,7 +20,7 @@ jobs:
skip_workflow: "${{ steps.smart_ci.outputs.skip_workflow }}"
steps:
- name: checkout action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
sparse-checkout: .github/actions/smart-ci
@ -34,7 +31,6 @@ jobs:
repository: ${{ github.repository }}
pr: ${{ github.event.number }}
commit_sha: ${{ github.sha }}
ref_name: ${{ github.ref_name }}
component_pattern: "category: (.*)"
repo_token: ${{ secrets.GITHUB_TOKEN }}
skip_when_only_listed_labels_set: 'docs'
@ -59,8 +55,6 @@ jobs:
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
SCCACHE_ERROR_LOG: /__w/openvino/sccache_log.txt
SCCACHE_LOG: warn
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
INSTALL_DIR: /__w/openvino/openvino/openvino_install
@ -73,7 +67,7 @@ jobs:
run: yum update -y && yum install -y git
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
path: ${{ env.OPENVINO_REPO }}
submodules: 'true'
@ -93,7 +87,7 @@ jobs:
run: bash ${OPENVINO_REPO}/install_build_dependencies.sh
- name: Install sccache
uses: mozilla-actions/sccache-action@2e7f9ec7921547d4b46598398ca573513895d0bd # v0.0.4
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.7.5"
@ -102,6 +96,7 @@ jobs:
python3 -m pip install -U pip
# For Python API: build and wheel packaging
python3 -m pip install -r ${OPENVINO_REPO}/src/bindings/python/wheel/requirements-dev.txt
python3 -m pip install -r ${OPENVINO_REPO}/src/bindings/python/src/compatibility/openvino/requirements-dev.txt
# For running ONNX frontend unit tests
python3 -m pip install --force-reinstall -r ${OPENVINO_REPO}/src/frontends/onnx/tests/requirements.txt
@ -127,6 +122,7 @@ jobs:
-DENABLE_NCC_STYLE=OFF \
-DENABLE_TESTS=ON \
-DENABLE_STRICT_DEPENDENCIES=OFF \
-DENABLE_SYSTEM_TBB=ON \
-DENABLE_SYSTEM_OPENCL=ON \
-DENABLE_PYTHON_PACKAGING=ON \
-DCPACK_GENERATOR=TGZ \
@ -161,27 +157,18 @@ jobs:
- name: Build RPM packages
run: |
cmake -UTBB* \
-DCPACK_GENERATOR=RPM \
-DENABLE_SYSTEM_TBB=ON \
cmake -DCPACK_GENERATOR=RPM \
-DENABLE_TESTS=OFF \
${BUILD_DIR}
cmake --build ${BUILD_DIR} --parallel --target package --verbose
#
# Upload build artifacts and logs
# Upload build artifacts
#
- name: Upload build logs
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: build_logs
path: ${{ env.SCCACHE_ERROR_LOG }}
if-no-files-found: 'ignore'
- name: Upload openvino package
if: ${{ always() }}
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_package
path: ${{ env.BUILD_DIR }}/openvino_package.tar.gz
@ -189,7 +176,7 @@ jobs:
- name: Upload openvino RPM packages
if: ${{ always() }}
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_rpm_packages
path: ${{ env.BUILD_DIR }}/*.rpm
@ -197,7 +184,7 @@ jobs:
- name: Upload openvino tests package
if: ${{ always() }}
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_tests
path: ${{ env.BUILD_DIR }}/openvino_tests.tar.gz
@ -217,7 +204,7 @@ jobs:
steps:
- name: Download OpenVINO RPM packages
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_rpm_packages
path: ${{ env.RPM_PACKAGES_DIR }}
@ -250,10 +237,10 @@ jobs:
python3 /usr/share/openvino/samples/python/hello_query_device/hello_query_device.py
python3 -c 'from openvino import Core; Core().get_property("CPU", "AVAILABLE_DEVICES")'
python3 -c 'from openvino import Core; Core().get_property("GPU", "AVAILABLE_DEVICES")'
python3 -c 'from openvino import Core; Core().get_property("AUTO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("MULTI", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("HETERO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("BATCH", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("AUTO", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("MULTI", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("HETERO", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("BATCH", "SUPPORTED_METRICS")'
python3 -c 'from openvino.frontend import FrontEndManager; assert len(FrontEndManager().get_available_front_ends()) == 6'
benchmark_app --help
ovc --help

View File

@ -5,13 +5,11 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions: read-all
jobs:
Check_Files_Size:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@v4
- name: git ls-tree
run: git ls-tree -r -t -l --full-name HEAD | sort -n -r -k 4

View File

@ -13,12 +13,10 @@ on:
required: false
default: null
permissions: read-all
jobs:
CPU_Functional_Tests:
name: CPU functional tests
timeout-minutes: 30
timeout-minutes: 25
runs-on: ${{ inputs.runner }}
container:
image: ${{ inputs.image }}
@ -32,14 +30,17 @@ jobs:
PARALLEL_TEST_SCRIPT: ${{ github.workspace }}/install/tests/functional_test_utils/layer_tests_summary/run_parallel.py
PARALLEL_TEST_CACHE: ${{ github.workspace }}/install/tests/test_cache.lst
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
@ -57,13 +58,17 @@ jobs:
pushd $INSTALL_DIR
tar -xzf openvino_package.tar.gz -C $INSTALL_DIR
popd
pushd $INSTALL_TEST_DIR
tar -xzf openvino_tests.tar.gz -C $INSTALL_DIR
popd
- name: Install OpenVINO dependencies (Linux)
if: runner.os == 'Linux'
run: $INSTALL_DIR/install_dependencies/install_openvino_dependencies.sh -c=core -c=dev -c=gpu -y
- name: Fetch setup_python action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
sparse-checkout: |
.github/actions/setup_python/action.yml
@ -81,7 +86,7 @@ jobs:
run: python3 -m pip install -r ${INSTALL_TEST_DIR}/functional_test_utils/layer_tests_summary/requirements.txt
- name: Restore tests execution time
uses: actions/cache/restore@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2
uses: actions/cache/restore@v4
with:
path: ${{ env.PARALLEL_TEST_CACHE }}
key: ${{ runner.os }}-${{ runner.arch }}-tests-functional-cpu-stamp-${{ github.sha }}
@ -94,22 +99,20 @@ jobs:
if [[ -f "${INSTALL_DIR}/setupvars.sh" ]]; then
source ${INSTALL_DIR}/setupvars.sh
fi
# Needed as ze_loader.so is under INSTALL_TEST_DIR
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${INSTALL_TEST_DIR}
python3 ${PARALLEL_TEST_SCRIPT} -e ${INSTALL_TEST_DIR}/ov_cpu_func_tests -c ${PARALLEL_TEST_CACHE} -w ${INSTALL_TEST_DIR} -s suite -rf 0 -- --gtest_print_time=1 --gtest_filter=*smoke*
timeout-minutes: 25
- name: Save tests execution time
uses: actions/cache/save@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2
uses: actions/cache/save@v4
if: github.ref_name == 'master'
with:
path: ${{ env.PARALLEL_TEST_CACHE }}
key: ${{ runner.os }}-${{ runner.arch }}-tests-functional-cpu-stamp-${{ github.sha }}
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
uses: actions/upload-artifact@v3
if: ${{ !cancelled() }}
with:
name: test-results-functional-cpu
path: |

View File

@ -17,8 +17,6 @@ on:
type: string
required: true
permissions: read-all
jobs:
CXX_Unit_Tests:
name: C++ unit tests
@ -34,14 +32,17 @@ jobs:
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
@ -61,6 +62,10 @@ jobs:
tar -xzf openvino_tests.tar.gz -C $INSTALL_DIR
popd
- name: Install OpenVINO dependencies (Linux)
if: runner.os == 'Linux'
run: $INSTALL_DIR/install_dependencies/install_openvino_dependencies.sh -c=core -c=dev -c=gpu -y
#
# Tests
#
@ -155,8 +160,23 @@ jobs:
${INSTALL_TEST_DIR}/ov_transformations_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-Transformations.xml
- name: Legacy Transformations func tests
if: fromJSON(inputs.affected-components).GNA.test &&
(runner.os != 'macOS' && runner.arch != 'ARM64')
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_legacy_transformations_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-LegacyTransformations.xml
- name: Inference Engine 1.0 unit tests
if: fromJSON(inputs.affected-components).GNA.test &&
(runner.os != 'macOS' && runner.arch != 'ARM64')
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/InferenceEngineUnitTests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-InferenceEngineUnitTests.xml
- name: Common test utils tests
if: ${{ runner.os != 'macOS' }} # Ticket: 134469
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_util_tests --gtest_print_time=1 \
@ -210,12 +230,12 @@ jobs:
--gtest_filter=*smoke* \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-TemplateFuncTests.xml
- name: OV utils unit tests
if: ${{ runner.os != 'macOS' }} # Ticket: 134469
- name: Inference Engine C API tests
if: fromJSON(inputs.affected-components).C_API.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_util_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ov_util_tests.xml
${INSTALL_TEST_DIR}/InferenceEngineCAPITests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-InferenceEngineCAPITests.xml
- name: OpenVINO C API tests
if: fromJSON(inputs.affected-components).C_API.test
@ -234,7 +254,7 @@ jobs:
if: fromJSON(inputs.affected-components).AUTO_BATCH.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_auto_batch_func_tests --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ov_auto_batch_func_tests.xml --gtest_filter="*smoke*"
${INSTALL_TEST_DIR}/ov_auto_batch_func_tests --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ov_auto_batch_func_tests.xml
- name: Proxy Plugin func tests
if: fromJSON(inputs.affected-components).PROXY.test
@ -252,10 +272,10 @@ jobs:
if: fromJSON(inputs.affected-components).HETERO.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_hetero_func_tests --gtest_print_time=1 --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-OVHeteroFuncTests.xml --gtest_filter="*smoke*"
${INSTALL_TEST_DIR}/ov_hetero_func_tests --gtest_print_time=1 --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-OVHeteroFuncTests.xml
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
if: ${{ !cancelled() }}
with:
name: test-results-cpp

View File

@ -13,8 +13,6 @@ on:
required: false
default: null
permissions: read-all
jobs:
Debian_Packages:
name: Debian Packages
@ -29,11 +27,10 @@ jobs:
DEBIAN_PACKAGES_DIR: ${{ github.workspace }}/packages
steps:
- name: Set apt retries
if: runner.os == 'Linux'
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO debian packages
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_debian_packages
path: ${{ env.DEBIAN_PACKAGES_DIR }}
@ -51,9 +48,10 @@ jobs:
apt-get install --no-install-recommends -y gnupg wget ca-certificates
wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
echo "deb https://apt.repos.intel.com/openvino/2024 ubuntu20 main" | tee /etc/apt/sources.list.d/intel-openvino-2024.list
echo "deb https://apt.repos.intel.com/openvino/2023 ubuntu20 main" | tee /etc/apt/sources.list.d/intel-openvino-2023.list
apt-get update -y
apt-get install -y openvino
apt-get install -y openvino-2023.2.0
rm /etc/apt/sources.list.d/intel-openvino-2023.list
fi
# install our local one and make sure the conflicts are resolved
@ -78,16 +76,15 @@ jobs:
python3 /usr/share/openvino/samples/python/hello_query_device/hello_query_device.py
python3 -c 'from openvino import Core; Core().get_property("CPU", "AVAILABLE_DEVICES")'
if [[ "${{ runner.arch }}" == "X64" ]]; then
python3 -c 'from openvino import Core; Core().get_property("GPU", "AVAILABLE_DEVICES")'
fi
python3 -c 'from openvino import Core; Core().get_property("AUTO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("MULTI", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("HETERO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("BATCH", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino.frontend import FrontEndManager; assert len(FrontEndManager().get_available_front_ends()) == 6'
benchmark_app --help
opt_in_out --help
ovc --help

View File

@ -1,134 +0,0 @@
name: GPU
on:
workflow_call:
inputs:
test_type:
description: 'Type of tests to execute'
type: string
required: true
device:
description: 'Device name (igpu or dgpu)'
type: string
required: true
runner:
description: 'Runner labels by which the runner will be chosen. Example: [ "self-hosted", "igpu" ]'
type: string
required: true
container:
description: 'JSON to be converted to the value of the "container" configuration for the job'
type: string
required: false
default: '{"image": null}'
jobs:
GPU:
timeout-minutes: 80
runs-on: ${{ fromJSON(inputs.runner) }}
container: ${{ fromJSON(inputs.container) }}
defaults:
run:
shell: bash
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
GTEST_PARALLEL_SCRIPT: ${{ github.workspace }}/gtest_parallel.py
steps:
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: 'openvino_package'
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: 'openvino_tests'
path: ${{ env.INSTALL_TEST_DIR }}
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
run: |
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
echo "GTEST_PARALLEL_SCRIPT=$GITHUB_WORKSPACE/gtest_parallel.py" >> "$GITHUB_ENV"
- name: Extract OpenVINO packages
run: |
pushd $INSTALL_DIR
tar -xzf openvino_package.tar.gz -C $INSTALL_DIR
popd
pushd $INSTALL_TEST_DIR
tar -xzf openvino_tests.tar.gz -C $INSTALL_DIR
popd
- name: Install dependencies (Linux)
run: |
$INSTALL_DIR/install_dependencies/install_openvino_dependencies.sh -c=core -c=dev -c=gpu -y
apt-get update && apt-get install -y wget software-properties-common ca-certificates gpg-agent tzdata clinfo
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
TZ: "Europe/London" # to prevent tzdata from waiting user input
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Get gtest-parallel script
run: wget https://raw.githubusercontent.com/google/gtest-parallel/master/gtest_parallel.py
- name: Install compute runtime drivers
run: |
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.15985.7/intel-igc-core_1.0.15985.7_amd64.deb
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.15985.7/intel-igc-opencl_1.0.15985.7_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/intel-level-zero-gpu-dbgsym_1.3.28454.6_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/intel-level-zero-gpu_1.3.28454.6_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/intel-opencl-icd-dbgsym_24.05.28454.6_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/intel-opencl-icd_24.05.28454.6_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/libigdgmm12_22.3.11_amd64.deb
dpkg -i *.deb
- name: Install media & display runtimes
if: ${{ inputs.device == 'dgpu' }}
run: |
apt-get update && apt-get install -y \
libegl-mesa0 libegl1-mesa libegl1-mesa-dev libgbm1 libgl1-mesa-dev libgl1-mesa-dri \
libglapi-mesa libgles2-mesa-dev libglx-mesa0 libigdgmm11 libxatracker2 mesa-va-drivers \
mesa-vdpau-drivers mesa-vulkan-drivers va-driver-all
- name: Verify devices
run: clinfo
#
# Tests
#
- name: OpenVINO GPU ${{ inputs.test_type }} Tests
id: run_tests
run: |
source ${INSTALL_DIR}/setupvars.sh
TEST_RESULTS_DIR="${{ inputs.device }}_${{ inputs.test_type }}_tests"
echo "test_results_dir=$TEST_RESULTS_DIR" >> $GITHUB_OUTPUT
rm -rf ${INSTALL_TEST_DIR}/${TEST_RESULTS_DIR} && mkdir -p ${INSTALL_TEST_DIR}/${TEST_RESULTS_DIR}
test_filter=''
if [[ "${{ inputs.test_type }}" == "unit" ]]; then
# Ticket: 138018
test_filter='-*scatter_nd_update_gpu.dynamic_padded_output*:*border_gpu.basic_zero_input*:*bicubic_zeros_no_align_data1x1*:*bicubic_border_align_batches*:*bilinear_zeros_no_align_data1x1*:*non_zero_gpu.empty_input*:*mark_shape_of_subgraphs.concat_with_empty_tensor_inputs*:*concat_cpu_impl.dynamic_4d_f*:*border_gpu.basic_zero_input_dynamic*:*network_test.model_with_empty_input_is_not_dynamic*:*bicubic_zeros_align_data1x1*'
else
test_filter='*smoke*'
fi
python3 ${GTEST_PARALLEL_SCRIPT} ${INSTALL_TEST_DIR}/ov_gpu_${{ inputs.test_type }}_tests --dump_json_test_results=${INSTALL_TEST_DIR}/${TEST_RESULTS_DIR}/ov_gpu_${{ inputs.test_type }}_tests.json -- --report_unique_name --gtest_filter=$test_filter
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: test-results-${{ inputs.test_type }}-${{ inputs.device }}
path: ${{ env.INSTALL_TEST_DIR }}/${{ steps.run_tests.outputs.test_results_dir }}
if-no-files-found: 'error'

View File

@ -13,8 +13,6 @@ on:
required: false
default: '{"image": null}'
permissions: read-all
jobs:
ONNX_Models_tests:
name: ONNX Models tests
@ -34,17 +32,19 @@ jobs:
# choose one of the replicas dynamically instead
# depending on GITHUB_RUN_NUMBER variable
NUMBER_OF_REPLICAS: 2
ONNX_MODEL_ZOO_SHA: "5faef4c33eba0395177850e1e31c4a6a9e634c82"
if: ${{ github.event_name != 'merge_group' }}
ONNX_MODEL_ZOO_SHA: "d58213534f2a4d1c4b19ba62b3bb5f544353256e"
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
@ -55,7 +55,7 @@ jobs:
echo "OPENVINO_REPO=$GITHUB_WORKSPACE/openvino" >> "$GITHUB_ENV"
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
echo "MODELS_SHARE_PATH=/mount/testdata$((GITHUB_RUN_NUMBER % NUMBER_OF_REPLICAS))" >> "$GITHUB_ENV"
echo "MODELS_SHARE_PATH=/mount/onnxtestdata$((GITHUB_RUN_NUMBER % NUMBER_OF_REPLICAS))" >> "$GITHUB_ENV"
echo $MODELS_SHARE_PATH
- name: Extract OpenVINO packages
@ -68,14 +68,27 @@ jobs:
tar -xzf openvino_tests.tar.gz -C ${INSTALL_DIR}
popd
- name: Fetch model_zoo_preprocess script
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- name: Fetch setup_python action and model_zoo_preprocess script
uses: actions/checkout@v4
with:
sparse-checkout: |
.github/actions/setup_python/action.yml
src/frontends/onnx/tests/tests_python/model_zoo_preprocess.sh
sparse-checkout-cone-mode: false
path: 'openvino'
- name: Install dependencies
run: |
# install git (required to build pip deps from the sources)
apt-get update && apt-get install --assume-yes --no-install-recommends git ca-certificates git-lfs
- name: Setup Python 3.11
uses: ./openvino/.github/actions/setup_python
with:
version: '3.11'
should-setup-pip-paths: 'false'
self-hosted-runner: ${{ contains(inputs.runner, 'aks') }}
- name: Update Models
run: bash ${OPENVINO_REPO}/src/frontends/onnx/tests/tests_python/model_zoo_preprocess.sh -d ${MODELS_SHARE_PATH} -o -s "${{ env.ONNX_MODEL_ZOO_SHA }}"

View File

@ -17,8 +17,6 @@ on:
type: string
required: true
permissions: read-all
jobs:
ONNX_Runtime:
name: ONNX Runtime Integration
@ -42,8 +40,11 @@ jobs:
ONNX_RUNTIME_UTILS: ${{ github.workspace }}/install/onnxruntime
ONNX_RUNTIME_BUILD_DIR: ${{ github.workspace }}/onnxruntime/build
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
@ -57,35 +58,63 @@ jobs:
echo "ONNX_RUNTIME_UTILS=$GITHUB_WORKSPACE/install/onnxruntime" >> "$GITHUB_ENV"
echo "ONNX_RUNTIME_BUILD_DIR=$GITHUB_WORKSPACE/onnxruntime/build" >> "$GITHUB_ENV"
- name: Fetch install_build_dependencies.sh and setup_python action
uses: actions/checkout@v4
with:
sparse-checkout: |
install_build_dependencies.sh
.github/actions/setup_python/action.yml
sparse-checkout-cone-mode: false
path: 'openvino'
- name: Install git
run: |
apt-get update
apt-get install --assume-yes --no-install-recommends git ca-certificates
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: ./openvino/.github/actions/setup_python
with:
version: '3.11'
should-setup-pip-paths: 'false'
- name: Extract OpenVINO package
run: |
pushd ${INSTALL_DIR}
tar -xzf openvino_package.tar.gz -C ${INSTALL_DIR}
popd
- name: Install OpenVINO dependencies
run: ${INSTALL_DIR}/install_dependencies/install_openvino_dependencies.sh -c=core -c=dev -y
- name: Clone ONNX Runtime
run: |
hash=`tr -s '\n ' < ${ONNX_RUNTIME_UTILS}/version`
git clone --recursive https://github.com/microsoft/onnxruntime.git ${ONNX_RUNTIME_REPO}
cd ${ONNX_RUNTIME_REPO}
git checkout $hash
branch=`tr -s '\n ' < ${ONNX_RUNTIME_UTILS}/version`
git clone --branch $branch --single-branch --recursive https://github.com/microsoft/onnxruntime.git ${ONNX_RUNTIME_REPO}
#
# Tests
#
- name: Install Build Dependencies
run: bash ${OPENVINO_REPO}/install_build_dependencies.sh
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.4
with:
version: "v0.7.5"
- name: Build Lin ONNX Runtime
run: |
source ${INSTALL_DIR}/setupvars.sh
${ONNX_RUNTIME_REPO}/build.sh \
--config RelWithDebInfo \
--use_openvino CPU \
--use_openvino CPU_FP32 \
--build_shared_lib \
--parallel \
--skip_tests \
--compile_no_warning_as_error \
--allow_running_as_root \
--build_dir ${ONNX_RUNTIME_BUILD_DIR}
env:
CXXFLAGS: "-Wno-error=deprecated-declarations"
@ -96,11 +125,6 @@ jobs:
- name: Run onnxruntime_test_all
if: ${{ runner.arch != 'ARM64' }} # Ticket: 126277
run: |
# see https://github.com/microsoft/onnxruntime/issues/13197#issuecomment-1264542497
apt-get update && apt-get install --assume-yes --no-install-recommends language-pack-en
locale-gen en_US.UTF-8
update-locale LANG=en_US.UTF-8
source ${INSTALL_DIR}/setupvars.sh
skip_tests=$(tr -s '\n ' ':' < ${ONNX_RUNTIME_UTILS}/skip_tests)
@ -119,6 +143,12 @@ jobs:
./onnxruntime_global_thread_pools_test
working-directory: ${{ env.ONNX_RUNTIME_BUILD_DIR }}/RelWithDebInfo/RelWithDebInfo
- name: Run onnxruntime_api_tests_without_env
run: |
source ${INSTALL_DIR}/setupvars.sh
./onnxruntime_api_tests_without_env
working-directory: ${{ env.ONNX_RUNTIME_BUILD_DIR }}/RelWithDebInfo/RelWithDebInfo
- name: Run pytorch-converted tests
run: |
source ${INSTALL_DIR}/setupvars.sh

View File

@ -13,8 +13,6 @@ on:
required: false
default: '{"image": null}'
permissions: read-all
jobs:
JS_API:
name: OpenVINO JS API
@ -28,10 +26,10 @@ jobs:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
OPENVINO_JS_DIR: ${{ github.workspace }}/openvino/src/bindings/js
OPENVINO_JS_LIBS_DIR: ${{ github.workspace }}/openvino/src/bindings/js/node/bin
NODE_VERSION: 20
NODE_VERSION: 18
steps:
- name: Fetch OpenVINO JS sources
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
sparse-checkout: |
src/bindings/js
@ -44,13 +42,13 @@ jobs:
echo "OPENVINO_JS_LIBS_DIR=$GITHUB_WORKSPACE/openvino/src/bindings/js/node/bin" >> "$GITHUB_ENV"
- name: Download OpenVINO JS package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_js_package
path: ${{ env.OPENVINO_JS_LIBS_DIR }}
- name: Setup Node ${{ env.NODE_VERSION }}
uses: actions/setup-node@60edb5dd545a775178f52524783378180af0d1f8 # v4.0.2
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}

View File

@ -17,8 +17,6 @@ on:
type: string
required: true
permissions: read-all
env:
PIP_CACHE_PATH: /mount/caches/pip/linux
PYTHON_VERSION: '3.11'
@ -39,15 +37,17 @@ jobs:
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
LAYER_TESTS_INSTALL_DIR: ${{ github.workspace }}/install/tests/layer_tests
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
@ -69,16 +69,20 @@ jobs:
tar -xzf openvino_tests.tar.gz -C $INSTALL_DIR
popd
- name: Install OpenVINO dependencies (Linux)
if: runner.os == 'Linux'
run: $INSTALL_DIR/install_dependencies/install_openvino_dependencies.sh -c=core -c=dev -y -c=gpu
- name: Fetch setup_python action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
sparse-checkout: |
.github/actions/setup_python/action.yml
sparse-checkout-cone-mode: false
path: 'action_root'
path: 'openvino'
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: ./action_root/.github/actions/setup_python
uses: ./openvino/.github/actions/setup_python
with:
version: ${{ env.PYTHON_VERSION }}
pip-cache-path: ${{ runner.os == 'Linux' && env.PIP_CACHE_PATH || '' }}
@ -93,9 +97,9 @@ jobs:
run: |
# Install the core OV wheel
python3 -m pip install ${INSTALL_DIR}/tools/openvino-*.whl
extras_to_install="caffe,kaldi,onnx,tensorflow2,pytorch"
if [[ "${{ runner.arch }}" != "ARM64" ]]; then
extras_to_install="mxnet,$extras_to_install"
fi
@ -120,7 +124,15 @@ jobs:
# Tests
#
- name: Python API Tests
- name: Python API 1.0 Tests
# if: fromJSON(inputs.affected-components).Python_API.test # Ticket: 127101
run: |
python3 -m pytest -s ${INSTALL_TEST_DIR}/pyngraph \
--junitxml=${INSTALL_TEST_DIR}/TEST-Pyngraph.xml \
--ignore=${INSTALL_TEST_DIR}/pyngraph/tests_compatibility/test_onnx/test_zoo_models.py \
--ignore=${INSTALL_TEST_DIR}/pyngraph/tests_compatibility/test_onnx/test_backend.py
- name: Python API 2.0 Tests
# if: ${{ fromJSON(inputs.affected-components).Python_API.test && runner.arch != 'ARM64' }} # Ticket: 126380, 127101
run: |
# for 'template' extension
@ -132,20 +144,22 @@ jobs:
- name: Model Optimizer unit tests
if: fromJSON(inputs.affected-components).MO.test
run: |
if [[ "${{ runner.os }}" == "Linux" ]] && [[ "${{ runner.arch }}" != "ARM64" ]]; then
skip_filter=''
if [[ "${{ runner.os }}" != "Linux" ]] && [[ "${{ runner.arch }} != "ARM64" ]] || [[ "${{ runner.os }} != "macOS" ]]; then
# required for MxNet
apt-get install -y libgomp1 libquadmath0
else
# Skips under Ticket: 122666
skip_filter='--ignore-glob=**/mo/unit_tests/mo/front/mxnet/**'
fi
# Skips under tickets: 133405, 122666
python3 -m pytest -s ${INSTALL_TEST_DIR}/mo/unit_tests \
--junitxml=${INSTALL_TEST_DIR}/TEST-ModelOptimizer.xml \
--ignore-glob="**/mo/unit_tests/mo/front/mxnet/**"
"$skip_filter"
- name: Python ONNX operators tests
if: (fromJSON(inputs.affected-components).Python_API.test ||
fromJSON(inputs.affected-components).ONNX_FE.test) &&
runner.arch != 'ARM64' # Ticket: 123325
if: fromJSON(inputs.affected-components).Python_API.test ||
fromJSON(inputs.affected-components).ONNX_FE.test && runner.os != 'macOS' # Ticket: 123325
run: |
# Skip test_onnx/test_zoo_models and test_onnx/test_backend due to long execution time - ONNX Model Zoo tests are run separately
python3 -m pytest -sv ${INSTALL_TEST_DIR}/onnx -k 'not cuda' \
@ -167,13 +181,13 @@ jobs:
# Import 'test_utils' installed in '<package_test>/tests/python/openvino'
export LD_LIBRARY_PATH=${PIP_INSTALL_PATH}/openvino/libs:$LD_LIBRARY_PATH
export PYTHONPATH=${INSTALL_TEST_DIR}/python
if [[ "${{ runner.os }}" == "Linux" ]] && [[ "${{ runner.arch }}" == "ARM64" ]]; then
# Find gomp lib
GOMP_LIB=$(find "${PIP_INSTALL_PATH}/torch/lib/../../torch.libs/" -name '*libgomp-*so*')
export LD_PRELOAD=${GOMP_LIB}
fi
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/mo_python_api_tests --junitxml=${INSTALL_TEST_DIR}/TEST-test_mo_convert.xml
env:
TEST_DEVICE: CPU
@ -185,13 +199,13 @@ jobs:
# Import 'test_utils' installed in '<package_test>/tests/python/openvino'
export PYTHONPATH=${INSTALL_TEST_DIR}/python
export LD_LIBRARY_PATH=${PIP_INSTALL_PATH}/openvino/libs:$LD_LIBRARY_PATH
if [[ "${{ runner.os }}" == "Linux" ]] && [[ "${{ runner.arch }}" == "ARM64" ]]; then
# Find gomp lib
GOMP_LIB=$(find "${PIP_INSTALL_PATH}/torch/lib/../../torch.libs/" -name '*libgomp-*so*')
export LD_PRELOAD=${GOMP_LIB}
fi
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/ovc_python_api_tests --junitxml=${INSTALL_TEST_DIR}/TEST-test_ovc_convert.xml
env:
TEST_DEVICE: CPU
@ -206,23 +220,14 @@ jobs:
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/py_frontend_tests --junitxml=${INSTALL_TEST_DIR}/TEST-test_py_fontend.xml
- name: PyTorch Layer Tests
if: ${{ fromJSON(inputs.affected-components).PyTorch_FE.test && runner.arch != 'ARM64' }} # Ticket: 126287, 142196
if: ${{ fromJSON(inputs.affected-components).PyTorch_FE.test && runner.arch != 'ARM64' }} # Ticket: 126287
run: python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/pytorch_tests -n logical -m precommit --junitxml=${INSTALL_TEST_DIR}/TEST-pytorch.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP32
- name: PyTorch torch.export Layer Tests
if: ${{ fromJSON(inputs.affected-components).PyTorch_FE.test && runner.arch != 'ARM64' }} # Ticket: 126287
run: |
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/pytorch_tests -n logical -m precommit_torch_export --junitxml=${INSTALL_TEST_DIR}/TEST-pytorch.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP32
PYTORCH_TRACING_MODE: EXPORT
- name: PyTorch torch.compile TORCHFX Layer Tests
if: ${{ fromJSON(inputs.affected-components).PyTorch_FE.test && runner.os != 'macOS' && runner.arch != 'ARM64' }} # Ticket: 126287
if: ${{ fromJSON(inputs.affected-components).PyTorch_FE.test && runner.os != 'macOS' }}
run: |
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/pytorch_tests -m precommit_fx_backend --junitxml=${INSTALL_TEST_DIR}/TEST-pytorch.xml
env:
@ -230,6 +235,15 @@ jobs:
TEST_PRECISION: FP32
PYTORCH_TRACING_MODE: TORCHFX
- name: PyTorch torch.compile TORCHSCRIPT Layer Tests
if: ${{ fromJSON(inputs.affected-components).PyTorch_FE.test && runner.os != 'macOS' }}
run: |
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/pytorch_tests -m precommit_ts_backend --junitxml=${INSTALL_TEST_DIR}/TEST-pytorch.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP32
PYTORCH_TRACING_MODE: TORCHSCRIPT
- name: ONNX Layer Tests
if: fromJSON(inputs.affected-components).ONNX_FE.test
run: |
@ -240,11 +254,41 @@ jobs:
TEST_DEVICE: CPU
TEST_PRECISION: FP16
- name: TensorFlow 1 Layer Tests - TF FE
if: fromJSON(inputs.affected-components).TF_FE.test
run: |
# requires 'unit_tests' from 'mo'
export PYTHONPATH=${INSTALL_TEST_DIR}/mo
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/tensorflow_tests/ --use_new_frontend -m precommit_tf_fe --junitxml=${INSTALL_TEST_DIR}/TEST-tf_fe.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP16
- name: TensorFlow 2 Layer Tests - TF FE
if: fromJSON(inputs.affected-components).TF_FE.test && runner.os != 'macOS' # Ticket: 123322
run: |
# requires 'unit_tests' from 'mo'
export PYTHONPATH=${INSTALL_TEST_DIR}/mo
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/tensorflow2_keras_tests/ --use_new_frontend -m precommit_tf_fe --junitxml=${INSTALL_TEST_DIR}/TEST-tf2_fe.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP16
- name: JAX Layer Tests - TF FE
if: ${{ fromJSON(inputs.affected-components).TF_FE.test && runner.arch != 'ARM64' && runner.os != 'macOS' }}
if: ${{ fromJSON(inputs.affected-components).TF_FE.test && runner.arch != 'ARM64' }}
run: python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/jax_tests/ -m precommit --junitxml=${INSTALL_TEST_DIR}/TEST-jax.xml
env:
TEST_DEVICE: CPU
- name: TensorFlow 1 Layer Tests - Legacy FE
if: fromJSON(inputs.affected-components).TF_FE.test
run: python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/tensorflow_tests/test_tf_Roll.py --ir_version=10 --junitxml=${INSTALL_TEST_DIR}/TEST-tf_Roll.xml
- name: TensorFlow 2 Layer Tests - Legacy FE
if: fromJSON(inputs.affected-components).TF_FE.test
run: python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/tensorflow2_keras_tests/test_tf2_keras_activation.py --ir_version=11 -k "sigmoid" --junitxml=${INSTALL_TEST_DIR}/TEST-tf2_Activation.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP16
- name: TensorFlow Lite Layer Tests - TFL FE
@ -254,25 +298,11 @@ jobs:
TEST_DEVICE: CPU
TEST_PRECISION: FP16
- name: TensorFlow 1 Layer Tests - Legacy FE
if: fromJSON(inputs.affected-components).TF_FE.test
run: python3 -m pytest ${{ env.LAYER_TESTS_INSTALL_DIR }}/tensorflow_tests/test_tf_Roll.py --use_legacy_frontend --ir_version=10 --junitxml=${{ env.INSTALL_TEST_DIR }}/TEST-tf_Roll.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP16
- name: TensorFlow 2 Layer Tests - Legacy FE
if: fromJSON(inputs.affected-components).TF_FE.test
run: python3 -m pytest ${{ env.LAYER_TESTS_INSTALL_DIR }}/tensorflow2_keras_tests/test_tf2_keras_activation.py --use_legacy_frontend --ir_version=11 -k "sigmoid" --junitxml=${{ env.INSTALL_TEST_DIR }}/TEST-tf2_Activation.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP16
- name: Clone API snippets
if: runner.os != 'macOS'
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
sparse-checkout: docs/snippets
sparse-checkout: openvino/docs/snippets
path: ${{ env.OPENVINO_REPO }}
submodules: 'false'
@ -286,7 +316,7 @@ jobs:
python3 ${OPENVINO_REPO}/docs/snippets/main.py
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
if: ${{ !cancelled() }}
with:
name: test-results-python

View File

@ -17,8 +17,6 @@ on:
type: string
required: true
permissions: read-all
jobs:
PyTorch_Models_Tests:
name: PyTorch Models tests
@ -40,7 +38,6 @@ jobs:
run: if [ "$(id -u)" -eq 0 ]; then apt update && apt --assume-yes install sudo; fi
- name: Set apt retries
if: runner.os == 'Linux'
run: |
if [ "$(id -u)" -eq 0 ]; then
echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
@ -49,19 +46,13 @@ jobs:
fi
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tokenizers extension
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_tokenizers_wheel
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
@ -85,7 +76,7 @@ jobs:
popd
- name: Fetch setup_python action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
sparse-checkout: |
.github/actions/setup_python/action.yml
@ -107,46 +98,21 @@ jobs:
self-hosted-runner: ${{ contains(inputs.runner, 'aks') }}
- name: Install OpenVINO Python wheels
run: |
# To enable pytest parallel features
python3 -m pip install pytest-xdist[psutil]
python3 -m pip install ${INSTALL_DIR}/tools/openvino-*
python3 -m pip install ${INSTALL_DIR}/openvino_tokenizers-*
run: python3 -m pip install ${INSTALL_DIR}/tools/openvino-*
- name: Install PyTorch tests requirements
run: |
python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch/requirements.txt
python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch/requirements_secondary.txt
python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/torch_tests/requirements.txt
python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/torch_tests/requirements_secondary.txt
echo "Available storage:"
df -h
env:
CPLUS_INCLUDE_PATH: ${{ env.Python_ROOT_DIR }}/include/python${{ env.PYTHON_VERSION }}
- name: PyTorch Models Tests Timm and Torchvision
- name: PyTorch Models Tests
run: |
export PYTHONPATH=${MODEL_HUB_TESTS_INSTALL_DIR}:$PYTHONPATH
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch/ -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-torch_model_timm_tv_tests.html --self-contained-html -v -n 4 -k "TestTimmConvertModel or TestTorchHubConvertModel"
env:
TYPE: ${{ inputs.event == 'schedule' && 'nightly' || 'precommit'}}
TEST_DEVICE: CPU
OP_REPORT_FILE: ${{ env.INSTALL_TEST_DIR }}/TEST-torch_unsupported_ops.log
- name: PyTorch Models Tests Not Timm or Torchvision
if: always()
run: |
export PYTHONPATH=${MODEL_HUB_TESTS_INSTALL_DIR}:$PYTHONPATH
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-torch_model_tests.html --self-contained-html -v -k "not (TestTimmConvertModel or TestTorchHubConvertModel)"
env:
TYPE: ${{ inputs.event == 'schedule' && 'nightly' || 'precommit'}}
TEST_DEVICE: CPU
USE_SYSTEM_CACHE: False
OP_REPORT_FILE: ${{ env.INSTALL_TEST_DIR }}/TEST-torch_unsupported_ops.log
- name: PagedAttention Test
if: always()
run: |
export PYTHONPATH=${MODEL_HUB_TESTS_INSTALL_DIR}:$PYTHONPATH
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch/test_pa_transformation.py -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-torch_pagedattention_tests.html --self-contained-html -v --tb=short
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/torch_tests -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-torch_model_tests.html --self-contained-html -v
env:
TYPE: ${{ inputs.event == 'schedule' && 'nightly' || 'precommit'}}
TEST_DEVICE: CPU
@ -156,15 +122,15 @@ jobs:
- name: Reformat unsupported ops file
if: '!cancelled()'
run: |
python3 ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch/scripts/process_op_report.py ${INSTALL_TEST_DIR}/TEST-torch_unsupported_ops.log
python3 ${MODEL_HUB_TESTS_INSTALL_DIR}/torch_tests/scripts/process_op_report.py ${INSTALL_TEST_DIR}/TEST-torch_unsupported_ops.log
- name: Available storage after tests
run: |
echo "Available storage:"
df -h
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
if: ${{ !cancelled() }}
with:
name: test-results-torch-models

View File

@ -17,8 +17,6 @@ on:
type: string
required: true
permissions: read-all
jobs:
Samples:
runs-on: ${{ inputs.runner }}
@ -33,14 +31,17 @@ jobs:
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
BUILD_DIR: ${{ github.workspace }}/build
steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
@ -61,12 +62,16 @@ jobs:
tar -xzf openvino_tests.tar.gz -C $INSTALL_DIR
popd
- name: Install OpenVINO dependencies (Linux)
if: runner.os == 'Linux'
run: $INSTALL_DIR/install_dependencies/install_openvino_dependencies.sh -c=core -c=dev -y
- name: Install OpenVINO dependencies (mac)
if: runner.os == 'macOS'
run: brew install coreutils
- name: Fetch setup_python action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
sparse-checkout: |
.github/actions/setup_python/action.yml
@ -78,6 +83,7 @@ jobs:
with:
version: '3.11'
should-setup-pip-paths: 'false'
self-hosted-runner: ${{ runner.os == 'Linux' }}
- name: Build cpp samples - GCC
run: $INSTALL_DIR/samples/cpp/build_samples.sh -i $INSTALL_DIR -b $BUILD_DIR/cpp_samples
@ -87,7 +93,7 @@ jobs:
- name: Build cpp samples - Clang
if: runner.os == 'Linux'
run: |
apt-get update && apt-get install -y clang
apt-get install -y clang
$INSTALL_DIR/samples/cpp/build_samples.sh -i $INSTALL_DIR -b $BUILD_DIR/cpp_samples_clang
env:
CMAKE_COMPILE_WARNING_AS_ERROR: 'ON'
@ -111,10 +117,19 @@ jobs:
export IE_APP_PYTHON_PATH=$INSTALL_DIR/samples/python
export SHARE=$INSTALL_TEST_DIR/smoke_tests/samples_smoke_tests_data
# Install Python benchmark_app by installing openvino-*.whl
python3 -m pip install --ignore-installed PyYAML -r $INSTALL_TEST_DIR/smoke_tests/requirements.txt $INSTALL_DIR/tools/openvino-*.whl
python3 -m pip install --ignore-installed PyYAML -r $INSTALL_TEST_DIR/smoke_tests/requirements.txt
export LD_LIBRARY_PATH=${IE_APP_PATH}:$LD_LIBRARY_PATH
source ${INSTALL_DIR}/setupvars.sh
PYTHONCOERCECLOCALE=warn python3 -bb -W error -X dev -m pytest $INSTALL_TEST_DIR/smoke_tests
python3 -m pytest -sv $INSTALL_TEST_DIR/smoke_tests \
--env_conf $INSTALL_TEST_DIR/smoke_tests/env_config.yml \
--junitxml=$INSTALL_TEST_DIR/TEST-SamplesSmokeTests.xml
- name: Upload Test Results
uses: actions/upload-artifact@v3
if: ${{ !cancelled() }}
with:
name: test-results-samples
path: ${{ env.INSTALL_TEST_DIR }}/TEST*.xml
if-no-files-found: 'warn'

View File

@ -0,0 +1,120 @@
name: TensorFlow Hub Models tests
on:
workflow_call:
inputs:
runner:
description: 'Machine on which the tests would run'
type: string
required: true
container:
description: 'JSON to be converted to the value of the "container" configuration for the job'
type: string
required: false
default: '{"image": null}'
event:
description: 'Event that triggered the workflow. E.g., "schedule" for nightly runs'
type: string
required: true
jobs:
TensorFlow_Hub_Models_Tests:
name: TensorFlow Hub Models tests
timeout-minutes: ${{ inputs.event == 'schedule' && 400 || 25 }}
runs-on: ${{ inputs.runner }}
container: ${{ fromJSON(inputs.container) }}
defaults:
run:
shell: bash
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
OPENVINO_REPO: ${{ github.workspace }}/openvino
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
MODEL_HUB_TESTS_INSTALL_DIR: ${{ github.workspace }}/install/tests/model_hub_tests
steps:
- name: Check sudo
if: ${{ runner.os == 'Linux' }}
run: if [ "$(id -u)" -eq 0 ]; then apt update && apt --assume-yes install sudo; fi
- name: Set apt retries
run: |
if [ "$(id -u)" -eq 0 ]; then
echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
else
sudo sh -c "echo 'Acquire::Retries \"10\";' >> /etc/apt/apt.conf.d/80-retries"
fi
- name: Download OpenVINO package
uses: actions/download-artifact@v3
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@v3
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
run: |
echo "OPENVINO_REPO=$GITHUB_WORKSPACE/openvino" >> "$GITHUB_ENV"
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
echo "MODEL_HUB_TESTS_INSTALL_DIR=$GITHUB_WORKSPACE/install/tests/model_hub_tests" >> "$GITHUB_ENV"
- name: Extract OpenVINO packages
run: |
pushd ${INSTALL_DIR}
tar -xzf openvino_package.tar.gz -C ${INSTALL_DIR}
popd
pushd ${INSTALL_TEST_DIR}
tar -xzf openvino_tests.tar.gz -C ${INSTALL_DIR}
popd
- name: Fetch setup_python action
uses: actions/checkout@v4
with:
sparse-checkout: |
.github/actions/setup_python/action.yml
sparse-checkout-cone-mode: false
path: 'openvino'
- name: Install dependencies
if: ${{ runner.os == 'Linux' }}
run: |
# install git (required to build pip deps from the sources)
sudo apt-get install --assume-yes --no-install-recommends g++ git ca-certificates
- name: Setup Python 3.11
uses: ./openvino/.github/actions/setup_python
with:
version: '3.11'
should-setup-pip-paths: 'false'
self-hosted-runner: ${{ contains(inputs.runner, 'aks') }}
- name: Install OpenVINO Python wheels
run: python3 -m pip install ${INSTALL_DIR}/tools/openvino-*
- name: Install TF Hub tests requirements
run: python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/tf_hub_tests/requirements.txt
- name: TensorFlow Hub Tests - TF FE
run: |
export PYTHONPATH=${MODEL_HUB_TESTS_INSTALL_DIR}:$PYTHONPATH
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/tf_hub_tests/ -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-tf_hub_tf_fe.html --self-contained-html -v
env:
TYPE: ${{ inputs.event == 'schedule' && 'nightly' || 'precommit'}}
TEST_DEVICE: CPU
- name: Upload Test Results
uses: actions/upload-artifact@v3
if: ${{ !cancelled() }}
with:
name: test-results-tensorflow-hub-models
path: |
${{ env.INSTALL_TEST_DIR }}/TEST*.html
if-no-files-found: 'error'

View File

@ -1,168 +0,0 @@
name: TensorFlow Layer Tests
on:
workflow_call:
inputs:
runner:
description: 'Machine on which the tests would run'
type: string
required: true
shell:
description: "shell to override the default shell settings in the runner's operating system."
type: string
required: true
container:
description: 'JSON to be converted to the value of the "container" configuration for the job'
type: string
required: false
default: '{"image": null}'
affected-components:
description: 'Components that are affected by changes in the commit defined by the Smart CI Action'
type: string
required: true
permissions: read-all
env:
PIP_CACHE_PATH: /mount/caches/pip/linux
PYTHON_VERSION: '3.11'
jobs:
TensorFlow_Layer_Tests:
name: TensorFlow Layer Tests
timeout-minutes: 30
runs-on: ${{ inputs.runner }}
container: ${{ fromJSON(inputs.container) }}
defaults:
run:
shell: ${{ inputs.shell }}
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
OPENVINO_REPO: ${{ github.workspace }}/openvino
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
LAYER_TESTS_INSTALL_DIR: ${{ github.workspace }}/install/tests/layer_tests
steps:
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
- name: Download OpenVINO tokenizers extension
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_tokenizers_wheel
path: ${{ env.INSTALL_DIR }}
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
if: runner.os != 'Windows'
run: |
echo "OPENVINO_REPO=$GITHUB_WORKSPACE/openvino" >> "$GITHUB_ENV"
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
echo "LAYER_TESTS_INSTALL_DIR=$GITHUB_WORKSPACE/install/tests/layer_tests" >> "$GITHUB_ENV"
- name: Extract OpenVINO packages (Linux, macOS)
if: runner.os != 'Windows'
run: |
pushd $INSTALL_DIR
tar -xzf openvino_package.tar.gz -C $INSTALL_DIR
popd
pushd $INSTALL_TEST_DIR
tar -xzf openvino_tests.tar.gz -C $INSTALL_DIR
popd
- name: Extract OpenVINO packages (Windows)
if: runner.os == 'Windows'
run: |
pushd ${{ env.INSTALL_DIR }}
Expand-Archive openvino_package.zip -DestinationPath ${{ env.INSTALL_DIR }}
popd
pushd ${{ env.INSTALL_TEST_DIR }}
Expand-Archive openvino_tests.zip -DestinationPath ${{ env.INSTALL_DIR }}
popd
- name: Fetch setup_python action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
sparse-checkout: |
.github/actions/setup_python/action.yml
sparse-checkout-cone-mode: false
path: 'openvino'
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: ./openvino/.github/actions/setup_python
with:
version: ${{ env.PYTHON_VERSION }}
pip-cache-path: ${{ runner.os == 'Linux' && env.PIP_CACHE_PATH || '' }}
should-setup-pip-paths: ${{ runner.os == 'Linux' }}
self-hosted-runner: ${{ runner.os == 'Linux' }}
#
# Tests
#
- name: Install OpenVINO Python wheels (Linux and macOS)
if: runner.os != 'Windows'
run: |
# Install the core OV wheel
python3 -m pip install ${INSTALL_DIR}/tools/openvino-*.whl
# Install the core OV Tokenizers wheel
python3 -m pip install ${INSTALL_DIR}/openvino_tokenizers-*.whl
- name: Install OpenVINO Python wheels (Windows)
if: runner.os == 'Windows'
run: |
# Find and install the core OV wheel
$ovCoreWheelPath=Get-ChildItem -Path ${{ env.INSTALL_DIR }}\tools -Filter openvino-*.whl | % { $_.FullName }
python3 -m pip install "$ovCoreWheelPath"
# Find and install the core OV Tokenizers wheel
$ovCoreWheelPath=Get-ChildItem -Path ${{ env.INSTALL_DIR }} -Filter openvino_tokenizers-*.whl | % { $_.FullName }
python3 -m pip install "$ovCoreWheelPath"
- name: Install Python Layer tests dependencies
run: |
# To enable pytest parallel features
python3 -m pip install pytest-xdist[psutil]
# layer test requirements
python3 -m pip install -r ${{ env.LAYER_TESTS_INSTALL_DIR }}/requirements.txt
#
# Tests
#
- name: TensorFlow 1 Layer Tests - TF FE
if: fromJSON(inputs.affected-components).TF_FE.test
run: |
python3 -m pytest ${{ env.LAYER_TESTS_INSTALL_DIR }}/tensorflow_tests/ -n logical -m precommit --junitxml=${{ env.INSTALL_TEST_DIR }}/TEST-tf_fe.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP16
- name: TensorFlow 2 Layer Tests - TF FE
if: fromJSON(inputs.affected-components).TF_FE.test && runner.os != 'macOS' # Ticket: 123322
run: |
python3 -m pytest ${{ env.LAYER_TESTS_INSTALL_DIR }}/tensorflow2_keras_tests/ -n logical -m precommit --junitxml=${{ env.INSTALL_TEST_DIR }}/TEST-tf2_fe.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP16
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: ${{ !cancelled() }}
with:
name: test-results-python-tf-layers
path: |
${{ env.INSTALL_TEST_DIR }}/TEST*.html
${{ env.INSTALL_TEST_DIR }}/TEST*.xml
if-no-files-found: 'warn'

View File

@ -1,123 +0,0 @@
name: TensorFlow Models tests
on:
workflow_call:
inputs:
runner:
description: 'Machine on which the tests would run'
type: string
required: true
container:
description: 'JSON to be converted to the value of the "container" configuration for the job'
type: string
required: false
default: '{"image": null}'
model_scope:
description: 'Scope of models for testing.'
type: string
required: true
permissions: read-all
jobs:
TensorFlow_Models_Tests:
name: TensorFlow Models tests
timeout-minutes: ${{ inputs.model_scope != 'precommit' && 400 || 40 }}
runs-on: ${{ inputs.runner }}
container: ${{ fromJSON(inputs.container) }}
defaults:
run:
shell: bash
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
OPENVINO_REPO: ${{ github.workspace }}/openvino
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
MODEL_HUB_TESTS_INSTALL_DIR: ${{ github.workspace }}/install/tests/model_hub_tests
NUMBER_OF_REPLICAS: 2
steps:
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tokenizers extension
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_tokenizers_wheel
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
run: |
echo "OPENVINO_REPO=$GITHUB_WORKSPACE/openvino" >> "$GITHUB_ENV"
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
echo "MODEL_HUB_TESTS_INSTALL_DIR=$GITHUB_WORKSPACE/install/tests/model_hub_tests" >> "$GITHUB_ENV"
echo "TFHUB_CACHE_DIR=/mount/testdata$((GITHUB_RUN_NUMBER % NUMBER_OF_REPLICAS))/tfhub_models" >> "$GITHUB_ENV"
echo $TFHUB_CACHE_DIR
echo "HF_HUB_CACHE=/mount/testdata$((GITHUB_RUN_NUMBER % NUMBER_OF_REPLICAS))/hugging_face" >> "$GITHUB_ENV"
echo $HF_HUB_CACHE
- name: Extract OpenVINO packages
run: |
pushd ${INSTALL_DIR}
tar -xzf openvino_package.tar.gz -C ${INSTALL_DIR}
popd
pushd ${INSTALL_TEST_DIR}
tar -xzf openvino_tests.tar.gz -C ${INSTALL_DIR}
popd
- name: Fetch setup_python action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
sparse-checkout: |
.github/actions/setup_python/action.yml
sparse-checkout-cone-mode: false
path: 'openvino'
- name: Setup Python 3.11
uses: ./openvino/.github/actions/setup_python
with:
version: '3.11'
should-setup-pip-paths: 'false'
self-hosted-runner: ${{ contains(inputs.runner, 'aks') }}
- name: Install OpenVINO Python wheels
run: |
python3 -m pip install ${INSTALL_DIR}/tools/openvino-*
python3 -m pip install ${INSTALL_DIR}/openvino_tokenizers-*
- name: Install TF Models tests requirements
run: python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/tensorflow/requirements.txt
- name: TensorFlow Models Tests - TF FE
run: |
export PYTHONPATH=${MODEL_HUB_TESTS_INSTALL_DIR}:$PYTHONPATH
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/tensorflow/test_tf_convert_model.py -m ${{ inputs.model_scope }} \
--html=${INSTALL_TEST_DIR}/TEST-tf_fe_models_convert_model_${{ inputs.model_scope }}.html --self-contained-html -v
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/tensorflow/test_tf_read_model.py -m ${{ inputs.model_scope }} \
--html=${INSTALL_TEST_DIR}/TEST-tf_fe_models_read_model_${{ inputs.model_scope }}.html --self-contained-html -v
# decouple notebook tests due to GitHub issue in tensorflow_hub https://github.com/tensorflow/hub/issues/903
# and use WA to switch to (legacy) Keras 2
TF_USE_LEGACY_KERAS=1 python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/tensorflow/test_tf_hub_api_notebooks.py -m ${{ inputs.model_scope }} \
--html=${INSTALL_TEST_DIR}/TEST-tf_fe_models_notebooks_${{ inputs.model_scope }}.html --self-contained-html -v
env:
TEST_DEVICE: CPU
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: ${{ !cancelled() }}
with:
name: test-results-tensorflow-models-${{ inputs.model_scope }}
path: |
${{ env.INSTALL_TEST_DIR }}/TEST*.html
if-no-files-found: 'error'

Some files were not shown because too many files have changed in this diff Show More