Compare commits

..

54 Commits

Author SHA1 Message Date
Sebastian Golebiewski 9bdb35aafa
Fix images for notebooks (#22129) 2024-01-12 14:18:57 +01:00
Sebastian Golebiewski d5bf02b634
Updating thumbnails for notebooks (#21814) 2023-12-21 12:55:06 +01:00
Karol Blaszczak 67591efda5
[DOCS] NPU update 23.2 (#21810) 2023-12-21 10:08:15 +01:00
Maciej Smyk 4a7063285b
Update GPU.md (#21758) 2023-12-19 15:19:08 +01:00
Karol Blaszczak 899b47dccd
[DOCS] fix auto toctree 23.2 (#21757) 2023-12-19 14:20:04 +01:00
Alexander Suvorov 61a072baaf
[DOCS] Update 2023.2 Selector Tool with nightly arch (#21671)
port: https://github.com/openvinotoolkit/openvino/pull/21629
2023-12-14 18:01:16 +01:00
Karol Blaszczak a97aa14ae5
[DOCS] NPU articles port to 23.2 (#21659)
merging with the reservation that additional changes will be done in a follow-up PR
2023-12-14 17:31:26 +01:00
Sebastian Golebiewski 1223b2190d
Updating Interactive Tutorials (#21554) 2023-12-13 10:33:24 +01:00
Maciej Smyk 80f94dd5d3
Update learn_openvino.md (#21541) 2023-12-08 11:28:38 +01:00
Tatiana Savina c3190f1d43
[DOCS] Port notes to 23.2 (#21471)
* change wording (#21428)

* add legacy note to mo section

* port note

* change header

* add note to pot

* port deprecation note

* resolve conflicts

* fix directive
2023-12-05 12:01:10 +01:00
Tatiana Savina 293293bbfe
[DOCS] Port optimization doc changes to release docs (#21366)
* [DOCS] Update reference in quantization_w_accuracy_control.md (#21337)

* Update quantization_w_accuracy_control.md

avoid 404

* Update quantization_w_accuracy_control.md

* migrate to new convert API (#21323)

---------

Co-authored-by: Maksim Proshin <maksim.proshin@intel.com>
Co-authored-by: Alexander Suslov <alexander.suslov@intel.com>
2023-11-29 12:12:45 +01:00
Andrey Babushkin e1da28e4a4
Rename AKS runners' labels for consistency (#21282) (#21345) 2023-11-28 22:47:26 +04:00
Sebastian Golebiewski 2a4a5f6ab5
Update of Sample Articles (#21316) 2023-11-27 16:15:18 +01:00
Karol Blaszczak dc4265b2d3
[DOCS] fixes lts disclaimer 23.2 2023-11-27 14:57:54 +01:00
Andrzej Kopytko a836462f77
[DOCS] tabs styling (#21272) (#21313) 2023-11-27 14:50:09 +01:00
Anastasia Kuporosova 613ef81471
[Docs][PyOV] add missing docs (#21201) (#21232)
* [Docs][PyOV] add missing docs (#21201)

* [Docs][PyOV] add missing docs

* more dirs

* one more missed

* missed opsets
2023-11-22 15:54:34 +04:00
Maciej Smyk 4c31a9bef7
[DOCS] README files for Samples + name update for Get Started for 23.2 (#21159)
Co-authored-by: Sebastian Golebiewski <sebastianx.golebiewski@intel.com>
2023-11-21 13:20:16 +01:00
Sebastian Golebiewski 9d94661392
Update reference to Supported Model Formats (#21218) 2023-11-21 13:18:34 +01:00
Sebastian Golebiewski 338aa3d85b
Fix math formula in Elu_1 (#21204) 2023-11-21 12:09:41 +01:00
Sebastian Golebiewski 4e992aef3a
Fixing links in notebooks (#21210) 2023-11-21 12:09:15 +01:00
Sebastian Golebiewski 7e18bd074a
[DOCS] New page design for notebooks (#21158) 2023-11-17 16:34:42 +01:00
Andrzej Kopytko d8e7ea51b5
[DOCS] Benchmark update files pory
port #21152
2023-11-17 14:02:26 +01:00
Karol Blaszczak a60c921189
[DOCS] fix npu mention port 23.2
port: https://github.com/openvinotoolkit/openvino/pull/21147
2023-11-17 12:58:27 +00:00
Ilya Lavrenov fb0aee2d1c
Fixed install rules for case when only ONNX FE is enabled (#21120) 2023-11-16 16:26:47 +04:00
Sebastian Golebiewski 1a6a4443a1
[DOCS] Selector tool for 2023.2 (#21119)
* Update selector tool

* update ov version title

* fix versions

---------

Co-authored-by: Alexander Suvorov <alexander.suvorov@intel.com>
2023-11-16 13:23:18 +01:00
Sebastian Golebiewski 40fbe51621
[DOCS] Update optimization docs with ov.save_model() instead of ov.serialize() - for 23.2 (#21121)
* introduced ov.save_model(...) to the ptq code examples

* replied to comments

* fixed rendering

---------

Co-authored-by: Alexander Suslov <alexander.suslov@intel.com>
2023-11-16 13:10:21 +01:00
Tatiana Savina 560798f00c
update links (#21118) 2023-11-16 11:15:53 +00:00
Roman Kazantsev b868e6e271
[TF Hub] Move pre-commit to Kaggle model links (#21113) (#21116)
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
2023-11-16 15:05:59 +04:00
Tatiana Savina 575f51e3cf
change install (#21117) 2023-11-16 11:47:00 +01:00
Sebastian Golebiewski 69b5836648
[DOCS] GraphJs refactor- port for 23.2 (#21112)
* [DOCS]InitGraphChanges

* [DOCS] Benchmark files update

* [DOCS] update files from 0635

* [DOCS] fix

* [DOCS ]fix2

---------

Co-authored-by: Kopytko, AndrzejX <andrzejx.kopytko@intel.com>
2023-11-16 11:03:22 +01:00
Karol Blaszczak 2cabc6b6bd
[DOCS] final update of relnotes (#21108) 2023-11-16 10:23:04 +01:00
Andrzej Kopytko 553d15622e
[DOCS] update version of conf.py (#21093) 2023-11-16 09:20:34 +01:00
Karol Blaszczak db4724aeb5
[DOCS] home-tile-removal (#21095) 2023-11-15 16:21:39 +01:00
Karol Blaszczak cadce8fbd4
[DOCS] sysreq n relnotes 23.2 2023-11-15 15:31:30 +01:00
Tatiana Savina 4e702d96e1
deprecate deployment manager (#21089) 2023-11-15 13:48:54 +01:00
Tatiana Savina c84cdeb264
remove link to page (#21074) 2023-11-14 18:44:47 +01:00
Maciej Smyk 5d4de9117c
Port of 21010 & 20988 (#21069) 2023-11-14 17:29:32 +01:00
Maciej Smyk 62460e1e2b
[DOCS] Updating links for 23.2 2023-11-13 12:23:49 +01:00
Vladislav Golubev 0140796841
FuseU4WeightsAndZeroPoint tests: avoid std::vector<std::int8_t> usage (#21022) 2023-11-13 14:19:11 +04:00
Maciej Smyk 49c8526e20
port from 20812 (#20959) 2023-11-08 15:53:24 +01:00
Maciej Smyk 17ad6116c8
[DOCS] Small fixes in articles for 23.2 (#20952)
* Fixes

* Update deployment_intro.md

* Update docs/articles_en/openvino_workflow/deployment_intro.md

Co-authored-by: Sebastian Golebiewski <sebastianx.golebiewski@intel.com>

---------

Co-authored-by: Sebastian Golebiewski <sebastianx.golebiewski@intel.com>
2023-11-08 13:40:09 +01:00
Sebastian Golebiewski cda12b6de5
Updating notebooks (#20937) 2023-11-08 12:26:49 +01:00
Irina Efode fc678416a6
[CONFORMANCE] Partial cherry-pick of PR20756 to exclude from model list for conformance (#20955) 2023-11-08 11:20:39 +00:00
Karol Blaszczak 5ee4090e10
[DOCS] improving the "conversion" section v2 (#20904)
adjustments to conversion and workflow
2023-11-08 11:12:01 +01:00
Sebastian Golebiewski 3a0abdfaa8
Fixing OS list in System Requirements for YUM (#20936) 2023-11-07 17:21:56 +01:00
Sebastian Golebiewski 210608ca3a
Fixing link in Get Started article for 23.2 (#20931)
Porting https://github.com/openvinotoolkit/openvino/pull/20881
2023-11-07 16:35:12 +01:00
Piotr Krzemiński 98072bbbae
[SPEC] Reapply multinomial docs changes (#20805) 2023-11-03 07:18:43 +01:00
Anastasiia Pnevskaia 29d43d85ce
[DOC] Update list of TF formats imported from memory. (#20834) (#20843)
* Update list of TF formats.

* Minor correction.

* Added comment.

* Update docs/articles_en/openvino_workflow/model_preparation/Convert_Model_From_TensorFlow.md



* Model changed.

* Update docs/articles_en/openvino_workflow/model_preparation/Convert_Model_From_TensorFlow.md



---------

Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
2023-11-02 18:35:37 +04:00
Ivan Tikhonov cfd42bd2cb
Handle Reshape's special zero in SimplifySecondInputOfReshape (#20785) (#20820)
* Handle Reshape's special zero in SimplifySecondInputOfReshape

SimplifySecondInputOfReshape detects ShapeOf->Gather->Concat
subgraphs on Reshape's second input and replaces ShapeOf->Gather
with a Constant with zero(s). Currently it works only with Reshapes
that have special_zero set to true, but it can work for Reshapes
with special_zero == false if non-Gather inputs to Concat are Constants
and don't contain any zero.

Ticket: CVS-123434

* fix no default output

Co-authored-by: Mateusz Tabaka <mateusz.tabaka@intel.com>
2023-11-02 17:43:25 +04:00
Andrei Kochin 663bf04208
Revert Bitwise ops in PyTorch FE (#20813)
* revert bitwise op for PT FE

* revert coverity fixes
2023-11-02 13:08:55 +00:00
Surya Siddharth Pemmaraju 3ebb8eb315
Added fix for concat in torchfx (#20806) 2023-11-02 15:18:53 +04:00
Gorokhov Dmitriy 757b466c5d
[CPU] Fixed port mismatch in Eltwise fusion graph optimization (#20807) 2023-11-02 11:31:20 +04:00
Alina Kladieva df19f18f37
Update refs for public CI pipelines (#20798) 2023-10-31 16:39:41 +00:00
Tatiana Savina 017eb943d5
[DOCS] Update PyPI links and pre-release note (#20799)
* update links and rm pre-release note

* update ov-dev
2023-10-31 13:56:33 +00:00
20666 changed files with 917089 additions and 847149 deletions

View File

@ -0,0 +1,53 @@
# Copyright (C) 2018-2021 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""
Analyze GTest logs
"""
import re
from argparse import ArgumentParser
def get_passed_tests(log_file_path):
"""Gets passed tests with OK status"""
ok_test_line_pattern = "[ OK ] "
ok_tests = []
with open(log_file_path) as log_file_obj:
for line in log_file_obj.readlines():
if ok_test_line_pattern in line:
ok_tests.append(line.split(ok_test_line_pattern)[1])
return ok_tests
def get_total_time(tests):
"""Gets total execution time (sec)"""
re_compile_time = re.compile(r".+ \(([0-9]+) ms\)")
total_time = 0.0
for test in tests:
re_time = re_compile_time.match(test)
if re_time:
total_time += int(re_time.group(1)) / 1000
else:
print("No time in the test line:", test)
return total_time
def main():
"""The main entry point function"""
arg_parser = ArgumentParser()
arg_parser.add_argument(
"--log-file", metavar="PATH", default="gtest.log", help="Path to GTest log file"
)
args = arg_parser.parse_args()
passed_tests = get_passed_tests(args.log_file)
print("PASSED tests count:", len(passed_tests))
print("Total execution time of passed tests (sec):", get_total_time(passed_tests))
print("\nPASSED tests:")
print("".join(sorted(passed_tests)))
if __name__ == "__main__":
main()

View File

@ -0,0 +1,6 @@
TransposeOpTest.NHWC2NCHW
TransposeOpTest.NCHW2NHWC
TransposeOpTest.TwoDim_int16
GatherOpTest.Gather_axis1_indices2d_int16
SoftmaxOperator.ThreeDimsAxis1
SoftmaxOperator.ThreeDimsAxis0

View File

@ -0,0 +1 @@
rel-1.14.0

View File

@ -0,0 +1,165 @@
resources:
repositories:
- repository: openvino_contrib
type: github
endpoint: openvinotoolkit
name: openvinotoolkit/openvino_contrib
ref: releases/2023/2
variables:
- group: github
jobs:
- job: Lin
# About 150% of total time
timeoutInMinutes: '90'
pool:
name: LIN_VMSS_VENV_F16S_U20_WU2
variables:
system.debug: true
VSTS_HTTP_RETRY: 5
VSTS_HTTP_TIMEOUT: 200
BUILD_TYPE: Release
REPO_DIR: $(Build.Repository.LocalPath)
OPENVINO_CONTRIB_REPO_DIR: $(REPO_DIR)/../openvino_contrib
WORK_DIR: $(Pipeline.Workspace)/_w
BUILD_DIR: $(WORK_DIR)/build
BUILD_SAMPLES_DIR: $(WORK_DIR)/build_samples
INSTALL_DIR: $(WORK_DIR)/install_pkg
SETUPVARS: $(INSTALL_DIR)/setupvars.sh
TMP_DIR: /mnt/tmp
SHARE_DIR: /mount/cinfsshare/onnxtestdata
CCACHE_DIR: $(SHARE_DIR)/ccache/master/linux_coverity
LD_LIBRARY_PATH: $(Agent.ToolsDirectory)/Python/$(OV_PYTHON_VERSION)/x64/lib
OV_PYTHON_VERSION: 3.11.2 # Full version of Python its required for LD_LIBRARY_PATH. More details https://github.com/microsoft/azure-pipelines-tool-lib/blob/master/docs/overview.md#tool-cache
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '$(OV_PYTHON_VERSION)' # Setting only major & minor version will download latest release from GH repo example 3.10 will be 3.10.10.
addToPath: true
disableDownloadFromRegistry: false
architecture: 'x64'
githubToken: $(auth_token)
displayName: Setup Python 3.11
name: setupPython
- bash: |
#!/bin/bash
python -V
- script: |
curl -H Metadata:true --noproxy "*" "http://169.254.169.254/metadata/instance?api-version=2019-06-01"
whoami
uname -a
echo Python3 info ; which python3 ; python3 --version
echo Python info ; which python ; python --version
echo Java info ; which java ; java -version
echo gcc info ; which gcc ; gcc --version
echo cmake info ; which cmake ; cmake --version
lsb_release
env
cat /proc/cpuinfo
cat /proc/meminfo
cat /etc/fstab
vmstat -s
df
lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"
free -h
displayName: 'System info'
- script: |
set -e
rm -rf $(WORK_DIR) ; mkdir $(WORK_DIR)
rm -rf $(BUILD_DIR) ; mkdir $(BUILD_DIR)
rm -rf $(BUILD_SAMPLES_DIR) ; mkdir $(BUILD_SAMPLES_DIR)
sudo rm -rf $(TMP_DIR) ; sudo mkdir $(TMP_DIR) ; sudo chmod 777 -R $(TMP_DIR)
sudo mkdir -p $(SHARE_DIR)
sudo apt --assume-yes update && sudo apt --assume-yes install nfs-common
sudo mount -vvv -t nfs cinfsshare.file.core.windows.net:/cinfsshare/onnxtestdata $(SHARE_DIR) -o vers=4,minorversion=1,sec=sys
mkdir -p $(CCACHE_DIR)
displayName: 'Make dir'
- checkout: self
clean: 'true'
submodules: 'true'
path: openvino
- checkout: openvino_contrib
clean: 'true'
submodules: 'true'
path: openvino_contrib
- script: |
set -e
sudo -E $(REPO_DIR)/install_build_dependencies.sh
# Move jdk into contrib
sudo apt --assume-yes install openjdk-11-jdk
# Speed up build
sudo apt -y --no-install-recommends install unzip
wget https://github.com/ninja-build/ninja/releases/download/v1.10.2/ninja-linux.zip
unzip ninja-linux.zip
sudo cp -v ninja /usr/local/bin/
displayName: 'Install dependencies'
- task: CMake@1
inputs:
# Coverity has too many PARSE_ERROR errors with ENABLE_FASTER_BUILD=ON. Disabling FASTER_BUILD.
cmakeArgs: >
-G "Ninja Multi-Config"
-DENABLE_CPPLINT=OFF
-DCMAKE_VERBOSE_MAKEFILE=ON
-DENABLE_FASTER_BUILD=OFF
-DENABLE_STRICT_DEPENDENCIES=OFF
-DBUILD_nvidia_plugin=OFF
-DOPENVINO_EXTRA_MODULES=$(OPENVINO_CONTRIB_REPO_DIR)/modules
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
-DCMAKE_C_COMPILER_LAUNCHER=ccache
-S $(REPO_DIR)
-B $(BUILD_DIR)
displayName: "Cmake configure"
- script: ls -alR $(REPO_DIR)/temp/
displayName: 'List temp SDKs'
- script: ccache --zero-stats --max-size=50G --show-config
displayName: 'Clean ccache stats'
- script: |
set -e
wget https://scan.coverity.com/download/linux64 --post-data "token=$(COVERITY_TOKEN)&project=openvino" -O coverity_tool.tgz
tar xvf coverity_tool.tgz
rm coverity_tool.tgz
workingDirectory: $(WORK_DIR)
displayName: 'Install coverity tool'
- script: |
$(WORK_DIR)/cov-analysis*/bin/cov-build --dir $(BUILD_DIR)/cov-int \
cmake --build $(BUILD_DIR) --parallel --config $(BUILD_TYPE)
env:
CCACHE_DIR: $(CCACHE_DIR)
CCACHE_TEMPDIR: $(TMP_DIR)/ccache
CCACHE_BASEDIR: $(Pipeline.Workspace)
CCACHE_MAXSIZE: 50G
displayName: 'Build Lin with Coverity'
- script: ccache --show-stats
displayName: 'Show ccache stats'
- script: ls -alR $(REPO_DIR)/bin/
displayName: 'List bin files'
- script: tar -C $(BUILD_DIR) -czvf openvino.tgz cov-int
workingDirectory: $(BUILD_DIR)
displayName: 'Pack cov-int folder for submission'
- script: |
curl --form token=$(COVERITY_TOKEN) \
--form email=$(COVERITY_USER) \
--form file=@openvino.tgz \
--form version="$(Build.SourceVersion)" \
--form description="https://github.com/openvinotoolkit/openvino/runs/$(Build.BuildNumber)" \
https://scan.coverity.com/builds?project=openvino
workingDirectory: $(BUILD_DIR)
displayName: 'Submit for analysis'

View File

@ -0,0 +1,126 @@
trigger:
branches:
include:
- 'master'
- 'releases/*'
paths:
exclude:
- '*/docs/*'
- 'docs/*'
- '*/*.md'
- '*.md'
- '*/layer_tests_summary/*'
- '*/conformance/*'
- 'tools/*'
- 'tests/layer_tests/*'
pr:
drafts: 'false'
branches:
include:
- 'master'
- releases/*
paths:
exclude:
- '*/docs/*'
- 'docs/*'
- '*/*.md'
- '*.md'
- '*/layer_tests_summary/*'
- '*/conformance/*'
- 'tools/*'
- 'tests/layer_tests/*'
jobs:
- job: OpenVINO_ONNX_CI
strategy:
matrix:
Release:
BUILD_TYPE: 'Release'
TOX_COMMAND: 'tox && tox -e zoo_models'
Debug:
BUILD_TYPE: 'Debug'
TOX_COMMAND: 'tox'
maxParallel: '2'
# About 300% of total time
timeoutInMinutes: '90'
pool:
name: LIN_VMSS_VENV_ONNX_U20_WU2
variables:
system.debug: true
VSTS_HTTP_RETRY: 5
VSTS_HTTP_TIMEOUT: 200
REPO_DIR: $(Build.Repository.LocalPath)
WORK_DIR: $(Pipeline.Workspace)/_w
MODELS_DIR: /mount/cinfsshare/onnxtestdata
TMP_DIR: /mnt/tmp
ONNX_MODEL_ZOO_SHA: "d58213534f2a4d1c4b19ba62b3bb5f544353256e"
steps:
- script: |
curl -H Metadata:true --noproxy "*" "http://169.254.169.254/metadata/instance?api-version=2019-06-01"
whoami
uname -a
echo Python3 info ; which python3 ; python3 --version
echo Python info ; which python ; python --version
echo gcc info ; which gcc ; gcc --version
echo cmake info ; which cmake ; cmake --version
lsb_release
env
cat /proc/cpuinfo
cat /proc/meminfo
cat /etc/fstab
vmstat -s
df
lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"
free -h
displayName: 'System info'
- script: |
set -e
rm -rf $(WORK_DIR) ; mkdir $(WORK_DIR)
sudo mkdir -p $(MODELS_DIR)
sudo apt --assume-yes update && sudo apt --assume-yes install nfs-common
sudo apt install nfs-common -y
sudo mount -vvv -t nfs cinfsshare.file.core.windows.net:/cinfsshare/onnxtestdata $(MODELS_DIR) -o vers=4,minorversion=1,sec=sys
mkdir -p $(MODELS_DIR)/models_data
displayName: 'Make dirs'
- checkout: self
clean: 'true'
submodules: 'true'
path: openvino
- script: |
set -e
apt-get update && apt-get install -y lsb-release && apt-get clean all
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
displayName: 'Install dependencies'
- script:
src/frontends/onnx/tests/tests_python/model_zoo_preprocess.sh -d $(MODELS_DIR)/models_data -o -s "$(ONNX_MODEL_ZOO_SHA)"
displayName: 'Update models'
condition: ne(variables['BUILD_TYPE'], 'Debug')
- script: |
sudo docker build \
--tag=openvino-onnx-ci-image \
--file=.ci/openvino-onnx/Dockerfile \
--build-arg BUILD_TYPE=$(BUILD_TYPE) .
displayName: 'Docker build $(BUILD_TYPE)'
- script: sudo fallocate -l 64G /swapfile ; sudo mkswap /swapfile ; sudo swapon /swapfile ; df ; free -h
displayName: 'Create swap'
- script: |
sudo docker run \
--name openvino-onnx-ci-container \
--volume $(MODELS_DIR)/models_data/model_zoo/onnx_model_zoo_$(ONNX_MODEL_ZOO_SHA):/root/.onnx/model_zoo/onnx_model_zoo \
--volume $(MODELS_DIR)/msft:/root/.onnx/model_zoo/MSFT openvino-onnx-ci-image \
/bin/bash -c "$(TOX_COMMAND)"
displayName: 'Docker run $(BUILD_TYPE)'

320
.ci/azure/windows.yml Normal file
View File

@ -0,0 +1,320 @@
trigger:
branches:
include:
- 'master'
- 'releases/*'
paths:
exclude:
- '*/docs/*'
- 'docs/*'
- '*/*.md'
- '*.md'
- '*/layer_tests_summary/*'
- '*/conformance/*'
pr:
branches:
include:
- 'master'
- 'releases/*'
paths:
exclude:
- '*/docs/*'
- 'docs/*'
- '*/*.md'
- '*.md'
- '*/layer_tests_summary/*'
- '*/conformance/*'
resources:
repositories:
- repository: openvino_contrib
type: github
endpoint: openvinotoolkit
name: openvinotoolkit/openvino_contrib
ref: releases/2023/2
jobs:
- job: Win
strategy:
matrix:
Static:
CMAKE_BUILD_SHARED_LIBS: 'OFF'
# Dynamic:
# CMAKE_BUILD_SHARED_LIBS: 'ON'
maxParallel: '2'
# About 150% of total time
timeoutInMinutes: '270' #Temporary change
pool:
name: WIN_VMSS_VENV_D8S_WU2
variables:
system.debug: true
VSTS_HTTP_RETRY: 5
VSTS_HTTP_TIMEOUT: 200
BUILD_TYPE: Release
REPO_DIR: $(Build.Repository.LocalPath)
OPENVINO_CONTRIB_REPO_DIR: $(REPO_DIR)\..\openvino_contrib
WORK_DIR: $(Pipeline.Workspace)\_w
BUILD_DIR: $(WORK_DIR)\build
BUILD_SAMPLES_DIR: $(WORK_DIR)\build_samples
BUILD_SAMPLES_TESTS_DIR: $(WORK_DIR)\build_samples_tests
MSVS_VARS_PATH: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat
MSVC_COMPILER_PATH: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\bin\Hostx64\x64\cl.exe
INSTALL_DIR: $(WORK_DIR)\install_pkg
INSTALL_TEST_DIR: $(INSTALL_DIR)\tests
SETUPVARS: $(INSTALL_DIR)\setupvars.bat
CMAKE_VERSION: 3.24.0
CMAKE_CMD: $(WORK_DIR)\cmake-$(CMAKE_VERSION)-windows-x86_64\cmake-$(CMAKE_VERSION)-windows-x86_64\bin\cmake.exe
OV_CMAKE_TOOLCHAIN_FILE: $(REPO_DIR)\cmake\toolchains\mt.runtime.win32.toolchain.cmake
PYTHON_EXE: C:\hostedtoolcache\windows\Python\3.8.2\x64\python.exe
steps:
- script: |
rd /Q /S $(WORK_DIR) & mkdir $(WORK_DIR)
rd /Q /S $(BUILD_DIR) & mkdir $(BUILD_DIR)
rd /Q /S $(BUILD_SAMPLES_DIR) & mkdir $(BUILD_SAMPLES_DIR)
rd /Q /S $(BUILD_SAMPLES_TESTS_DIR) & mkdir $(BUILD_SAMPLES_TESTS_DIR)
displayName: 'Make dir'
- script: |
powershell -command "Invoke-RestMethod -Headers @{\"Metadata\"=\"true\"} -Method GET -Uri http://169.254.169.254/metadata/instance/compute?api-version=2019-06-01 | format-custom"
where $(PYTHON_EXE)
$(PYTHON_EXE) --version
where java
java -version
wmic computersystem get TotalPhysicalMemory
wmic cpu list
wmic logicaldisk get description,name
wmic VOLUME list
set
displayName: 'System info'
- checkout: self
clean: 'true'
submodules: 'true'
path: openvino
- checkout: openvino_contrib
clean: 'true'
submodules: 'true'
path: openvino_contrib
- script: |
$(PYTHON_EXE) -m pip install --upgrade pip
rem For running Python API tests
$(PYTHON_EXE) -m pip install -r $(REPO_DIR)\src\bindings\python\src\compatibility\openvino\requirements-dev.txt
$(PYTHON_EXE) -m pip install -r $(REPO_DIR)\src\bindings\python\wheel\requirements-dev.txt
$(PYTHON_EXE) -m pip install -r $(REPO_DIR)\src\bindings\python\requirements.txt
rem For running Paddle frontend unit tests
$(PYTHON_EXE) -m pip install -r $(REPO_DIR)\src\frontends\paddle\tests\requirements.txt
rem For running ONNX frontend unit tests
$(PYTHON_EXE) -m pip install -r $(REPO_DIR)\src\frontends\onnx\tests\requirements.txt
rem For running TensorFlow frontend unit tests
$(PYTHON_EXE) -m pip install -r $(REPO_DIR)\src\frontends\tensorflow\tests\requirements.txt
rem For MO unit tests
$(PYTHON_EXE) -m pip install -r $(REPO_DIR)\tools\mo\requirements.txt
$(PYTHON_EXE) -m pip install -r $(REPO_DIR)\tools\mo\requirements_dev.txt
rem Speed up build
powershell -command "Invoke-WebRequest https://github.com/Kitware/CMake/releases/download/v$(CMAKE_VERSION)/cmake-$(CMAKE_VERSION)-windows-x86_64.zip -OutFile cmake-$(CMAKE_VERSION)-windows-x86_64.zip"
powershell -command "Expand-Archive -Force cmake-$(CMAKE_VERSION)-windows-x86_64.zip"
powershell -command "Invoke-WebRequest https://github.com/ninja-build/ninja/releases/download/v1.10.2/ninja-win.zip -OutFile ninja-win.zip"
powershell -command "Expand-Archive -Force ninja-win.zip"
workingDirectory: $(WORK_DIR)
displayName: 'Install dependencies'
- powershell: |
Write-Host "##vso[task.setvariable variable=CMAKE_TOOLCHAIN_FILE]$(OV_CMAKE_TOOLCHAIN_FILE)"
condition: eq(variables['CMAKE_BUILD_SHARED_LIBS'], 'ON')
displayName: "Set cmake toolchain"
- script: |
set PATH=$(WORK_DIR)\ninja-win;%PATH% && ^
call "$(MSVS_VARS_PATH)" && $(CMAKE_CMD) ^
-G "Ninja Multi-Config" ^
-DENABLE_CPPLINT=OFF ^
-DENABLE_ONEDNN_FOR_GPU=$(CMAKE_BUILD_SHARED_LIBS) ^
-DBUILD_SHARED_LIBS=$(CMAKE_BUILD_SHARED_LIBS) ^
-DENABLE_FASTER_BUILD=ON ^
-DENABLE_TESTS=ON ^
-DCMAKE_COMPILE_WARNING_AS_ERROR=ON ^
-DENABLE_STRICT_DEPENDENCIES=OFF ^
-DPython3_EXECUTABLE=$(PYTHON_EXE) ^
-DENABLE_PYTHON=ON ^
-DBUILD_nvidia_plugin=OFF ^
-DCUSTOM_OPERATIONS="calculate_grid;complex_mul;fft;grid_sample;sparse_conv;sparse_conv_transpose" ^
-DOPENVINO_EXTRA_MODULES=$(OPENVINO_CONTRIB_REPO_DIR)\modules ^
-DCMAKE_C_COMPILER:PATH="$(MSVC_COMPILER_PATH)" ^
-DCMAKE_CXX_COMPILER:PATH="$(MSVC_COMPILER_PATH)" ^
-S $(REPO_DIR) ^
-B $(BUILD_DIR)
displayName: 'CMake OpenVINO'
- script: dir $(REPO_DIR)\temp\ /s
displayName: 'List temp SDKs'
- script: |
set PATH=$(WORK_DIR)\ninja-win;%PATH% && ^
call "$(MSVS_VARS_PATH)" && $(CMAKE_CMD) --build $(BUILD_DIR) --parallel --config Release"
displayName: 'Build Win'
- script: dir $(REPO_DIR)\bin\ /s
displayName: 'List bin files'
- script: $(CMAKE_CMD) -DCMAKE_INSTALL_PREFIX=$(INSTALL_DIR) -P $(BUILD_DIR)/cmake_install.cmake
displayName: 'Install'
- script: dir $(INSTALL_DIR) /s
displayName: 'List install files'
- script: $(PYTHON_EXE) -m pip install openvino-dev --find-links=$(INSTALL_DIR)\tools
displayName: 'Install Wheels'
- script: |
call "$(MSVS_VARS_PATH)" && ^
$(CMAKE_CMD) ^
-DCMAKE_C_COMPILER:PATH="$(MSVC_COMPILER_PATH)" ^
-DCMAKE_CXX_COMPILER:PATH="$(MSVC_COMPILER_PATH)" ^
-S $(REPO_DIR)\tests\samples_tests ^
-B $(BUILD_SAMPLES_TESTS_DIR)
displayName: 'CMake Samples Tests'
- script: $(CMAKE_CMD) -DCOMPONENT=tests -DCMAKE_INSTALL_PREFIX=$(INSTALL_DIR) -P $(BUILD_SAMPLES_TESTS_DIR)\cmake_install.cmake
displayName: 'Install Samples Tests'
- script: |
$(INSTALL_DIR)\samples\cpp\build_samples_msvc.bat -i $(INSTALL_DIR)
if not exist %USERPROFILE%\Documents\Intel\OpenVINO\openvino_cpp_samples_build\ exit 1
displayName: 'Build cpp samples'
- script: |
$(INSTALL_DIR)\samples\c\build_samples_msvc.bat -i $(INSTALL_DIR)
if not exist %USERPROFILE%\Documents\Intel\OpenVINO\openvino_c_samples_build\ exit 1
displayName: 'Build c samples'
- script: $(PYTHON_EXE) -m pip install -r $(INSTALL_TEST_DIR)\smoke_tests\requirements.txt
displayName: 'Install dependencies for samples smoke tests'
- script: |
call $(SETUPVARS) && ^
$(PYTHON_EXE) -m pytest $(INSTALL_DIR)\tests\smoke_tests\ --env_conf $(INSTALL_TEST_DIR)\smoke_tests\env_config.yml -s --junitxml=$(INSTALL_TEST_DIR)/TEST-SamplesSmokeTests.xml
env:
IE_APP_PATH: $(INSTALL_DIR)\samples_bin
IE_APP_PYTHON_PATH: $(INSTALL_DIR)\samples\python\
SHARE: $(INSTALL_DIR)\tests\smoke_tests\samples_smoke_tests_data\
WORKSPACE: $(INSTALL_DIR)
displayName: 'Samples Smoke Tests'
- script: $(CMAKE_CMD) -DCMAKE_INSTALL_PREFIX=$(INSTALL_DIR) -DCOMPONENT=tests -P $(BUILD_DIR)\cmake_install.cmake
displayName: 'Install tests'
- script: dir $(INSTALL_DIR) /s
displayName: 'List install files'
- script: rd /Q /S $(BUILD_DIR)
displayName: 'Clean build dir'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_core_unit_tests --gtest_print_time=1 --gtest_filter=-*IE_GPU* --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-NGraphUT.xml
displayName: 'OV Core UT'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_inference_functional_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-InferenceFunc.xml
displayName: 'Inference Func Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_inference_unit_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-InferenceUnit.xml
displayName: 'Inference Unit Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_proxy_plugin_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-OVProxyTests.xml
displayName: 'OV Proxy Plugin Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_hetero_unit_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-OVHeteroUnitTests.xml
displayName: 'OV Hetero Unit Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_hetero_func_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-OVHeteroFuncTests.xml
displayName: 'OV Hetero Func Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_conditional_compilation_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ConditionalCompilation.xml
displayName: 'Conditional Compilation Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_ir_frontend_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-IRFrontend.xml
displayName: 'IR Frontend Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_onnx_frontend_tests --gtest_print_time=1 --gtest_filter=-*IE_GPU* --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ONNXFrontend.xml
displayName: 'ONNX Frontend Tests'
# TODO Reenable PDPD after paddlepaddle==2.5.2 with compliant protobuf is released (ticket 95904)
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\paddle_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-Paddle.xml
displayName: 'Paddle Frontend UT'
enabled: 'false'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_tensorflow_frontend_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-Tensorflow.xml
displayName: 'TensorFlow Frontend Unit Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_tensorflow_common_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-TensorflowCommon.xml
displayName: 'TensorFlow Common Unit Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_tensorflow_lite_frontend_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-TensorflowLite.xml
displayName: 'TensorFlow Lite Frontend Unit Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_lp_transformations_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\LpTransformations.xml
displayName: 'Low Precision Transformations Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_transformations_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\Transformations.xml
displayName: 'Transformations Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_legacy_transformations_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\LegacyTransformations.xml
displayName: 'Legacy Transformations Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_util_tests --gtest_print_time=1 --gtest_output=xml:$(INSTALL_TEST_DIR)\CommonUtilTests.xml
displayName: 'Common Utils Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\InferenceEngineUnitTests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-InferenceEngineUnitTests.xml
displayName: 'IE UT old'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_snippets_func_tests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_snippets_func_tests.xml
displayName: 'Snippets Func Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_cpu_unit_tests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_cpu_unit_tests.xml
displayName: 'Intel CPU Unit Tests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_gna_unit_tests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_gna_unit_tests.xml
displayName: 'GNA UT'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_auto_unit_tests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_auto_unit_tests.xml
displayName: 'AUTO UT'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_auto_func_tests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_auto_func_tests.xml
displayName: 'AUTO FuncTests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_auto_batch_unit_tests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_auto_batch_unit_tests.xml
displayName: 'AutoBatch UT'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_template_func_tests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-templateFuncTests.xml
displayName: 'TEMPLATE FuncTests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_auto_batch_func_tests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_auto_batch_func_tests.xml
displayName: 'AutoBatch FuncTests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\InferenceEngineCAPITests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-InferenceEngineCAPITests.xml
displayName: 'IE CAPITests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_capi_test --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_capi_test.xml
displayName: 'OV CAPITests'
- script: call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_cpu_func_tests --gtest_filter=*smoke* --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_cpu_func_tests.xml
displayName: 'CPU FuncTests'
condition: and(succeeded(), eq(variables['CMAKE_BUILD_SHARED_LIBS'], 'OFF'))
- task: PublishTestResults@2
condition: always()
inputs:
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit, cTest
testResultsFiles: '**/TEST-*.xml'
#searchFolder: '$(BUILD_DIR)'
mergeTestResults: false # Optional
#failTaskOnFailedTests: false # Optional
#testRunTitle: 'Pre/Post-Commit' # Optional
buildPlatform: 'x64' # Optional
buildConfiguration: 'Windows' # Optional
#publishRunAttachments: true # Optional

View File

@ -0,0 +1,172 @@
trigger:
branches:
include:
- 'master'
- 'releases/*'
paths:
exclude:
- '*/docs/*'
- 'docs/*'
- '*/*.md'
- '*.md'
- '*/layer_tests_summary/*'
- '*/conformance/*'
- 'tools/*'
pr:
drafts: 'false'
branches:
include:
- 'master'
- 'releases/*'
paths:
exclude:
- '*/docs/*'
- 'docs/*'
- '*/*.md'
- '*.md'
- '*/layer_tests_summary/*'
- '*/conformance/*'
- 'tools/*'
resources:
repositories:
- repository: testdata
type: github
endpoint: openvinotoolkit
name: openvinotoolkit/testdata
ref: releases/2023/2
variables:
- group: github
jobs:
- job: WinCC
# About 150% of total time
timeoutInMinutes: '120'
pool:
name: WIN_VMSS_VENV_F8S_WU2
variables:
system.debug: true
VSTS_HTTP_RETRY: 5
VSTS_HTTP_TIMEOUT: 200
BUILD_TYPE: Release
REPO_DIR: $(Build.Repository.LocalPath)
MODELS_PATH: $(REPO_DIR)\..\testdata
WORK_DIR: $(Pipeline.Workspace)\_w
BUILD_DIR: $(WORK_DIR)\build
BUILD_DIR_2: $(WORK_DIR)\build_s
MSVS_VARS_PATH: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat
MSVC_COMPILER_PATH: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\bin\Hostx64\x64\cl.exe
INSTALL_DIR: $(WORK_DIR)\install_pkg
SETUPVARS: $(INSTALL_DIR)\setupvars.bat
steps:
- script: |
powershell -command "Invoke-RestMethod -Headers @{\"Metadata\"=\"true\"} -Method GET -Uri http://169.254.169.254/metadata/instance/compute?api-version=2019-06-01 | format-custom"
where python
python --version
where java
java -version
where cmake
cmake --version
wmic computersystem get TotalPhysicalMemory
wmic cpu list
wmic logicaldisk get description,name
wmic VOLUME list
set
displayName: 'System info'
- script: |
rd /Q /S $(WORK_DIR) & mkdir $(WORK_DIR)
rd /Q /S $(BUILD_DIR) & mkdir $(BUILD_DIR)
rd /Q /S $(BUILD_DIR_2) & mkdir $(BUILD_DIR_2)
displayName: 'Make dir'
- checkout: self
clean: 'true'
submodules: 'true'
path: openvino
- script: |
rem Speed up build
powershell -command "Invoke-WebRequest https://github.com/ninja-build/ninja/releases/download/v1.10.2/ninja-win.zip -OutFile ninja-win.zip"
powershell -command "Expand-Archive -Force ninja-win.zip"
workingDirectory: $(WORK_DIR)
displayName: 'Install dependencies'
- checkout: testdata
clean: 'true'
lfs: 'true'
path: testdata
- script: |
set PATH=$(WORK_DIR)\ninja-win;%PATH%
call "$(MSVS_VARS_PATH)" && cmake ^
-G Ninja ^
-DENABLE_CPPLINT=OFF ^
-DENABLE_GAPI_PREPROCESSING=OFF ^
-DENABLE_PLUGINS_XML=ON ^
-DCMAKE_COMPILE_WARNING_AS_ERROR=ON ^
-DCMAKE_BUILD_TYPE=$(BUILD_TYPE) ^
-DENABLE_PROFILING_ITT=ON ^
-DSELECTIVE_BUILD=COLLECT ^
-DCMAKE_C_COMPILER:PATH="$(MSVC_COMPILER_PATH)" ^
-DCMAKE_CXX_COMPILER:PATH="$(MSVC_COMPILER_PATH)" ^
-S $(REPO_DIR) ^
-B $(BUILD_DIR)
displayName: 'CMake CC COLLECT'
- script: dir $(REPO_DIR)\temp\ /s
displayName: 'List temp SDKs'
- script: |
call "$(MSVS_VARS_PATH)" && cmake --build $(BUILD_DIR) --config $(BUILD_TYPE) --parallel --target ^
openvino_intel_cpu_plugin openvino_ir_frontend benchmark_app sea_itt_lib
displayName: 'Build CC COLLECT'
- script: dir $(REPO_DIR)\bin\ /s
displayName: 'List bin files'
- script: |
set path=%path%;$(REPO_DIR)\temp\tbb\bin
call "$(MSVS_VARS_PATH)" && python thirdparty\itt_collector\runtool\sea_runtool.py --bindir $(REPO_DIR)\bin\intel64\$(BUILD_TYPE) -o $(BUILD_DIR)\itt_stat ! $(REPO_DIR)\bin\intel64\$(BUILD_TYPE)\benchmark_app.exe -niter 1 -nireq 1 -m $(MODELS_PATH)\models\test_model\test_model_fp32.xml -d CPU
workingDirectory: $(REPO_DIR)
displayName: 'Code usage analysis'
- script: dir $(BUILD_DIR)\*.csv /s /p
displayName: 'List csv files'
- script: |
call "$(MSVS_VARS_PATH)" && cmake ^
-G "Visual Studio 16 2019" ^
-DCMAKE_VERBOSE_MAKEFILE=ON ^
-DENABLE_CPPLINT=OFF ^
-DENABLE_GAPI_PREPROCESSING=OFF ^
-DENABLE_PROFILING_ITT=OFF ^
-DSELECTIVE_BUILD=ON ^
-DCMAKE_COMPILE_WARNING_AS_ERROR=ON ^
-DSELECTIVE_BUILD_STAT=$(BUILD_DIR)\*.csv ^
-DCMAKE_C_COMPILER:PATH="$(MSVC_COMPILER_PATH)" ^
-DCMAKE_CXX_COMPILER:PATH="$(MSVC_COMPILER_PATH)" ^
-S $(REPO_DIR) ^
-B $(BUILD_DIR_2)
displayName: 'CMake CC ON'
- script: cmake --build $(BUILD_DIR_2) --config $(BUILD_TYPE) --parallel --target ^
openvino_intel_cpu_plugin openvino_ir_frontend benchmark_app
displayName: 'Build CC ON'
- script: dir $(REPO_DIR)\bin\ /s
displayName: 'List bin files ON'
- script: type $(BUILD_DIR_2)\src\common\conditional_compilation\conditional_compilation_gen.h
displayName: 'Check conditional_compilation_gen.h header'
- script: |
set path=%path%;$(REPO_DIR)\temp\tbb\bin
$(REPO_DIR)\bin\intel64\$(BUILD_TYPE)\benchmark_app.exe -niter 1 -nireq 1 -m $(MODELS_PATH)\models\test_model\test_model_fp32.xml -d CPU
workingDirectory: $(REPO_DIR)
displayName: 'Use OpenVINO after CC'

View File

@ -0,0 +1,76 @@
FROM ubuntu:23.04
LABEL version=2021.03.30.1
# Build configuration arguments
ARG BUILD_TYPE=Release
ARG http_proxy
ARG https_proxy
ENV http_proxy ${http_proxy}
ENV https_proxy ${https_proxy}
ENV CI=true
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED 1
# Install base dependencies
RUN apt-get update && apt-get install -y locales && apt-get clean autoclean && apt-get autoremove -y
# Set the locale to en_US.UTF-8
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN apt-get update && apt-get -y --no-install-recommends install \
# OpenVINO dependencies
build-essential \
ninja-build \
cmake \
curl \
git \
unzip \
libtbb-dev \
libpugixml-dev \
wget \
# Python dependencies
python3 \
python3-pip \
python3-dev \
pybind11-dev \
python3-virtualenv \
cython3 \
tox && \
apt-get clean autoclean && \
apt-get autoremove -y
# Build OpenVINO
COPY . /openvino/
WORKDIR /openvino/build
RUN cmake .. \
-G Ninja \
-DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
-DENABLE_INTEL_GNA=OFF \
-DENABLE_INTEL_GPU=OFF \
-DENABLE_HETERO=OFF \
-DENABLE_MULTI=OFF \
-DENABLE_AUTO_BATCH=OFF \
-DENABLE_GAPI_PREPROCESSING=OFF \
-DENABLE_CPPLINT=OFF \
-DENABLE_NCC_STYLE=OFF \
-DENABLE_PROFILING_ITT=OFF \
-DENABLE_SAMPLES=OFF \
-DENABLE_OV_PADDLE_FRONTEND=OFF \
-DENABLE_OV_PYTORCH_FRONTEND=ON \
-DENABLE_OV_TF_FRONTEND=OFF \
-DENABLE_OPENVINO_DEBUG=OFF \
-DCMAKE_INSTALL_PREFIX=/openvino/dist
RUN ninja install
# Run tests via tox
WORKDIR /openvino/src/bindings/python
ENV OpenVINO_DIR=/openvino/dist/runtime/cmake
ENV LD_LIBRARY_PATH=/openvino/dist/runtime/lib/intel64:/openvino/dist/runtime/3rdparty/tbb/lib
ENV PYTHONPATH=/openvino/bin/intel64/${BUILD_TYPE}/python:/openvino/tools/mo:${PYTHONPATH}
CMD tox

14
.ci/pot/Jenkinsfile vendored Normal file
View File

@ -0,0 +1,14 @@
#!groovy
properties([
parameters([
string(defaultValue: '',
description: 'Pipeline shared library version (branch/tag/commit). Determined automatically if empty',
name: 'library_version')
])
])
loadOpenVinoLibrary {
potEntrypoint(this)
}

View File

@ -1,3 +0,0 @@
*
!install_build_dependencies.sh
!scripts/install_dependencies/install_openvino_dependencies.sh

14
.gitattributes vendored
View File

@ -65,5 +65,15 @@
*.vsdx filter=lfs diff=lfs merge=lfs -text
*.bmp filter=lfs diff=lfs merge=lfs -text
*.svg filter=lfs diff=lfs merge=lfs -text
.github/scripts/workflow_rerun/tests/data/log_archive_with_error.zip filter=lfs diff=lfs merge=lfs -text
.github/scripts/workflow_rerun/tests/data/log_archive_wo_error.zip filter=lfs diff=lfs merge=lfs -text
#POT attributes
tools/pot/tests/data/test_cases_refs/* filter=lfs diff=lfs merge=lfs -text
tools/pot/tests/data/models/*/* filter=lfs diff=lfs merge=lfs -text
tools/pot/tests/data/reference_models/* filter=lfs diff=lfs merge=lfs -text
tools/pot/tests/data/video/* filter=lfs diff=lfs merge=lfs -text
tools/pot/tests/data/reference_fake_quantize_conf/* filter=lfs diff=lfs merge=lfs -text
/tools/pot/tests/** -pot_package
/tools/pot/tools/auxilary/** -pot_package
/tools/pot/tools/run_series_experiments.py -pot_package
/tools/pot/.pylintrc -pot_package
/tools/pot/README_dev.md -pot_package

19
.github/CODEOWNERS vendored
View File

@ -28,20 +28,13 @@
/src/bindings/python/ @openvinotoolkit/openvino-ie-python-api-maintainers
/src/bindings/c/ @openvinotoolkit/openvino-c-api-maintainers
/src/bindings/js/ @openvinotoolkit/openvino-js-api-maintainers
/src/common/*transformations/ @openvinotoolkit/openvino-ie-transformations-maintainers
/src/core/ @openvinotoolkit/openvino-ngraph-maintainers
# OpenVINO Transformations
/src/common/*transformations/ @openvinotoolkit/openvino-ie-transformations-maintainers
/src/core/src/pass/ @openvinotoolkit/openvino-ie-transformations-maintainers
/src/core/src/pattern/ @openvinotoolkit/openvino-ie-transformations-maintainers
/src/core/include/openvino/pass/ @openvinotoolkit/openvino-ie-transformations-maintainers
# OpenVINO Samples:
/samples/c/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-c-api-maintainers
/samples/cpp/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-maintainers
/samples/python/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-ie-python-api-maintainers
/samples/js/ @openvinotoolkit/openvino-samples-maintainers @openvinotoolkit/openvino-js-api-maintainers
/thirdparty/zlib/ @openvinotoolkit/openvino-samples-maintainers
/thirdparty/json/ @openvinotoolkit/openvino-samples-maintainers
/thirdparty/gflags/ @openvinotoolkit/openvino-samples-maintainers
@ -65,8 +58,8 @@
/src/tests/**/gpu/ @openvinotoolkit/openvino-ie-gpu-maintainers
/thirdparty/ocl/ @openvinotoolkit/openvino-ie-gpu-maintainers @openvinotoolkit/openvino-ie-gpu-developers
# OpenVINO NPU:
/src/plugins/intel_npu/ @openvinotoolkit/openvino-npu-maintainers @openvinotoolkit/openvino-npu-developers
# OpenVINO GNA:
/src/plugins/intel_gna/ @openvinotoolkit/openvino-ie-gna-maintainers
# OpenVINO Auto (MULTI) plugin:
/src/plugins/auto/ @openvinotoolkit/openvino-ie-auto-multi-maintainers
@ -101,7 +94,7 @@
/tests/layer_tests/tensorflow_tests @openvinotoolkit/openvino-tf-frontend-maintainers
/tests/layer_tests/jax_tests @openvinotoolkit/openvino-tf-frontend-maintainers
/tests/model_hub_tests @openvinotoolkit/openvino-tf-frontend-maintainers
/tests/model_hub_tests/pytorch @openvinotoolkit/openvino-pytorch-frontend-maintainers
/tests/model_hub_tests/torch_tests @openvinotoolkit/openvino-pytorch-frontend-maintainers
# Tools:
/tools/ @openvinotoolkit/openvino-tools-maintainers
@ -110,7 +103,8 @@
/tools/openvino_dev/ @openvinotoolkit/openvino-tools-maintainers @openvinotoolkit/openvino-ie-python-api-maintainers
/tools/mo/ @openvinotoolkit/openvino-mo-maintainers
/tools/ovc/ @openvinotoolkit/openvino-mo-maintainers
/thirdparty/open_model_zoo/ @openvinotoolkit/omz-maintainers
/tools/pot/ @openvinotoolkit/openvino-pot-maintainers
/thirdparty/open_model_zoo/ @openvinotoolkit/omz-maintainers @openvinotoolkit/openvino-pot-maintainers
# Documentation
/docs/ @openvinotoolkit/openvino-docs-maintainers
@ -124,6 +118,7 @@
/docs/snippets/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-maintainers
/docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino_contrib-arm_plugin-maintainers
/docs/OV_Runtime_UG/supported_plugins/CPU.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-cpu-maintainers
/docs/OV_Runtime_UG/supported_plugins/GNA.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-gna-maintainers
/docs/OV_Runtime_UG/supported_plugins/GPU*.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-gpu-maintainers
# Configuration management

View File

@ -41,7 +41,9 @@ body:
options:
- CPU
- GPU
- NPU
- GNA
- NCS2 (Intel Movidius)
- HDDL
- AUTO
- HETERO
- BATCH

View File

@ -72,7 +72,7 @@ body:
- type: textarea
id: build_script
attributes:
label: Build script or step-by-step to reproduce
label: Build scrip or step-by-step to reproduce
description: How can we reproduce your issue?
placeholder: Please provide detailed instructions on how to reproduce the issue
validations:

View File

@ -41,9 +41,10 @@ body:
Any materials related to the task, such as operator specifications,
discussions, guides.
value: |
- [Contribution guide - start here!](https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING.md)
- [Intel DevHub Discord channel](https://discord.gg/7pVRxUwdWG) - engage in discussions, ask questions and talk to OpenVINO developers
- [How to link your Pull Request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#manually-linking-a-pull-request-to-an-issue-using-the-pull-request-sidebar)
- [What is OpenVINO?](https://github.com/openvinotoolkit/openvino#what-is-openvino-toolkit)
- [Contribution guide](https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING.md)
- [Blog post on contributing to OpenVINO](https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING.md)
- [User documentation](https://docs.openvino.ai/)
validations:
required: true

View File

@ -1,4 +0,0 @@
lib/
dist/
node_modules/
coverage/

View File

@ -1 +0,0 @@
20.6.0

View File

@ -1,3 +0,0 @@
dist/
node_modules/
coverage/

View File

@ -1,16 +0,0 @@
{
"printWidth": 80,
"tabWidth": 2,
"useTabs": false,
"semi": true,
"singleQuote": true,
"quoteProps": "as-needed",
"jsxSingleQuote": false,
"trailingComma": "none",
"bracketSpacing": true,
"bracketSameLine": true,
"arrowParens": "avoid",
"proseWrap": "always",
"htmlWhitespaceSensitivity": "css",
"endOfLine": "lf"
}

View File

@ -1,286 +0,0 @@
/**
* Unit tests for the action's main functionality, src/cleanupImpl.js
*/
const core = require('@actions/core');
const path = require('path');
const os = require('os');
const fs = require('fs');
const cleanupImpl = require('../src/cleanupImpl');
// Mock the GitHub Actions core library
const getInputMock = jest.spyOn(core, 'getInput').mockImplementation();
const setFailedMock = jest.spyOn(core, 'setFailed').mockImplementation();
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-cleanup-'));
const cacheRemotePath = path.join(tempDir, 'cache_remote');
const cacheFiles = ['cache_1.cache', 'cache_2.cache', 'cache_3.cache'];
const minAccessTime = 7 * 24 * 60 * 60 * 1000; // 1 week
// Mock the action's main function
const runMock = jest.spyOn(cleanupImpl, 'cleanUp');
describe('cleanup', () => {
beforeEach(() => {
jest.clearAllMocks();
// Set up file system before each test
fs.mkdirSync(cacheRemotePath, { recursive: true });
const fileSizeInBytes = 1024 * 1024 * 1024; // 1 GB
const buffer = Buffer.alloc(fileSizeInBytes);
let id = 1;
for (const cache of cacheFiles) {
const cachePath = path.join(cacheRemotePath, cache);
fs.writeFileSync(cachePath, buffer);
fs.utimesSync(
cachePath,
new Date(Date.now() - minAccessTime * id++),
new Date()
);
}
});
// Clean up mock file system after each test
afterEach(() => {
fs.rmSync(tempDir, { recursive: true });
});
it('Cleanup old files using restore key', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 1;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
// cache2 and cache3 should be removed
for (const cache of cacheFiles.slice(1, 2)) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(false);
}
// check that file1 exists
expect(fs.existsSync(path.join(cacheRemotePath, cacheFiles[0]))).toBe(true);
});
it('Cleanup old files using key', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'cache-size':
return 1;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
// cache2 and cache3 should be removed
for (const cache of cacheFiles.slice(1, 2)) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(false);
}
// check that file1 exists
expect(fs.existsSync(path.join(cacheRemotePath, cacheFiles[0]))).toBe(true);
});
it('Skip cleanup old files by size', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 5;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
for (const cache of cacheFiles) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(true);
}
});
it('Skip cleanup old files by atime', async () => {
for (const cache of cacheFiles) {
const cachePath = path.join(cacheRemotePath, cache);
fs.utimesSync(
cachePath,
new Date(Date.now() - minAccessTime / 2),
new Date()
);
}
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 1;
case 'max-cache-size':
return 5;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
for (const cache of cacheFiles) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(true);
}
});
it('Cleanup recently used files by max cache limit', async () => {
for (const cache of cacheFiles) {
const cachePath = path.join(cacheRemotePath, cache);
fs.utimesSync(
cachePath,
new Date(Date.now() - minAccessTime / 2),
new Date()
);
}
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 1;
case 'max-cache-size':
return 2;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
// cache2 and cache3 should be removed
for (const cache of cacheFiles.slice(1, 2)) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(false);
}
// check that file1 exists
expect(fs.existsSync(path.join(cacheRemotePath, cacheFiles[0]))).toBe(true);
});
it('Test unexpected behaviour', async () => {
// Set folder permissions to read-only
fs.chmodSync(cacheRemotePath, '555');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 2;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
expect(setFailedMock).not.toHaveBeenCalled();
fs.chmodSync(cacheRemotePath, '777');
});
it('Cleanup absent directory', async () => {
const cachePathAbsent = path.join(tempDir, 'cache_remote_absent');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cachePathAbsent;
case 'key':
return 'cache';
case 'restore-keys':
return 'cache';
case 'cache-size':
return 2;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
expect(setFailedMock).not.toHaveBeenCalled();
});
it('Cleanup directory with subdirectory', async () => {
const cacheSubPath = path.join(cacheRemotePath, 'cache_subdir.cache');
fs.mkdirSync(cacheSubPath, { recursive: true });
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'restore-keys':
return 'cache';
case 'cache-size':
return 1;
default:
return '';
}
});
await cleanupImpl.cleanUp();
expect(runMock).toHaveReturned();
// cache2 and cache3 should be removed
for (const cache of cacheFiles.slice(1, 2)) {
expect(fs.existsSync(path.join(cacheRemotePath, cache))).toBe(false);
}
// check that file1 exists
expect(fs.existsSync(path.join(cacheRemotePath, cacheFiles[0]))).toBe(true);
// check that sub directory exists
expect(fs.existsSync(cacheSubPath)).toBe(true);
});
});

View File

@ -1,266 +0,0 @@
/**
* Unit tests for the action's main functionality, src/restoreImpl.js
*/
const core = require('@actions/core');
const path = require('path');
const os = require('os');
const fs = require('fs');
const tar = require('tar');
const restoreImpl = require('../src/restoreImpl');
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-restore-'));
const cacheLocalPath = path.join(tempDir, 'cache_local');
const cacheRemotePath = path.join(tempDir, 'cache_remote');
const cacheTmpPath = path.join(tempDir, 'cache_tmp');
const cacheFiles = ['cache_1.cache', 'cache_2.cache', 'cache_3.cache'];
const testFiles = ['file1.txt', 'file2.txt', 'file3.txt'];
// Mock the GitHub Actions core library
const getInputMock = jest.spyOn(core, 'getInput').mockImplementation();
const setOutputMock = jest.spyOn(core, 'setOutput').mockImplementation();
const setFailedMock = jest.spyOn(core, 'setFailed').mockImplementation();
// Clean up mock file system after each test
afterEach(() => {
fs.rmSync(tempDir, { recursive: true });
});
// Mock the action's main function
const runMock = jest.spyOn(restoreImpl, 'restore');
describe('restore', () => {
beforeEach(() => {
jest.clearAllMocks();
// Set up mock file system before each test
fs.mkdirSync(cacheLocalPath, { recursive: true });
fs.mkdirSync(cacheRemotePath, { recursive: true });
fs.mkdirSync(cacheTmpPath, { recursive: true });
// Define test files
const file1Path = path.join(cacheTmpPath, testFiles[0]);
fs.writeFileSync(file1Path, 'File 1 contents');
tar.c(
{
gzip: true,
file: cacheFiles[0],
cwd: cacheTmpPath,
sync: true
},
['.']
);
fs.renameSync(cacheFiles[0], path.join(cacheRemotePath, cacheFiles[0]));
fs.utimesSync(
path.join(cacheRemotePath, cacheFiles[0]),
new Date(Date.now() - 1500),
new Date(Date.now() - 1500)
);
//
const file2Path = path.join(cacheTmpPath, testFiles[1]);
fs.writeFileSync(file2Path, 'File 2 contents');
tar.c(
{
gzip: true,
file: cacheFiles[1],
cwd: cacheTmpPath,
sync: true
},
['.']
);
fs.renameSync(cacheFiles[1], path.join(cacheRemotePath, cacheFiles[1]));
fs.utimesSync(
path.join(cacheRemotePath, cacheFiles[1]),
new Date(Date.now() - 1000),
new Date(Date.now() - 1000)
);
//
const file3Path = path.join(cacheTmpPath, testFiles[2]);
fs.writeFileSync(file3Path, 'File 3 contents');
tar.c(
{
gzip: true,
file: cacheFiles[2],
cwd: cacheTmpPath,
sync: true
},
['.']
);
fs.renameSync(cacheFiles[2], path.join(cacheRemotePath, cacheFiles[2]));
fs.utimesSync(
path.join(cacheRemotePath, cacheFiles[2]),
new Date(Date.now() - 500),
new Date(Date.now() - 500)
);
});
it('gets the cache file', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await restoreImpl.restore();
expect(runMock).toHaveReturned();
// Verify that all of the core library functions were called correctly
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
cacheFiles[2]
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
let id = 1;
for (const filename of testFiles) {
const filePath = path.join(cacheLocalPath, filename);
expect(fs.existsSync(filePath)).toBe(true);
const fileContent = fs.readFileSync(filePath, 'utf8');
expect(fileContent).toBe(`File ${id++} contents`);
}
});
it('gets the updated cache file', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
// updated cache 2 access time
fs.utimesSync(
path.join(cacheRemotePath, cacheFiles[1]),
new Date(Date.now() - 100),
new Date(Date.now() - 100)
);
await restoreImpl.restore();
expect(runMock).toHaveReturned();
// Verify that all of the core library functions were called correctly
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
cacheFiles[1]
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
let id = 1;
for (const filename of testFiles.slice(0, 1)) {
const filePath = path.join(cacheLocalPath, filename);
expect(fs.existsSync(filePath)).toBe(true);
const fileContent = fs.readFileSync(filePath, 'utf8');
expect(fileContent).toBe(`File ${id++} contents`);
}
// check that file3 is absent
expect(fs.existsSync(path.join(cacheLocalPath, testFiles[2]))).toBe(false);
});
it('gets the cache file and extract to absent dir', async () => {
const cacheAbsentPath = path.join(tempDir, 'cache_absent');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'restore-keys':
return 'cache';
case 'path':
return cacheAbsentPath;
default:
return '';
}
});
await restoreImpl.restore();
expect(runMock).toHaveReturned();
// Verify that all of the core library functions were called correctly
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
cacheFiles[2]
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
let id = 1;
for (const filename of testFiles) {
const filePath = path.join(cacheAbsentPath, filename);
expect(fs.existsSync(filePath)).toBe(true);
const fileContent = fs.readFileSync(filePath, 'utf8');
expect(fileContent).toBe(`File ${id++} contents`);
}
});
it('Test when no cache found', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'key';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await restoreImpl.restore();
expect(runMock).toHaveReturned();
// Verify that all of the core library functions were called correctly
expect(setOutputMock).toHaveBeenNthCalledWith(1, 'cache-file', '');
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', false);
for (const filename of testFiles) {
const filePath = path.join(cacheLocalPath, filename);
expect(fs.existsSync(filePath)).toBe(false);
}
expect(setFailedMock).not.toHaveBeenCalled();
});
it('Unsupported cache found', async () => {
const cacheFakePath = path.join(cacheRemotePath, 'cache-fake.cache');
fs.writeFileSync(cacheFakePath, 'Fake content');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await restoreImpl.restore();
expect(runMock).toHaveReturned();
expect(setOutputMock).not.toHaveBeenCalled();
expect(setFailedMock).not.toHaveBeenCalled();
});
});

View File

@ -1,190 +0,0 @@
/**
* Unit tests for the action's main functionality, src/saveImpl.js
*/
const core = require('@actions/core');
const path = require('path');
const os = require('os');
const fs = require('fs');
const saveImpl = require('../src/saveImpl');
// Mock the GitHub Actions core library
const getInputMock = jest.spyOn(core, 'getInput').mockImplementation();
const setOutputMock = jest.spyOn(core, 'setOutput').mockImplementation();
const setFailedMock = jest.spyOn(core, 'setFailed').mockImplementation();
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-'));
const cacheLocalPath = path.join(tempDir, 'cache_local');
const cacheRemotePath = path.join(tempDir, 'cache_remote');
// Clean up mock file system after each test
afterEach(() => {
fs.rmSync(tempDir, { recursive: true });
});
// Mock the action's main function
const runMock = jest.spyOn(saveImpl, 'save');
describe('save', () => {
beforeEach(() => {
jest.clearAllMocks();
// Set up mock file system before each test
// Create cache pathes
fs.mkdirSync(cacheLocalPath, { recursive: true });
fs.mkdirSync(cacheRemotePath, { recursive: true });
// Define test files
const file1Path = path.join(cacheLocalPath, 'file1.txt');
const file2Path = path.join(cacheLocalPath, 'file2.txt');
// Write content to the test files
fs.writeFileSync(file1Path, 'File 1 contents');
fs.writeFileSync(file2Path, 'File 2 contents');
});
it('Cache files pass', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
'cache.cache'
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
expect(fs.existsSync(`${cacheRemotePath}/cache.cache`)).toBe(true); // Check if the tar file was created
});
it('Cache files: absent local cache dir', async () => {
// Set the action's inputs as return values from core.getInput()
const cachePathAbsent = path.join(tempDir, 'cache_local_absent');
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'path':
return cachePathAbsent;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(fs.existsSync(`${cacheRemotePath}/cache.cache`)).toBe(false); // Check that the tar file was not created
});
it('Cache files: absent remote cache dir', async () => {
// Set the action's inputs as return values from core.getInput()
const cacheRemotePathAbsent = path.join(tempDir, 'cache_remote_absent');
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePathAbsent;
case 'key':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).toHaveBeenNthCalledWith(
1,
'cache-file',
'cache.cache'
);
expect(setOutputMock).toHaveBeenNthCalledWith(2, 'cache-hit', true);
expect(fs.existsSync(`${cacheRemotePathAbsent}/cache.cache`)).toBe(true); // Check if the tar file was created
});
it('Cache files: key is not set', async () => {
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).not.toHaveBeenCalled();
expect(setFailedMock).not.toHaveBeenCalled();
});
it('Cache files: file already exists', async () => {
const cacheFakePath = path.join(cacheRemotePath, 'cache-fake.cache');
fs.writeFileSync(cacheFakePath, 'Fake content');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache-fake';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).not.toHaveBeenCalled();
expect(setFailedMock).not.toHaveBeenCalled();
});
it('Test unexpected behaviour', async () => {
// Set folder permissions to read-only
fs.chmodSync(cacheRemotePath, '555');
// Set the action's inputs as return values from core.getInput()
getInputMock.mockImplementation(name => {
switch (name) {
case 'cache-path':
return cacheRemotePath;
case 'key':
return 'cache';
case 'path':
return cacheLocalPath;
default:
return '';
}
});
await saveImpl.save();
expect(runMock).toHaveReturned();
expect(setOutputMock).not.toHaveBeenCalled();
expect(setFailedMock).toHaveBeenCalled();
});
});

View File

@ -1,60 +0,0 @@
name: 'Cache'
description: 'Cache to the local file system'
author: 'Mikhail Ryzhov'
inputs:
cache-path:
description: 'A local path where to search and save caches'
required: true
path:
description:
'A list of files, directories, and wildcard patterns to cache and restore'
required: true
key:
description: 'An explicit key for restoring and saving the cache'
required: true
restore-keys:
description:
'An ordered list of keys to use for restoring stale cache if no cache hit
occurred for key.'
default: ''
required: false
save-always:
description:
'Run the post step to save the cache even if another step before fails'
default: false
required: false
cleanup-always:
description:
'Run the post cleanup step to remove old cache files even if another step
before fails'
default: false
required: false
cache-size:
description:
'Allowed cache storage size in Gb. Least recently used caches will be
automatically evicted to limit the total cache storage. The cache size can
exceed this value, but it is limited by "max-cache-size" value'
default: 10
required: false
max-cache-size:
description:
'Maximum cache storage size in Gb. All cache files will be automatically
evicted to limit the total cache storage'
default: 50
required: false
outputs:
cache-hit:
description:
'A boolean value to indicate an exact match was found for the primary key'
cache-file:
description: 'Found cache file'
runs:
using: node20
main: 'dist/restore/index.js'
post: 'dist/save/index.js'
post-if:
'success() || github.event.inputs.save-always ||
github.event.inputs.cleanup-always'

View File

@ -1,37 +0,0 @@
name: 'Cleanup Cache'
description: 'Remove old cache artifacts'
author: 'OpenVINO Developers'
inputs:
cache-path:
description: 'A local path where to search and save caches'
required: true
key:
description: 'An explicit key for restoring and saving the cache'
required: true
restore-keys:
description:
'An ordered list of keys to use for restoring stale cache if no cache hit
occurred for key.'
default: ''
required: false
cache-size:
description:
'Allowed cache storage size in Gb. Least recently used caches will be
automatically evicted to limit the total cache storage. The cache size can
exceed this value, but it is limited by "max-cache-size" value'
default: 10
required: false
max-cache-size:
description:
'Maximum cache storage size in Gb. All cache files will be automatically
evicted to limit the total cache storage'
default: 50
required: false
runs:
using: 'node20'
main: '../dist/cleanup-only/index.js'
branding:
icon: 'archive'
color: 'gray-dark'

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@ -1,50 +0,0 @@
env:
commonjs: true
es6: true
jest: true
node: true
globals:
Atomics: readonly
SharedArrayBuffer: readonly
ignorePatterns:
- '!.*'
- '**/node_modules/.*'
- '**/dist/.*'
- '**/coverage/.*'
- '*.json'
parser: '@babel/eslint-parser'
parserOptions:
ecmaVersion: 2023
sourceType: module
requireConfigFile: false
babelOptions:
babelrc: false
configFile: false
presets:
- jest
plugins:
- jest
extends:
- eslint:recommended
- plugin:github/recommended
- plugin:jest/recommended
rules:
{
'camelcase': 'warn',
'eslint-comments/no-use': 'off',
'eslint-comments/no-unused-disable': 'off',
'i18n-text/no-en': 'off',
'import/no-commonjs': 'off',
'import/no-namespace': 'off',
'no-console': 'off',
'no-unused-vars': 'warn',
'prettier/prettier': 'error',
'semi': 'error'
}

View File

@ -1,18 +0,0 @@
# Unordered list style
MD004:
style: dash
# Ordered list item prefix
MD029:
style: one
# Spaces after list markers
MD030:
ul_single: 1
ol_single: 1
ul_multi: 1
ol_multi: 1
# Code block style
MD046:
style: fenced

View File

@ -1,10 +0,0 @@
rules:
document-end: disable
document-start:
level: warning
present: false
line-length:
level: warning
max: 80
allow-non-breakable-words: true
allow-non-breakable-inline-mappings: true

10054
.github/actions/cache/package-lock.json generated vendored

File diff suppressed because it is too large Load Diff

View File

@ -1,73 +0,0 @@
{
"name": "compressed-cache-action",
"description": "action to save/restore cache as tarballs",
"version": "0.0.1",
"author": "OpenVINO Developers",
"private": true,
"keywords": [
"GitHub",
"Actions",
"JavaScript"
],
"engines": {
"node": ">=20"
},
"main": "dist/restore/index.js",
"scripts": {
"bundle": "npm run format:write && npm run package",
"ci-test": "npx jest",
"coverage": "npx jest -- --coverage",
"format:write": "npx prettier --write .",
"format:check": "npx prettier --check .",
"lint": "npx eslint . -c ./linters/.eslintrc.yml ",
"package": "npx ncc build src/restore.js -o dist/restore && npx ncc build src/save.js -o dist/save && npx ncc build src/restore.js -o dist/restore-only && npx ncc build src/save.js -o dist/save-only && npx ncc build src/cleanup.js -o dist/cleanup-only",
"package:watch": "npm run package -- --watch",
"test": "npx jest",
"all": "npm run format:write && npm run lint && npm run test && npm run coverage && npm run package"
},
"eslintConfig": {
"extends": "./linters/.eslintrc.yml"
},
"jest": {
"verbose": true,
"clearMocks": true,
"testEnvironment": "node",
"moduleFileExtensions": [
"js"
],
"testMatch": [
"**/*.test.js"
],
"testPathIgnorePatterns": [
"/node_modules/",
"/dist/"
],
"coverageReporters": [
"json-summary",
"text",
"lcov"
],
"collectCoverage": true,
"collectCoverageFrom": [
"./src/**"
]
},
"dependencies": {
"@actions/core": "^1.10.1",
"fs": "^0.0.1-security",
"tar": "^6.2.0"
},
"devDependencies": {
"@babel/core": "^7.24.0",
"@babel/eslint-parser": "^7.23.10",
"@babel/preset-env": "^7.24.0",
"@vercel/ncc": "^0.38.1",
"babel-preset-jest": "^29.6.3",
"eslint": "^8.57.0",
"eslint-plugin-github": "^4.10.2",
"eslint-plugin-jest": "^27.9.0",
"fs": "0.0.1-security",
"jest": "^29.7.0",
"prettier": "^3.2.5"
}
}

View File

@ -1,35 +0,0 @@
name: 'Restore Cache'
description: 'Restore Cache artifacts to improve workflow execution time'
author: 'OpenVINO Developers'
inputs:
cache-path:
description: 'A local path where to search and save caches'
required: true
path:
description:
'A list of files, directories, and wildcard patterns to cache and restore'
required: true
key:
description: 'An explicit key for restoring and saving the cache'
required: true
restore-keys:
description:
'An ordered list of keys to use for restoring stale cache if no cache hit
occurred for key.'
default: ''
required: false
outputs:
cache-hit:
description:
'A boolean value to indicate an exact match was found for the primary key'
cache-file:
description: 'Found cache file'
runs:
using: 'node20'
main: '../dist/restore-only/index.js'
branding:
icon: 'archive'
color: 'gray-dark'

View File

@ -1,22 +0,0 @@
name: 'Save a cache'
description: 'Cache to the local file system'
author: 'OpenVINO Developers'
inputs:
cache-path:
description: 'A local path where to search and save caches'
required: true
path:
description:
'A list of files, directories, and wildcard patterns to cache and restore'
required: true
key:
description: 'An explicit key for restoring and saving the cache'
required: true
runs:
using: 'node20'
main: '../dist/save-only/index.js'
branding:
icon: 'archive'
color: 'gray-dark'

View File

@ -1,3 +0,0 @@
const { cleanUp } = require('./cleanupImpl');
cleanUp();

View File

@ -1,85 +0,0 @@
const core = require('@actions/core');
const fs = require('fs/promises');
const path = require('path');
const {
getSortedCacheFiles,
humanReadableFileSize,
calculateTotalSize
} = require('./utils');
// Function to remove old files if their combined size exceeds the allowed size
async function cleanUp() {
try {
const cacheRemotePath = core.getInput('cache-path', { required: true });
const key = core.getInput('key', { required: true });
const keysRestore = core
.getInput('restore-keys', { required: false })
.split('\n')
.map(s => s.replace(/^!\s+/, '!').trim())
.filter(x => x !== '');
const cacheSize = core.getInput('cache-size', { required: false });
const cacheMaxSize = core.getInput('max-cache-size', { required: false });
// Minimum time peroid in milliseconds when the files was not useds
const minAccessTime = 7 * 24 * 60 * 60 * 1000; // 1 week
const currentDate = new Date();
const minAccessDateAgo = new Date(currentDate - minAccessTime);
core.debug(`cache-path: ${cacheRemotePath}`);
core.debug(`key: ${key}`);
core.debug(`restore-keys: ${keysRestore}`);
core.debug(`cache-size: ${cacheSize}`);
core.debug(`max-cache-size: ${cacheMaxSize}`);
let keyPattern = key;
if (keysRestore && keysRestore.length) {
keyPattern = keysRestore.join('|');
}
const files = await getSortedCacheFiles(cacheRemotePath, keyPattern);
const minCacheSizeInBytes = cacheSize * 1024 * 1024 * 1024;
const maxCacheSizeInBytes = cacheMaxSize * 1024 * 1024 * 1024;
let totalSize = await calculateTotalSize(cacheRemotePath, files);
if (totalSize <= minCacheSizeInBytes) {
core.info(
`The cache storage size ${humanReadableFileSize(totalSize)} less then the allowed size ${humanReadableFileSize(minCacheSizeInBytes)}`
);
return;
}
core.info(
`The cache storage size ${humanReadableFileSize(totalSize)} exceeds allowed size ${humanReadableFileSize(minCacheSizeInBytes)}`
);
for (const file of files.reverse()) {
const filePath = path.join(cacheRemotePath, file);
const fileStats = await fs.stat(filePath);
// skipping recently used files if total cache size less then maxCacheSizeInBytes
if (
!fileStats.isFile() ||
(fileStats.atime >= minAccessDateAgo && totalSize < maxCacheSizeInBytes)
) {
core.info(`Skipping: ${filePath}`);
continue;
}
core.info(`Removing file: ${filePath}`);
await fs.unlink(filePath);
core.info(`${filePath} removed successfully`);
totalSize -= fileStats.size;
// Exit if total size is within limit
if (totalSize <= minCacheSizeInBytes) {
core.info('Old cache files removed successfully');
return;
}
}
} catch (error) {
core.error(`Error removing old cache files: ${error.message}`);
}
}
module.exports = {
cleanUp
};

View File

@ -1,6 +0,0 @@
/**
* The entrypoint for the action.
*/
const { restore } = require('./restoreImpl');
restore();

View File

@ -1,84 +0,0 @@
const core = require('@actions/core');
const fs = require('fs/promises');
const path = require('path');
const tar = require('tar');
const os = require('os');
const {
getSortedCacheFiles,
humanReadableFileSize,
checkFileExists
} = require('./utils');
/**
* The main function for the action.
* @returns {Promise<void>} Resolves when the action is complete.
*/
async function restore() {
try {
const cacheRemotePath = core.getInput('cache-path', { required: true });
const cacheLocalPath = core.getInput('path', { required: true });
const key = core.getInput('key', { required: true });
const keysRestore = core
.getInput('restore-keys', { required: false })
.split('\n')
.map(s => s.replace(/^!\s+/, '!').trim())
.filter(x => x !== '');
core.debug(`cache-path: ${cacheRemotePath}`);
core.debug(`path: ${cacheLocalPath}`);
core.debug(`key: ${key}`);
core.debug(`restore-keys: ${keysRestore}`);
let keyPattern = key;
if (keysRestore && keysRestore.length) {
keyPattern = keysRestore.join('|');
}
core.info(`Looking for ${keyPattern} in ${cacheRemotePath}`);
const files = await getSortedCacheFiles(cacheRemotePath, keyPattern);
if (files.length) {
const cacheFile = files[0];
const cachePath = path.join(cacheRemotePath, cacheFile);
const cacheStat = await fs.stat(cachePath);
core.info(
`Found cache file: ${cacheFile}, size: ${humanReadableFileSize(cacheStat.size)}`
);
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'cache-'));
// copy file to temp dir
await fs.copyFile(cachePath, path.join(tempDir, cacheFile));
core.info(`${cacheFile} was copied to ${tempDir}/${cacheFile}`);
// extract
if (!(await checkFileExists(cacheLocalPath))) {
await fs.mkdir(cacheLocalPath);
}
core.info(`Extracting ${cacheFile} to ${cacheLocalPath}`);
tar.x({
file: path.join(tempDir, cacheFile),
cwd: cacheLocalPath,
sync: true
});
core.info(`Cache extracted to ${cacheLocalPath}`);
core.setOutput('cache-file', cacheFile);
core.setOutput('cache-hit', true);
} else {
core.warning(
`Could not found any suitable cache files in ${cacheRemotePath} with key ${key}`
);
core.setOutput('cache-file', '');
core.setOutput('cache-hit', false);
}
} catch (error) {
// do not fail action if cache could not be restored
core.error(error.message);
}
}
module.exports = {
restore
};

View File

@ -1,14 +0,0 @@
const core = require('@actions/core');
const { save } = require('./saveImpl');
const { cleanUp } = require('./cleanupImpl');
const saveAlways = core.getInput('save-always', { required: false });
const cleanUpAlways = core.getInput('cleanup-always', { required: false });
if (saveAlways === 'true') {
save();
}
if (cleanUpAlways === 'true') {
cleanUp();
}

View File

@ -1,71 +0,0 @@
const core = require('@actions/core');
const tar = require('tar');
const fs = require('fs/promises');
const path = require('path');
const { humanReadableFileSize, checkFileExists } = require('./utils');
/**
* The main function for the action.
* @returns {Promise<void>} Resolves when the action is complete.
*/
async function save() {
try {
const cacheRemotePath = core.getInput('cache-path', { required: true });
const toCachePath = core.getInput('path', { required: true });
const key = core.getInput('key', { required: true });
core.debug(`cache-path: ${cacheRemotePath}`);
core.debug(`path: ${toCachePath}`);
core.debug(`key: ${key}`);
if (!key) {
core.warning(`Key ${key} is not specified.`);
return;
}
const tarName = `${key}.cache`;
const tarPath = path.join(cacheRemotePath, tarName);
const tarNameTmp = `${key}.tmp`;
const tarPathTmp = path.join(cacheRemotePath, tarNameTmp);
if (await checkFileExists(tarPath)) {
core.warning(`Cache file ${tarName} already exists`);
return;
}
core.info(`Preparing cache archive ${tarName}`);
tar.c(
{
gzip: true,
file: tarName,
cwd: toCachePath,
sync: true
},
['.']
);
const tarStat = await fs.stat(tarName);
core.info(
`Created cache tarball: ${tarName}, size: ${humanReadableFileSize(tarStat.size)}`
);
// remote cache directory may not be created yet
if (!(await checkFileExists(cacheRemotePath))) {
await fs.mkdir(cacheRemotePath, { recursive: true });
}
core.info('Copying cache...');
await fs.copyFile(tarName, tarPathTmp);
// After copying is done, rename file
await fs.rename(tarPathTmp, tarPath);
core.info(`${tarName} copied to ${tarPath}`);
core.setOutput('cache-file', tarName);
core.setOutput('cache-hit', true);
} catch (error) {
core.setFailed(error.message);
}
}
module.exports = {
save
};

View File

@ -1,72 +0,0 @@
const core = require('@actions/core');
const fs = require('fs');
const path = require('path');
async function getSortedCacheFiles(cachePath, key = '') {
if (!(await checkFileExists(cachePath))) {
core.warning(`${cachePath} doesn't exist`);
return [];
}
const cachePattern = new RegExp(`^((${key}).*[.]cache)$`);
const filesSorted = (await fs.promises.readdir(cachePath))
.filter(fileName => cachePattern.test(fileName))
.map(fileName => ({
name: fileName,
time: fs.statSync(path.join(cachePath, fileName)).atimeMs
}))
.sort((a, b) => b.time - a.time)
.map(file => file.name);
core.debug(
filesSorted.map(fileName => ({
name: fileName,
atime: fs.statSync(path.join(cachePath, fileName)).atimeMs
}))
);
return filesSorted;
}
function humanReadableFileSize(sizeInBytes) {
const units = ['B', 'KB', 'MB', 'GB', 'TB'];
let id = 0;
while (sizeInBytes >= 1024 && id < units.length - 1) {
sizeInBytes /= 1024;
id++;
}
return `${sizeInBytes.toFixed(2)} ${units[id]}`;
}
// Function to calculate the total size of files in bytes
async function calculateTotalSize(dir, files) {
let totalSize = 0;
for (const file of files) {
const filePath = path.join(dir, file);
const fileStats = await fs.promises.stat(filePath);
if (fileStats.isFile()) {
totalSize += fileStats.size;
}
}
return totalSize;
}
async function checkFileExists(filePath) {
try {
await fs.promises.access(filePath);
return true;
} catch (error) {
return false;
}
}
module.exports = {
getSortedCacheFiles,
humanReadableFileSize,
calculateTotalSize,
checkFileExists
};

View File

@ -1,72 +0,0 @@
name: 'Handle Docker images'
description: 'Builds, tags and pushes a given Docker image when needed'
inputs:
images:
description: 'Image names (registry name + namespace + base name)'
required: true
registry:
description: 'Docker registry'
required: true
dockerfiles_root_dir:
description: 'Path to dockerfiles root dir relative to repository root'
required: true
push:
description: 'Push built images to registry'
required: false
default: 'true'
changed_components:
description: 'Components changed by a pull request'
required: true
outputs:
images:
description: "Images to use in workflow"
value: ${{ steps.handle_images.outputs.images }}
runs:
using: 'composite'
steps:
- name: Checkout head
uses: actions/checkout@v4
- name: Checkout base
uses: actions/checkout@v4
with:
ref: ${{ github.base_ref || github.event.merge_group.base_ref }}
sparse-checkout: ${{ inputs.dockerfiles_root_dir }}/docker_tag
path: base
- name: Install Python dependencies
uses: py-actions/py-dependency-install@v4
with:
path: "${{ github.action_path }}/requirements.txt"
update-setuptools: "false"
update-wheel: "false"
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v3
- name: Handle docker images
id: handle_images
shell: bash
run: |
images=$(echo "${{ inputs.images }}" | tr '\n' ',' | sed 's/,*$//')
pr="${{ github.event.pull_request.number }}"
python3 .github/actions/handle_docker/get_images_to_build.py \
-d "${{ inputs.dockerfiles_root_dir }}" \
-r "${{ inputs.registry }}" \
--images "$images" \
--head_tag_file "${{ inputs.dockerfiles_root_dir }}/docker_tag" \
--base_tag_file "base/${{ inputs.dockerfiles_root_dir }}/docker_tag" \
--docker_env_changed "${{ fromJSON(inputs.changed_components).docker_env }}" \
--dockerfiles_changed "${{ fromJSON(inputs.changed_components).dockerfiles }}" \
--docker_builder "${{ steps.buildx.outputs.name}}" \
--repo "${{ github.repository }}" \
--ref_name "${{ github.ref_name }}" \
$([[ -n $pr ]] && echo "--pr $pr" || echo '-s ${{ github.sha }}') \
$([[ -n "${{ inputs.push }}" ]] && echo "--push" || echo '')
env:
GITHUB_TOKEN: ${{ github.token }}

View File

@ -1,129 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import argparse
import json
import re
import sys
from distutils.util import strtobool
from helpers import *
from images_api import *
def parse_args():
parser = argparse.ArgumentParser(description='Returns list of Docker images to build for a given workflow')
parser.add_argument('-i', '--images', required=True, help='Comma-separated docker images')
parser.add_argument('-d', '--dockerfiles_root', required=True, help='Path to dockerfiles')
parser.add_argument('-r', '--registry', required=True, help='Docker registry name')
parser.add_argument('-s', '--commit', required=False, help='Commit SHA. If not set, --pr is used')
parser.add_argument('-b', '--docker_builder', required=False, help='Docker buildx builder name')
parser.add_argument('--pr', type=int, required=False, help='PR number, if event is pull_request')
parser.add_argument('--head_tag_file', default='.github/dockerfiles/docker_tag', help='Head docker tag file path')
parser.add_argument('--base_tag_file', default=None, required=False, help='Base docker tag file path')
parser.add_argument('--ref_name', required=False, default='', help='GitHub ref name')
parser.add_argument('--repo', default='openvinotoolkit/openvino', help='GitHub repository')
parser.add_argument('--docker_env_changed', type=lambda x: bool(strtobool(x)), default=True,
help='Whether PR changes docker env')
parser.add_argument('--dockerfiles_changed', type=lambda x: bool(strtobool(x)), default=True,
help='Whether PR changes dockerfiles')
parser.add_argument('--action_path', default='.github/actions/handle_docker', help='Path to this GitHub action')
parser.add_argument('--push', action='store_true', required=False, help='Whether to push images to registry')
parser.add_argument('--dry_run', action='store_true', required=False, help='Dry run')
args = parser.parse_args()
return args
def main():
init_logger()
logger = logging.getLogger(__name__)
args = parse_args()
for arg, value in sorted(vars(args).items()):
logger.info(f"Argument {arg}: {value}")
head_tag = Path(args.head_tag_file).read_text().strip()
base_tag_exists = args.base_tag_file and Path(args.base_tag_file).exists()
base_tag = Path(args.base_tag_file).read_text().strip() if base_tag_exists else None
all_dockerfiles = Path(args.dockerfiles_root).rglob('**/*/Dockerfile')
images = ImagesHandler(args.dry_run)
for image in all_dockerfiles:
images.add_from_dockerfile(image, args.dockerfiles_root, args.registry, head_tag, base_tag)
requested_images = set(args.images.split(','))
skip_workflow = False
missing_only = False
merge_queue_target_branch = next(iter(re.findall(f'^gh-readonly-queue/(.*)/', args.ref_name)), None)
if args.pr:
environment_affected = args.docker_env_changed or args.dockerfiles_changed
if environment_affected:
expected_tag = f'pr-{args.pr}'
if head_tag != expected_tag:
logger.error(f"Please update docker tag in {args.head_tag_file} to {expected_tag}")
sys.exit(1)
elif merge_queue_target_branch:
environment_affected = head_tag != base_tag
if environment_affected:
logger.info(f"Environment is affected by PR(s) in merge group")
else:
environment_affected = False
if environment_affected:
changeset = get_changeset(args.repo, args.pr, merge_queue_target_branch, args.commit)
changed_dockerfiles = [p for p in changeset if p.startswith(args.dockerfiles_root) and p.endswith('Dockerfile')]
if args.docker_env_changed:
logger.info(f"Common docker environment is modified, will build all requested images")
changed_images = requested_images
else:
logger.info(f"Common docker environment is not modified, will build only changed and missing images")
changed_images = set([name_from_dockerfile(d, args.dockerfiles_root) for d in changed_dockerfiles])
unchanged_images = requested_images - changed_images
unchanged_with_no_base = images.get_missing(unchanged_images, base=True)
if unchanged_with_no_base:
logger.info("The following images were unchanged, but will be built anyway since the base for them "
f"is missing in registry: {unchanged_with_no_base}")
images_to_tag = unchanged_images.difference(unchanged_with_no_base)
images_to_build = requested_images.intersection(changed_images).union(unchanged_with_no_base)
only_dockerfiles_changed = len(changeset) == len(changed_dockerfiles)
if only_dockerfiles_changed and not images_to_build:
skip_workflow = True
else:
logger.info(f"Environment is not affected, will build only missing images, if any")
images_to_build = requested_images
images_to_tag = []
missing_only = True
if not images_to_build:
logger.info(f"No images to build, will return the list of pre-built images with a new tag")
built_images = images.build(images_to_build, missing_only, args.push, args.docker_builder)
if not built_images:
logger.info(f"No images were built, a new tag will be applied to a pre-built base image if needed")
# When a custom builder is used, it allows to push the image automatically once built. Otherwise, pushing manually
if args.push and not args.docker_builder:
images.push(images_to_build, missing_only)
if environment_affected and base_tag:
images.tag(images_to_tag)
images_output = images_to_output(images.get(requested_images))
set_github_output("images", json.dumps(images_output))
if skip_workflow:
logger.info(f"Docker image changes are irrelevant for current workflow, workflow may be skipped")
set_github_output("skip_workflow", str(skip_workflow))
main()

View File

@ -1,75 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import logging
import os
import subprocess
from ghapi.all import GhApi
from pathlib import Path
def init_logger():
logging.basicConfig(level=logging.INFO,
format='%(asctime)s %(name)-15s %(levelname)-8s %(message)s',
datefmt='%m-%d-%Y %H:%M:%S')
def set_github_output(name: str, value: str, github_output_var_name: str = 'GITHUB_OUTPUT'):
"""Sets output variable for a GitHub Action"""
logger = logging.getLogger(__name__)
# In an environment variable "GITHUB_OUTPUT" GHA stores path to a file to write outputs to
with open(os.environ.get(github_output_var_name), 'a+') as file:
logger.info(f"Add {name}={value} to {github_output_var_name}")
print(f'{name}={value}', file=file)
def images_to_output(images: list):
images_output = {}
for image in images:
image_name, os_name = image.name.split('/', 1)
if image_name not in images_output:
images_output[image_name] = {}
images_output[image_name][os_name] = image.ref()
return images_output
def get_changeset(repo: str, pr: str, target_branch: str, commit_sha: str):
"""Returns changeset either from PR or commit"""
owner, repository = repo.split('/')
gh_api = GhApi(owner=owner, repo=repository, token=os.getenv("GITHUB_TOKEN"))
if pr:
changed_files = gh_api.pulls.list_files(pr)
elif target_branch:
target_branch_head_commit = gh_api.repos.get_branch(target_branch).commit.sha
changed_files = gh_api.repos.compare_commits(f'{target_branch_head_commit}...{commit_sha}').get('files', [])
else:
raise ValueError(f'Either "pr" or "target_branch" parameter must be non-empty')
return set([f.filename for f in changed_files])
def run(cmd: str, dry_run: bool = False, fail_on_error: bool = True):
logger = logging.getLogger('run')
logger.info(cmd)
if dry_run:
return 0, ''
with subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) as proc:
for line in proc.stdout:
logger.info(line.strip())
proc.communicate()
if proc.returncode != 0:
msg = f"Command '{cmd}' returned non-zero exit status {proc.returncode}"
if fail_on_error:
raise RuntimeError(msg)
logger.warning(msg)
return proc.returncode
def name_from_dockerfile(dockerfile: str | Path, dockerfiles_root: str | Path) -> str:
image_name = str(Path(dockerfile).relative_to(dockerfiles_root).parent.as_posix())
return image_name

View File

@ -1,137 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import logging
import os
import subprocess
from pathlib import Path
from typing import Iterable
from helpers import run, name_from_dockerfile
class Image:
def __init__(self, name: str, dockerfile: Path, registry: str):
self.logger = logging.getLogger(self.__class__.__name__)
self.name = name
self.dockerfile = dockerfile
self.registry = registry
self.tag = 'latest'
self.base_tag = None
def __str__(self):
return self.name
def __eq__(self, img):
return img.name == self.name if img else False
def with_tag(self, tag: str):
self.tag = tag
return self
def with_base_tag(self, tag: str):
self.base_tag = tag
return self
def ref(self):
return f"{self.registry}/{self.name}:{self.tag}"
def base_ref(self):
if not self.base_tag:
return None
return f"{self.registry}/{self.name}:{self.base_tag}"
def push(self, dry: bool = False):
cmd = f"docker push {self.ref()} "
run(cmd, dry)
def build(self, dry: bool = False, push: bool = True, docker_builder: str = None, import_cache: bool = True,
export_cache: bool = True):
cache_cmd = ""
if import_cache:
cache_cmd += f"--cache-from type=registry,ref={self.ref()}-cache "
if self.base_tag:
cache_cmd += f"--cache-from type=registry,ref={self.base_ref()}-cache "
if export_cache:
cache_cmd += f"--cache-to type=registry,ref={self.ref()}-cache,mode=max "
build_cmd = f"docker buildx build --builder={docker_builder}" if docker_builder else "docker build"
push_cmd = f"--push" if push else ""
cmd = f"{build_cmd} " \
f"--file {self.dockerfile} " \
f"--tag {self.ref()} " \
f"{cache_cmd} " \
f"{push_cmd} " \
"."
run(cmd, dry)
def tag_base(self, dry: bool = False):
if not self.base_tag:
raise AttributeError("Tag for base image is not specified")
cmd = f"docker buildx imagetools create -t {self.ref()} {self.base_ref()}"
run(cmd, dry)
def is_missing(self, dry: bool = False, base: bool = False) -> bool:
image = self.base_ref() if base else self.ref()
if base and not image:
self.logger.warning(f"Base ref for image {self.ref()} is missing")
return True
cmd = f"docker manifest inspect {image}"
is_missing = False
self.logger.info(cmd)
if not dry:
try:
subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True, text=True)
except subprocess.CalledProcessError:
self.logger.warning(f"{image} is missing in registry")
is_missing = True
return is_missing
# Making it a class, so it's a little easier to switch to a tree structure for building inherited images if we want
class ImagesHandler:
def __init__(self, dry_run: bool = False):
self.logger = logging.getLogger(self.__class__.__name__)
self.images = dict()
self.dry_run = dry_run
def add_from_dockerfile(self, dockerfile: str | Path, dockerfiles_root: str | Path, registry: str, tag: str,
base_tag: str = None):
image_name = name_from_dockerfile(dockerfile, dockerfiles_root)
image = Image(image_name, Path(dockerfile), registry).with_tag(tag).with_base_tag(base_tag)
self.add(image)
def add(self, image: Image):
self.images[image.name] = image
def get(self, image_names: Iterable = None) -> list:
images = [self.images[name] for name in image_names] if image_names is not None else self.images.values()
return images
def get_missing(self, image_names: Iterable = None, base: bool = False) -> list:
missing_images = [image.name for image in self.get(image_names) if image.is_missing(self.dry_run, base)]
return missing_images
def build(self, image_names: Iterable = None, missing_only: bool = False, push: bool = True, builder: str = None,
import_cache: bool = True, export_cache: bool = True):
to_build = self.get(self.get_missing(image_names)) if missing_only else self.get(image_names)
for image in to_build:
image.build(self.dry_run, push, builder, import_cache, export_cache)
return to_build
def push(self, image_names: Iterable = None, missing_only: bool = False):
to_push = self.get(self.get_missing(image_names)) if missing_only else self.get(image_names)
for image in to_push:
image.push(self.dry_run)
def tag(self, image_names: Iterable = None):
for image in self.get(image_names):
image.tag_base(self.dry_run)

View File

@ -1 +0,0 @@
ghapi~=1.0.5

View File

@ -1,82 +0,0 @@
name: 'Setup Python and pip cache'
description: 'Setups Python with the provided version and sets up the pip cache'
inputs:
version:
description: 'Python version to install'
required: true
pip-cache-path:
description: 'Path on share where the pip cache is stored'
required: false
should-setup-pip-paths:
description: 'If the action should setup `PIP_CACHE_DIR` & `PIP_INSTALL_PATH` env variables'
required: false
default: 'false'
self-hosted-runner:
description: 'If the runner is self-hosted'
required: false
default: 'true'
show-cache-info:
description: 'If the action should show the share space occupied by cache'
required: false
default: 'false'
runs:
using: 'composite'
steps:
- name: Check if Python is already installed (Linux)
if: ${{ runner.os == 'Linux' }}
shell: bash
id: check_python
run: |
PYTHON_INSTALLED=$(python${{ inputs.version }} -V) || true
if [[ $PYTHON_INSTALLED ]]; then
echo "installed=true" >> $GITHUB_OUTPUT
else
echo "installed=false" >> $GITHUB_OUTPUT
fi
- if: ${{ runner.os == 'Linux' && inputs.self-hosted-runner == 'true' && steps.check_python.outputs.installed == 'false' }}
name: Install 'actions/setup-python@v4' dependencies
shell: bash
run: apt-get update && apt-get install -y ca-certificates software-properties-common gpg-agent tzdata
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
TZ: "Europe/London" # to prevent tzdata from waiting user input
- if: ${{ runner.os == 'Linux' && runner.arch == 'ARM64' && steps.check_python.outputs.installed == 'false' }}
name: Setup sudo and python3
shell: bash
run: apt-get update && apt-get install -y sudo python3 # Needed for the deadsnakes action
- if: ${{ runner.os == 'Linux' && runner.arch == 'ARM64' && steps.check_python.outputs.installed == 'false' }}
name: Setup Python ${{ inputs.version }}
uses: akashchi/deadsnakes-action@92417281055a5878a0450f240a5b95883eb2d7e2
with:
python-version: ${{ inputs.version }}
- if: ${{ runner.os == 'macOS' || runner.os == 'Windows' || (runner.os == 'Linux' && runner.arch != 'ARM64' && steps.check_python.outputs.installed == 'false' ) }}
name: Setup Python ${{ inputs.version }}
uses: actions/setup-python@v5
with:
python-version: ${{ inputs.version }}
env:
PIP_CACHE_DIR: ${{ inputs.self-hosted-runner == 'true' && inputs.pip-cache-path || '' }}
- if: ${{ inputs.should-setup-pip-paths == 'true' }}
name: Setup pip variables (cache and install path)
shell: bash
run: |
PIP_VER=$(python3 -c "import pip; print(pip.__version__)")
echo "Using pip version: ${PIP_VER}"
echo "PIP_CACHE_DIR=${{ inputs.pip-cache-path }}/${PIP_VER}" >> $GITHUB_ENV
echo "PIP_INSTALL_PATH=$(python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')" >> $GITHUB_ENV
- if: ${{ inputs.show-cache-info == 'true' }}
name: Get pip cache info
shell: bash
run: |
echo "Cache size: "
du -h -d2 ${{ env.PIP_CACHE_DIR }}
echo "Cache info: "
python3 -m pip cache info
continue-on-error: true

View File

@ -1,106 +0,0 @@
name: "Smart CI action"
description: "Returns product components affected by PR or commit"
inputs:
repository:
description: "GitHub repository"
required: true
repo_token:
description: "Token for access to GitHub repository"
required: true
pr:
description: "GitHub PR number. If not set - commit is used"
required: false
commit_sha:
description: "GitHub commit hash. Used if no PR number is set"
required: false
ref_name:
description: "GitHub ref name"
required: false
component_pattern:
description: "Pattern to extract component name from PR label. If not set, any label is considered a component name"
required: false
labeler_check_name:
description: "Name of the labeler check"
required: false
default: "triage"
components_config:
description: "Path to components configuration file"
required: false
default: ".github/components.yml"
components_config_schema:
description: "Path to the schema file for components configuration"
required: false
default: ".github/actions/smart-ci/components_schema.yml"
labeler_config:
description: "Path to labeler configuration file"
required: false
default: ".github/labeler.yml"
skip_when_only_listed_labels_set:
description: "Comma-separated list of labels. If PR has only these labels set,
return indicator that CI can be skipped"
required: false
skip_when_only_listed_files_changed:
description: "Comma-separated list of patterns (fnmatch-style). If PR has only matching files changed,
return indicator that CI can be skipped"
required: false
outputs:
all_components:
description: "All components listed in configuration"
value: ${{ steps.smart_ci.outputs.all_components }}
affected_components:
description: "Affected components to run validation for and their validation scope"
value: ${{ steps.smart_ci.outputs.affected_components }}
changed_components:
description: "Actually changed components (for push events everything is marked as changed)"
value: ${{ steps.smart_ci.outputs.changed_components }}
skip_workflow:
description: "Whether the workflow should be run with Smart CI rules applied or skipped completely"
value: ${{ steps.smart_ci.outputs.skip_workflow }}
runs:
using: "composite"
steps:
- name: Wait for labeler to finish
uses: lewagon/wait-on-check-action@v1.3.1
if: ${{ github.event_name == 'pull_request' }}
with:
ref: ${{ github.event.pull_request.head.sha }}
check-name: ${{ inputs.labeler_check_name }}
repo-token: ${{ inputs.repo_token }}
wait-interval: 10
- name: checkout components file
uses: actions/checkout@v4
with:
sparse-checkout: .github/components.yml
sparse-checkout-cone-mode: false
- name: Install Python dependencies
uses: py-actions/py-dependency-install@v4
with:
path: "${{ github.action_path }}/requirements.txt"
update-setuptools: "false"
update-wheel: "false"
- name: Test functionality
run: |
python ${{ github.action_path }}/smart_ci_test.py
shell: bash
- name: Smart CI
id: smart_ci
run: |
python ${{ github.action_path }}/smart_ci.py \
$([[ -n "${{ inputs.pr }}" ]] && echo '--pr ${{ inputs.pr }}' || echo '-s ${{ inputs.commit_sha }}') \
-r ${{ inputs.repository }} \
-f "${{ inputs.ref_name }}" \
-p "${{ inputs.component_pattern }}" \
-c "${{ inputs.components_config }}" \
-m "${{ inputs.components_config_schema }}" \
-l "${{ inputs.labeler_config }}" \
--skip-when-only-listed-labels-set "${{ inputs.skip_when_only_listed_labels_set }}" \
--skip-when-only-listed-files-changed "${{ inputs.skip_when_only_listed_files_changed }}"
shell: bash
env:
GITHUB_TOKEN: ${{ inputs.repo_token }}

View File

@ -1,41 +0,0 @@
# YAML schema for Smart CI configuration file components.yml (see https://json-schema.org)
definitions:
component_name:
type: string
pattern: "^[a-zA-Z_][a-zA-Z0-9_]*$"
component_data:
type: object # dict
additionalProperties: false
properties:
cmake:
type: array
uniqueItems: true
items:
'$ref': '#/definitions/component_name'
revalidate:
oneOf:
- type: array
uniqueItems: true
items:
'$ref': '#/definitions/component_name'
- type: string
enum: ['all']
build:
oneOf:
- type: array
uniqueItems: true
items:
'$ref': '#/definitions/component_name'
- type: string
enum: ['all']
propertyNames: # Validates component names
'$ref': '#/definitions/component_name'
patternProperties:
".*": # Component (name validated via propertyNames)
'$ref': '#/definitions/component_data'
additionalProperties: false

View File

@ -1,3 +0,0 @@
ghapi~=1.0.4
pyyaml~=6.0.1
jsonschema~=4.19.1

View File

@ -1,258 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import os
import re
import argparse
import yaml
import json
import jsonschema
import logging
from pathlib import Path
from ghapi.all import GhApi
from fnmatch import fnmatch
class ComponentConfig:
FullScope = {'build', 'test'}
ScopeKeys = {'build', 'revalidate'}
def __init__(self, config: dict, schema: dict, all_possible_components: set):
self.config = config
self.log = logging.getLogger(self.__class__.__name__)
self.all_defined_components = set(self.config.keys()) # already defined in components.yml
# can be added to components.yml (based on labeler.yml)
self.all_possible_components = all_possible_components.union(self.all_defined_components)
self.validate(schema)
def validate(self, schema: dict) -> None:
"""Validates syntax of configuration file"""
jsonschema.validate(self.config, schema)
for component_name, data in self.config.items():
dependent_components = set()
for key in self.ScopeKeys:
scope = data.get(key)
dependent_components = dependent_components.union(set(scope) if isinstance(scope, list) else set())
invalid_dependents = dependent_components.difference(self.all_possible_components)
if invalid_dependents:
error_msg = f"dependent components of {component_name} are invalid: " \
f"{invalid_dependents} are not listed in components config: {self.all_possible_components}"
raise jsonschema.exceptions.ValidationError(error_msg)
def get_affected_components(self, changed_components_names: set) -> dict:
"""Returns changed components, their dependencies and validation scope for them"""
affected_components = dict()
# If some changed components were not defined in config or no changed components detected at all,
# run full scope for everything (just in case)
changed_not_defined_components = changed_components_names.difference(self.all_defined_components)
if not changed_components_names or changed_not_defined_components:
self.log.info(f"Changed components {changed_not_defined_components} are not defined in smart ci config, "
"run full scope")
affected_components.update({name: self.FullScope for name in self.all_possible_components})
return affected_components
# Else check changed components' dependencies and add them to affected
for name in changed_components_names:
component_scopes = {k: v for k, v in self.config.get(name, dict()).items() if k in self.ScopeKeys}
for key, dependents in component_scopes.items():
if dependents == 'all':
dependents = self.all_possible_components
for dep_name in dependents:
affected_components[dep_name] = affected_components.get(dep_name, set())
scope = self.FullScope if key == 'revalidate' else {key}
affected_components[dep_name] = affected_components[dep_name].union(scope)
if not component_scopes:
self.log.info(f"Changed component '{name}' doesn't have {self.ScopeKeys} keys in components config. "
f"Assuming that it affects everything, the whole scope will be started")
for dep_name in self.all_possible_components:
affected_components[dep_name] = self.FullScope
# If the component was explicitly changed, run full scope for it
affected_components.update({name: self.FullScope for name in changed_components_names})
self.log.info(f"Changed components with dependencies: {affected_components}")
# For non-affected components that are not defined in config - run full scope
affected_components.update({name: self.FullScope for name in self.all_possible_components
if name not in self.all_defined_components})
return affected_components
def get_static_data(self, components_names: set, data_key: str, default: str = None) -> dict:
"""Returns requested generic static data defined for each component"""
data = {name: self.config[name].get(data_key, default) for name in components_names}
return data
def component_name_from_label(label: str, component_pattern: str = None) -> str:
"""Extracts component name from label"""
component = label
if component_pattern:
matches = re.findall(component_pattern, label)
component = matches[0] if matches else None
component = component.replace(' ', '_') if component else None
return component
def get_changed_component_names(pr, all_possible_components: set, component_pattern: str = None) -> set:
"""Returns component names changed in a given PR"""
components = set()
for label in pr.labels:
component = component_name_from_label(label.name, component_pattern)
if component:
components.add(component)
elif label.name in all_possible_components:
# Allow any labels defined explicitly in labeler config as components
# (predefined labels, such as "do not merge", are still ignored)
components.add(label.name)
return components
def get_changeset(gh_api, pr, target_branch, commit_sha):
"""Returns changeset either from PR or commit"""
if pr:
return gh_api.pulls.list_files(pr)
if target_branch:
target_branch_head_commit = gh_api.repos.get_branch(target_branch).commit.sha
# In merge-queue branch all commits between head of target branch and head of current branch (commit_sha)
# contain changes added to queue earlier to be validated together. Getting all of them + changes from
# commit_sha below
changed_files = gh_api.repos.compare_commits(f'{target_branch_head_commit}...{commit_sha}').get('files', [])
return changed_files
raise ValueError(f'Either "pr" or "target_branch" parameter must be non-empty')
def parse_args():
parser = argparse.ArgumentParser(description='Returns product components changed in a given PR or commit')
parser.add_argument('--pr', type=int, required=False, help='PR number. If not set, --commit is used')
parser.add_argument('-s', '--commit-sha', required=False, help='Commit SHA. If not set, --pr is used')
parser.add_argument('-r', '--repo', help='GitHub repository')
parser.add_argument('-f', '--ref_name', required=False, help='GitHub ref name')
parser.add_argument('-p', '--pattern', default=None, help='Pattern to extract component name from PR label. '
'If not set, any label is considered a component name')
parser.add_argument('-c', '--components-config', default='.github/components.yml',
help='Path to config file with info about dependencies between components')
parser.add_argument('-m', '--components-config-schema', default='.github/actions/smart-ci/components_schema.yml',
help='Path to the schema file for components config')
parser.add_argument('-l', '--labeler-config', default='.github/labeler.yml',
help='Path to PR labeler config file')
parser.add_argument('--skip-when-only-listed-labels-set',
help="Comma-separated list of labels. If PR has only these labels set, "
"return indicator that CI can be skipped")
parser.add_argument('--skip-when-only-listed-files-changed',
help="Comma-separated list of patterns (fnmatch-style). If PR has only matching files changed, "
"return indicator that CI can be skipped")
args = parser.parse_args()
return args
def init_logger():
logging.basicConfig(level=logging.INFO,
format='%(asctime)s %(name)-15s %(levelname)-8s %(message)s',
datefmt='%m-%d-%Y %H:%M:%S')
def set_github_output(name: str, value: str, github_output_var_name: str = 'GITHUB_OUTPUT'):
"""Sets output variable for a GitHub Action"""
logger = logging.getLogger(__name__)
# In an environment variable "GITHUB_OUTPUT" GHA stores path to a file to write outputs to
with open(os.environ.get(github_output_var_name), 'a+') as file:
logger.info(f"Add {name}={value} to {github_output_var_name}")
print(f'{name}={value}', file=file)
def main():
init_logger()
logger = logging.getLogger(__name__)
args = parse_args()
for arg, value in sorted(vars(args).items()):
logger.info(f"Argument {arg}: {value}")
with open(Path(args.components_config), 'r') as config:
components_config = yaml.safe_load(config)
owner, repository = args.repo.split('/')
gh_api = GhApi(owner=owner, repo=repository, token=os.getenv("GITHUB_TOKEN"))
pr = gh_api.pulls.get(args.pr) if args.pr else None
with open(Path(args.components_config_schema), 'r') as schema_file:
schema = yaml.safe_load(schema_file)
with open(Path(args.labeler_config), 'r') as labeler_file:
labeler_config = yaml.safe_load(labeler_file)
all_possible_components = set()
for label in labeler_config.keys():
component_name = component_name_from_label(label, args.pattern)
all_possible_components.add(component_name if component_name else label)
run_full_scope = False
# For now, we don't want to apply smart ci rules for post-commits
is_postcommit = not pr
merge_queue_prefix = 'gh-readonly-queue/'
is_merge_queue = args.ref_name.startswith(merge_queue_prefix)
merge_queue_target_branch = re.findall(f'^{merge_queue_prefix}(.*)/', args.ref_name)[0] if is_merge_queue else None
if is_merge_queue:
logger.info(f"The run is a merge-queue run, executing full validation scope for all components, if "
f"not all queued changes match patterns in 'skip-when-only-listed-files-changed'")
run_full_scope = True
elif is_postcommit:
logger.info(f"The run is a post-commit run, executing full validation scope for all components")
run_full_scope = True
else:
no_match_files_changed = 'no-match-files' in [label.name for label in pr.labels]
if no_match_files_changed:
logger.info(f"There are changed files that don't match any pattern in labeler config, "
f"executing full validation scope for all components")
run_full_scope = True
# In post-commits - validate all components regardless of changeset
# In pre-commits - validate only changed components with their dependencies
all_defined_components = components_config.keys()
changed_by_pr = get_changed_component_names(pr, all_possible_components, args.pattern) if pr else None
changed_component_names = set(all_defined_components) if run_full_scope else changed_by_pr
logger.info(f"changed_component_names: {changed_component_names}")
cfg = ComponentConfig(components_config, schema, all_possible_components)
affected_components = cfg.get_affected_components(changed_component_names)
skip_workflow = False
if is_merge_queue or (args.pr and not run_full_scope):
if args.skip_when_only_listed_labels_set:
excepted_labels = set(args.skip_when_only_listed_labels_set.split(','))
excepted_labels_only = changed_component_names - excepted_labels == set()
skip_workflow = excepted_labels_only
if not skip_workflow and args.skip_when_only_listed_files_changed:
# To avoid spending extra API requests running step below only if necessary
changed_files = get_changeset(gh_api, args.pr, merge_queue_target_branch, args.commit_sha)
patterns = set(args.skip_when_only_listed_files_changed.split(','))
matched_files_only = all(any(fnmatch(f.filename, pattern) for pattern in patterns) for f in changed_files)
logger.debug(f"matched files only: {matched_files_only}")
skip_workflow = matched_files_only
if skip_workflow:
logger.info(f"All changes are marked for skip, workflow may be skipped")
set_github_output("skip_workflow", str(skip_workflow))
# Syntactic sugar for easier use in GHA pipeline
affected_components_output = {name: {s: True for s in scope} for name, scope in affected_components.items()}
set_github_output("affected_components", json.dumps(affected_components_output))
# Components actually changed by a pull request are marked as True (if event is PR;
# otherwise all components considered changed).
changed_components_output = {name: True if not pr or name in changed_by_pr else False
for name in all_possible_components}
set_github_output("changed_components", json.dumps(changed_components_output))
if __name__ == '__main__':
main()

View File

@ -1,217 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import logging
import sys
import unittest
from smart_ci import ComponentConfig
log = logging.getLogger()
log.level = logging.DEBUG
def log_handler(func):
def wrapper(*args, **kwargs):
stream_handler = logging.StreamHandler(sys.stdout)
log.addHandler(stream_handler)
result = func(*args, **kwargs)
log.removeHandler(stream_handler)
return result
return wrapper
class TestComponentConfig(unittest.TestCase):
def setUp(self):
self.all_possible_components = {'comp1', 'comp2', 'comp3', 'comp4'}
ComponentConfig.ScopeKeys = {'build', 'revalidate', '_scope_1', '_scope_2', '_scope_3'}
@log_handler
def validate(self, config_data: dict, changed_components: set, expected_result: dict):
log.info(f"{self._testMethodName}:")
config = ComponentConfig(config_data, {}, self.all_possible_components)
result = config.get_affected_components(changed_components)
self.assertEqual(expected_result, result)
def test_no_changed_components(self):
config_data = {
'comp1': {'build': {}, 'revalidate': {}},
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {}, 'revalidate': {}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = set()
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': ComponentConfig.FullScope,
'comp3': ComponentConfig.FullScope,
'comp4': ComponentConfig.FullScope,
}
self.validate(config_data, changed_components, expected_result)
def test_all_components_changed(self):
config_data = {
'comp1': {'build': {}, 'revalidate': {}},
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {}, 'revalidate': {}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = {'comp1', 'comp2', 'comp3', 'comp4'}
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': ComponentConfig.FullScope,
'comp3': ComponentConfig.FullScope,
'comp4': ComponentConfig.FullScope,
}
self.validate(config_data, changed_components, expected_result)
def test_changed_component_not_defined(self):
config_data = {
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {}, 'revalidate': {}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = {'comp1'}
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': ComponentConfig.FullScope,
'comp3': ComponentConfig.FullScope,
'comp4': ComponentConfig.FullScope,
}
self.validate(config_data, changed_components, expected_result)
def test_component_changed_no_scope_keys(self):
config_data = {
'comp1': {},
'comp2': {},
'comp3': {},
'comp4': {},
}
changed_components = {'comp1'}
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': ComponentConfig.FullScope,
'comp3': ComponentConfig.FullScope,
'comp4': ComponentConfig.FullScope,
}
self.validate(config_data, changed_components, expected_result)
def test_one_component_changed_dependents_empty(self):
config_data = {
'comp1': {'build': {}, 'revalidate': {}},
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {}, 'revalidate': {}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = {'comp1'}
expected_result = {
'comp1': ComponentConfig.FullScope,
}
self.validate(config_data, changed_components, expected_result)
def test_not_changed_dependent_component(self):
config_data = {
'comp1': {'build': {'comp2'}, 'revalidate': {}},
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {}, 'revalidate': {}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = {'comp1'}
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': {'build'}
}
self.validate(config_data, changed_components, expected_result)
def test_changed_dependent_component(self):
config_data = {
'comp1': {'build': {'comp2'}, 'revalidate': {}},
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {}, 'revalidate': {}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = {'comp1', 'comp2'}
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': ComponentConfig.FullScope
}
self.validate(config_data, changed_components, expected_result)
def test_dependent_component_multiple_parents(self):
config_data = {
'comp1': {'_scope_1': {'comp2'}, 'revalidate': {}},
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {}, '_scope_2': {'comp2'}, '_scope_3': {'comp2'}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = {'comp1', 'comp3'}
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': {'_scope_1', '_scope_2', '_scope_3'},
'comp3': ComponentConfig.FullScope
}
self.validate(config_data, changed_components, expected_result)
def test_dependent_component_empty_scopes(self):
config_data = {
'comp1': {'build': {}, 'revalidate': {'comp2'}},
'comp2': {},
'comp3': {},
'comp4': {},
}
changed_components = {'comp1', 'comp3'}
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': ComponentConfig.FullScope,
'comp3': ComponentConfig.FullScope,
'comp4': ComponentConfig.FullScope
}
self.validate(config_data, changed_components, expected_result)
def test_changed_component_empty_dependencies(self):
config_data = {
'comp1': {'build': {}, 'revalidate': {}},
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {}, 'revalidate': {}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = {'comp1'}
expected_result = {
'comp1': ComponentConfig.FullScope,
}
self.validate(config_data, changed_components, expected_result)
def test_multiple_dependents(self):
config_data = {
'comp1': {'build': {'comp2'}, 'revalidate': {'comp3'}},
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {'comp4'}, 'revalidate': {}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = {'comp1'}
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': {'build'},
'comp3': ComponentConfig.FullScope,
# We don't consider dependencies of dependencies affected, so comp4 is not expected here
}
self.validate(config_data, changed_components, expected_result)
def test_all_as_dependents(self):
config_data = {
'comp1': {'build': 'all', 'revalidate': {}},
'comp2': {'build': {}, 'revalidate': {}},
'comp3': {'build': {}, 'revalidate': {}},
'comp4': {'build': {}, 'revalidate': {}},
}
changed_components = {'comp1'}
expected_result = {
'comp1': ComponentConfig.FullScope,
'comp2': {'build'},
'comp3': {'build'},
'comp4': {'build'},
}
self.validate(config_data, changed_components, expected_result)
if __name__ == '__main__':
unittest.main(verbosity=2)

227
.github/components.yml vendored
View File

@ -1,227 +0,0 @@
Core:
revalidate: 'all'
transformations:
revalidate: 'all'
LP_transformations:
revalidate:
- CPU
- GPU
- PyTorch_FE
- TF_FE
- TFL_FE
- ONNX_FE
- PDPD_FE
preprocessing:
revalidate:
- inference
- C_API
- Python_API
inference:
revalidate: 'all'
CPU:
revalidate:
- C_API
- Python_API
- samples
- ONNX_RT
- PyTorch_FE
- TF_FE
- ONNX_FE
build:
- AUTO
- HETERO
- AUTO_BATCH
- TEMPLATE
- IR_FE
GPU:
build:
- AUTO
- HETERO
- AUTO_BATCH
- TEMPLATE
- IR_FE
- PROXY
NPU:
revalidate: []
build: []
HETERO:
revalidate:
- Python_API
build:
- TEMPLATE
- IR_FE
AUTO_BATCH:
revalidate:
- GPU
- C_API
- Python_API
build:
- TEMPLATE
- IR_FE
TEMPLATE:
revalidate:
- CPU
- GPU
- HETERO
- AUTO_BATCH
- AUTO
- C_API
- Python_API
- NVIDIA
build:
- IR_FE
AUTO:
revalidate:
- C_API
- Python_API
build:
- TEMPLATE
- IR_FE
PROXY:
revalidate:
- inference
- GPU
build: []
IR_FE:
revalidate:
- C_API
- Python_API
- samples
build:
- CPU
ONNX_FE:
revalidate:
- MO
- ONNX_RT
build:
- CPU
- Python_API
PDPD_FE:
revalidate:
- MO
build:
- CPU
- Python_API
TF_FE:
revalidate:
- MO
build:
- CPU
- Python_API
- TOKENIZERS
TFL_FE:
revalidate:
- MO
build:
- CPU
- Python_API
PyTorch_FE:
revalidate:
- MO
build:
- CPU
- Python_API
- TOKENIZERS
C_API:
build:
- CPU
- GPU
- HETERO
- AUTO_BATCH
- AUTO
- IR_FE
Python_API:
revalidate:
- samples
- MO
- tools
build:
- CPU
- HETERO
- AUTO_BATCH
- TEMPLATE
- AUTO
- IR_FE
- ONNX_FE
- PDPD_FE
- TF_FE
- TFL_FE
- PyTorch_FE
JS_API:
build:
- CPU
- IR_FE
samples:
build:
- CPU
- AUTO_BATCH
- AUTO
- IR_FE
- C_API
- Python_API
IE_Tests:
revalidate:
- CPU
- GPU
- HETERO
- AUTO_BATCH
- TEMPLATE
- AUTO
- NVIDIA
build:
- IR_FE
MO:
revalidate:
- PyTorch_FE
- TF_FE
build:
- Python_API
tools:
build:
- CPU
- Python_API
docs:
revalidate: []
build: []
licensing:
revalidate: []
build: []
NVIDIA:
revalidate: []
build: []
ONNX_RT:
revalidate: []
build: []
TOKENIZERS:
revalidate: []
build: []

View File

@ -61,6 +61,23 @@ updates:
dependency-type: "production"
versioning-strategy: increase-if-necessary
# POT requirements
- package-ecosystem: pip
directory: "/tools/pot"
schedule:
interval: "daily"
time: "09:00"
timezone: "Asia/Dubai"
open-pull-requests-limit: 3
assignees:
- "AlexKoff88"
- "KodiaqQ"
- "jiwaszki"
- "p-wysocki"
- "akuporos"
- "rkazants"
versioning-strategy: increase-if-necessary
#
# Python Samples
#

View File

@ -5,18 +5,9 @@ allow-licenses:
- 'BSD-2-Clause AND BSD-3-Clause'
- 'MIT'
- 'Apache-2.0'
- 'ISC'
- 'Apache-2.0 AND MIT'
- 'BlueOak-1.0.0'
- '0BSD'
- 'Python-2.0'
- 'LGPL-3.0'
fail-on-scopes:
- 'runtime'
- 'development'
- 'unknown'
license-check: true
vulnerability-check: true
allow-dependencies-licenses:
- 'pkg:pypi/PyGithub@2.2.0'
- 'pkg:pypi/psycopg2-binary'

View File

@ -1 +0,0 @@
pr-24878

View File

@ -1,73 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
libhdf5-dev \
# For Java API
default-jdk \
# Compiler
gcc-10 \
g++-10 \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set gcc-10 as a default compiler
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 30 && \
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 30
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-aarch64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,72 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
# For Java API
default-jdk \
# Compiler \
gcc-10 \
g++-10 \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set gcc-10 as a default compiler
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 30 && \
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 30
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,89 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/nvidia/cuda:11.8.0-runtime-ubuntu20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
wget \
git \
ca-certificates \
gpg-agent \
tzdata \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
# For Java API
default-jdk \
# Compiler \
gcc-10 \
g++-10 \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set gcc-10 as a default compiler
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 30 && \
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 30
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Install CUDA
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin && \
mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \
apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub && \
add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
RUN apt update && apt install -y \
libcudnn8=8.9.4.*-1+cuda11.8 \
libcudnn8-dev=8.9.4.*-1+cuda11.8 \
libcudnn8-samples=8.9.4.*-1+cuda11.8 \
cuda-runtime-11-8 \
cuda-11-8 \
libcutensor1=1.6.1.5-1 \
libcutensor-dev=1.6.1.5-1 \
cuda-drivers=520.61.05-1 && \
rm -rf /var/lib/apt/lists/*
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,73 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:22.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
libtbb2 \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
# For Java API
default-jdk \
# Compiler \
gcc-10 \
g++-10 \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set gcc-10 as a default compiler
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 30 && \
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 30
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,72 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:22.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
libtbb2 \
# Pythons
python3.8-dev \
python3.8-venv \
python3.8-distutils \
python3.11-dev \
python3.11-venv \
python3.11-distutils \
# For Java API
default-jdk \
# Compiler \
clang \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Set clang as a default compiler
RUN update-alternatives --install /usr/bin/cc cc /usr/bin/clang 100 && \
update-alternatives --install /usr/bin/c++ c++ /usr/bin/clang++ 100
# Install sscache
ARG SCCACHE_VERSION="v0.7.5"
ENV SCCACHE_HOME="/opt/sccache" \
SCCACHE_PATH="/opt/sccache/sccache"
RUN mkdir ${SCCACHE_HOME} && cd ${SCCACHE_HOME} && \
SCCACHE_ARCHIVE="sccache-${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" && \
curl -SLO https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_ARCHIVE} && \
tar -xzf ${SCCACHE_ARCHIVE} --strip-components=1 && rm ${SCCACHE_ARCHIVE}
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}

View File

@ -1,51 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Python
python3.11-dev \
python3.11-venv \
python3.11-distutils \
libhdf5-dev \
&& \
rm -rf /var/lib/apt/lists/*
# Install openvino dependencies
ADD scripts/install_dependencies/install_openvino_dependencies.sh /install_openvino_dependencies.sh
RUN chmod +x /install_openvino_dependencies.sh && \
/install_openvino_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}
ENV PIP_INSTALL_PATH=/venv/lib/python3.11/site-packages

View File

@ -1,52 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Python
python3.11-dev \
python3.11-venv \
python3.11-distutils \
libhdf5-dev \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3.8 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default instead of Python 3.8
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}
ENV PIP_INSTALL_PATH=/venv/lib/python3.11/site-packages

View File

@ -1,52 +0,0 @@
FROM openvinogithubactions.azurecr.io/dockerhub/ubuntu:22.04
USER root
# APT configuration
RUN echo 'Acquire::Retries "10";' > /etc/apt/apt.conf && \
echo 'APT::Get::Assume-Yes "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::Fix-Broken "true";' >> /etc/apt/apt.conf && \
echo 'APT::Get::no-install-recommends "true";' >> /etc/apt/apt.conf
ENV DEBIAN_FRONTEND="noninteractive" \
TZ="Europe/London"
RUN apt-get update && \
apt-get install software-properties-common && \
add-apt-repository --yes --no-update ppa:git-core/ppa && \
add-apt-repository --yes --no-update ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install \
curl \
git \
ca-certificates \
gpg-agent \
tzdata \
# Python
python3.11-dev \
python3.11-venv \
python3.11-distutils \
libhdf5-dev \
&& \
rm -rf /var/lib/apt/lists/*
# Install build dependencies
ADD install_build_dependencies.sh /install_build_dependencies.sh
RUN chmod +x /install_build_dependencies.sh && \
/install_build_dependencies.sh && \
rm -rf /var/lib/apt/lists/*
# Setup pip
ENV PIP_VERSION="24.0"
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python3 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
python3.11 get-pip.py --no-cache-dir pip==${PIP_VERSION} && \
rm -f get-pip.py
# Use Python 3.11 as default
# Using venv here 'cause other methods to switch the default Python on Ubuntu 20 break both system and wheels build
RUN python3.11 -m venv venv
ENV PATH="/venv/bin:$SCCACHE_HOME:$PATH"
ENV PIP_CACHE_DIR=/mount/caches/pip/linux/${PIP_VERSION}
ENV PIP_INSTALL_PATH=/venv/lib/python3.11/site-packages

View File

@ -6,6 +6,7 @@
"openvino-ci",
"openvino-pushbot",
"workbench-ci-bot",
"openvino-pot-ci",
"sysicvvpux",
"ote-ci-bot"
],
@ -23,6 +24,7 @@
"openvino-docs-maintainers": "category: docs",
"openvino-ie-maintainers": "category: inference",
"openvino-ie-cpu-maintainers": "category: CPU",
"openvino-ie-gna-maintainers": "category: GNA",
"openvino-ie-gpu-maintainers": "category: GPU",
"openvino-ie-lpt-maintainers": "category: LP transformations",
"openvino-ie-transformations-maintainers": "category: transformations",
@ -41,6 +43,7 @@
"openvino-scripts-maintainers": "category: build",
"openvino-tests-maintainers": "category: IE Tests",
"openvino-tools-maintainers": "category: tools",
"openvino-pot-maintainers": "category: POT",
"openvino-configuration-mgmt": "category: dependency_changes",
"openvino-samples-maintainers": "category: samples",
"openvino-c-api-maintainers": "category: C API"

67
.github/labeler.yml vendored
View File

@ -48,46 +48,39 @@
- 'scripts/**/*'
- '.gitmodules'
- '**/setup.py'
- 'conan.lock'
- 'conanfile.txt'
- 'vcpkg.json'
- any: ['thirdparty/**/*',
'!thirdparty/**/CMakeLists.txt']
'category: docs':
- 'docs/**/*'
- '**/*.md'
- any: ['docs/**/*',
'!docs/snippets/**/*']
'category: docs_snippets':
- 'docs/snippets/**/*'
'category: extensions':
- 'src/core/include/openvino/core/extension.hpp'
- 'src/frontends/common/include/openvino/frontend/extension.hpp'
- 'src/frontends/common/include/openvino/frontend/extension/**/*'
'category: GNA':
- 'src/plugins/intel_gna/**/*'
'category: GPU':
- 'src/plugins/intel_gpu/**/*'
- 'src/tests/**/gpu/**/*'
- 'thirdparty/ocl/**/*'
'category: NPU':
- 'src/plugins/intel_npu/**/*'
'category: HETERO':
- 'src/plugins/hetero/**/*'
'category: PROXY':
- 'src/plugins/proxy/**/*'
'category: IE Tests':
- 'thirdparty/gtest/**/*'
- 'src/frontends/tests/frontend/shared/**/*'
- 'src/tests/**/*'
- any: ['src/tests/**/*',
'!src/tests/**/gpu/**/*',
'!src/tests/**/inference_engine/**/*']
'category: inference':
- 'src/inference/**/*'
- 'src/cmake/**/*'
- 'src/tests/functional/inference_engine/**/*'
'category: IR FE':
- 'src/frontends/ir/**/*'
@ -98,25 +91,21 @@
'category: MO':
- 'tools/mo/**/*'
- 'tools/ovc/**/*'
- 'tests/layer_tests/mo_python_api_tests/**/*'
- 'tests/layer_tests/ovc_python_api_tests/**/*'
'category: ONNX FE':
- 'src/frontends/onnx/**/*'
- 'thirdparty/onnx/**/*'
- 'tests/layer_tests/onnx_tests/**/*'
'category: packaging':
- 'cmake/**/packaging/**/*'
- 'src/bindings/python/wheel/**/*'
- any: ['src/bindings/js/node/CMakeLists.txt',
'src/bindings/js/node/package.json',
'src/bindings/js/node/package-lock.json']
- 'tools/openvino_dev/**/*'
'category: PDPD FE':
- 'src/frontends/paddle/**/*'
- 'tests/layer_tests/py_frontend_tests/test_paddle_frontend.py'
'category: POT':
- 'tools/pot/**/*'
'category: preprocessing':
- 'src/common/preprocessing/**/*'
@ -124,16 +113,12 @@
'category: Python API':
- 'src/bindings/python/**/*'
'category: JS API':
- 'src/bindings/js/**/*'
'category: samples':
- 'samples/**/*'
- 'thirdparty/zlib/**/*'
- 'thirdparty/gflags/**/*'
- 'thirdparty/json/**/*'
- 'thirdparty/cnpy/**/*'
- 'tests/samples_tests/smoke_tests/**/*'
'category: TEMPLATE':
- 'src/plugins/template/**/*'
@ -142,10 +127,6 @@
- 'src/frontends/tensorflow/**/*'
- 'src/frontends/tensorflow_common/**/*'
- 'tests/layer_tests/tensorflow_tests/**/*'
- 'tests/layer_tests/tensorflow2_keras_tests/**/*'
- 'tests/layer_tests/jax_tests/**/*'
- any: ['tests/model_hub_tests/**',
'!tests/model_hub_tests/pytorch/**/*']
'category: TFL FE':
- 'src/frontends/tensorflow_lite/**/*'
@ -156,32 +137,12 @@
- 'src/frontends/pytorch/**/*'
- 'tests/layer_tests/pytorch_tests/**/*'
- 'src/bindings/python/src/openvino/frontend/pytorch/**/*'
- 'tests/layer_tests/py_frontend_tests/test_torch_decoder.py'
- 'tests/layer_tests/py_frontend_tests/test_torch_frontend.py'
- any: ['tests/model_hub_tests/**',
'!tests/model_hub_tests/tensorflow/**/*']
'category: tools':
- any: ['tools/**',
'!tools/mo/**/*',
'!tools/ovc/**/*']
'!tools/pot/**/*',
'!tools/mo/**/*']
'category: transformations':
- 'src/common/transformations/**/*'
- 'src/common/offline_transformations/**/*'
- 'src/core/pattern/**/*'
- 'src/core/pass/**/*'
- 'src/core/include/openvino/pass/**/*'
'category: licensing':
- 'licensing/**/*'
- 'LICENSE'
'category: docker_env':
- '.dockerignore'
- 'install_build_dependencies.sh'
- '**/install_openvino_dependencies.sh'
- '.github/actions/handle_docker/**/*'
'category: dockerfiles':
- '.github/dockerfiles/**/*'

View File

@ -1,225 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
#!/usr/bin/env python3
from github import Github
from psycopg2 import sql
import os
import logging
import psycopg2
import dateutil
import argparse
def init_logger():
LOGLEVEL = os.environ.get('LOGLEVEL', 'INFO').upper()
logging.basicConfig(level=LOGLEVEL,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d-%Y %H:%M:%S')
def make_parser():
parser = argparse.ArgumentParser()
parser.add_argument('-r', '--repository-name', type=str, required=True,
help='Repository name in OWNER/REPOSITORY format')
parser.add_argument('--run-id', type=str, required=True,
help='Workflow Run ID')
return parser
def create_db_tables(conn, cur):
cur.execute('''CREATE TABLE IF NOT EXISTS workflow_runs(
id SERIAL PRIMARY KEY,
run_id BIGINT,
html_url TEXT,
name VARCHAR(255),
run_started_at TIMESTAMP,
created_at TIMESTAMP,
updated_at TIMESTAMP,
triggering_actor_login VARCHAR(255),
conclusion VARCHAR(25),
event VARCHAR(50),
run_attempt INT,
repository_full_name VARCHAR(255),
head_repository_full_name VARCHAR(255),
head_branch VARCHAR(255),
status VARCHAR(25),
display_title TEXT,
path TEXT,
total_duration_seconds INT
);
''')
cur.execute('''CREATE TABLE IF NOT EXISTS workflow_jobs(
id SERIAL PRIMARY KEY,
job_id BIGINT,
parent_run_id BIGINT,
html_url TEXT,
name VARCHAR(255),
created_at TIMESTAMP,
started_at TIMESTAMP,
completed_at TIMESTAMP,
queued_duration_seconds INT,
duration_seconds INT,
runner_name VARCHAR(255),
status VARCHAR(25),
conclusion VARCHAR(25),
head_branch VARCHAR(255),
run_attempt INT,
workflow_name TEXT
);
''')
cur.execute('''CREATE TABLE IF NOT EXISTS workflow_steps(
id SERIAL PRIMARY KEY,
parent_job_id BIGINT,
name VARCHAR(255),
conclusion VARCHAR(25),
number INT,
started_at TIMESTAMP,
completed_at TIMESTAMP,
duration_seconds INT
);
''')
conn.commit()
def main():
init_logger()
parser = make_parser()
args = parser.parse_args()
logger = logging.getLogger(__name__)
github_token = os.environ.get('GITHUB_TOKEN')
if not github_token:
raise ValueError('GITHUB_TOKEN environment variable is not set!')
run_id = args.run_id
repo_name = args.repository_name
# this should be specified in runner's env
db_username = os.environ.get('PGUSER')
db_password = os.environ.get('PGPASSWORD')
db_host = os.environ.get('PGHOST')
db_database = os.environ.get('PGDATABASE')
db_port = os.environ.get('PGPORT')
conn = psycopg2.connect(host=db_host,
port=db_port,
user=db_username,
password=db_password,
database=db_database)
# Create tables
cur = conn.cursor()
create_db_tables(conn, cur)
# Get the data
g = Github(github_token)
repo = g.get_repo(repo_name)
run = repo.get_workflow_run(int(run_id))
logger.info('Processing run ID %s - %s, URL: %s', run_id, run.name, run.html_url)
if run.status != 'completed':
logger.error('Run %s is not completed! Only completed runs should be in the database', run_id)
raise SystemExit(1)
# We rely on the following assumptions:
# - The workflow run is completed. When run.status != 'completed' we should not add it to the database
# theoretically the second attempt can be triggerred right after the completion of the first one
# or while the runner which executes this script is deploying
#
# - Job's queued duration equals "job.started_at - job.created_at" if started_at > created_at.
# Otherwise the job should not be added to the database
total_duration_seconds = round(run.timing().run_duration_ms / 1000)
workflow_data_query = sql.SQL('''INSERT INTO workflow_runs(
run_id, html_url, name,
run_started_at, created_at, updated_at, triggering_actor_login, conclusion,
event, run_attempt, repository_full_name, head_repository_full_name,
head_branch, status, display_title, path, total_duration_seconds)
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);
''')
workflow_query_args = (run_id, run.html_url, run.name,
run.run_started_at, run.created_at, run.updated_at,
run.raw_data['triggering_actor']['login'],
run.conclusion, run.event,
run.run_attempt, run.raw_data['repository']['full_name'],
run.raw_data['head_repository']['full_name'],
run.head_branch, run.status,
run.display_title, run.path, total_duration_seconds)
logger.debug('Workflow run query: %s', cur.mogrify(workflow_data_query, workflow_query_args))
cur.execute(workflow_data_query, workflow_query_args)
for job in run.jobs():
job_id = job.id
logger.info('Processing job %s, URL: %s', job.name, job.html_url)
queued_duration_seconds = 0
duration_seconds = 0
job_created_at_date = dateutil.parser.parse(job.raw_data['created_at'])
if job_created_at_date > job.started_at:
logger.warning('Skipping job %s of run %s - most likely a stub \
job created after workflow restart', job.name, run_id)
continue
queued_duration_timedelta = job.started_at - job_created_at_date
queued_duration_seconds = round(queued_duration_timedelta.total_seconds())
duration_timedelta = job.completed_at - job.started_at
duration_seconds = round(duration_timedelta.total_seconds())
job_data_query = sql.SQL('''
INSERT INTO workflow_jobs(
job_id, parent_run_id, html_url, name,
created_at, started_at, completed_at,
queued_duration_seconds, duration_seconds,
runner_name, status, conclusion, head_branch,
run_attempt, workflow_name
)
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);
''')
job_query_args = (job_id, run_id, job.html_url, job.name,
job.raw_data['created_at'], job.started_at, job.completed_at,
queued_duration_seconds, duration_seconds,
job.raw_data['runner_name'], job.status, job.conclusion,
job.raw_data['head_branch'], job.raw_data['run_attempt'], job.raw_data['workflow_name'])
logger.debug('Job query: %s', cur.mogrify(job_data_query, job_query_args))
cur.execute(job_data_query, job_query_args)
conn.commit()
for step in job.steps:
logger.info('Processing step %s', step.name)
# It seems like a GitHub bug, but
# sometimes there're steps "In progress"
# despite the workflow being completed
step_duration_seconds = 0
if step.status != 'completed':
logger.warning('Step %s isn\'t completed. Setting duration to zero.', step.name)
step_duration_seconds = 0
else:
step_duration_seconds_timedelta = step.completed_at - step.started_at
step_duration_seconds = round(step_duration_seconds_timedelta.total_seconds())
step_data_query = sql.SQL('''
INSERT INTO workflow_steps(
parent_job_id, name, conclusion,
number, started_at, completed_at,
duration_seconds)
VALUES (%s, %s, %s, %s, %s, %s, %s);
''')
step_query_args = (
job_id, step.name, step.conclusion,
step.number, step.started_at, step.completed_at,
step_duration_seconds)
logger.debug('Step query: %s', cur.mogrify(step_data_query, step_query_args))
cur.execute(step_data_query, step_query_args)
conn.commit()
cur.close()
conn.close()
g.close()
if __name__ == "__main__":
main()

View File

@ -1,75 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import argparse
import logging
import os
from enum import Enum
from github import Github, Auth
class ExternalPRLabels(str, Enum):
ExternalPR = 'ExternalPR'
ExternalIntelPR = 'ExternalIntelPR'
def get_arguments() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument(
'-r',
'--repository-name',
type=str,
required=True,
help='Repository name in the OWNER/REPOSITORY format',
)
parser.add_argument(
'--pr-number', type=int, required=True, help='PR number to label'
)
return parser.parse_args()
def init_logger():
logging.basicConfig(
level=os.environ.get('LOGLEVEL', 'INFO').upper(),
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d-%Y %H:%M:%S',
)
if __name__ == '__main__':
init_logger()
LOGGER = logging.getLogger('labeller')
args = get_arguments()
pr_number = args.pr_number
repository_name = args.repository_name
github = Github(auth=Auth.Token(token=os.environ.get('GITHUB_TOKEN')))
gh_repo = github.get_repo(full_name_or_id=repository_name)
pr = gh_repo.get_pull(number=pr_number)
LOGGER.info(f'CONTEXT: PR #{pr_number}. USER: {pr.user.login}. ALL PR LABELS: {list(pr.get_labels())}')
if not gh_repo.has_in_collaborators(pr.user.login):
LOGGER.info(f'THE {pr.user.login} IS NOT A COLLABORATOR')
for label in pr.get_labels():
if label.name in (ExternalPRLabels.ExternalPR, ExternalPRLabels.ExternalIntelPR):
LOGGER.info(f'THE PR ALREADY HAS THE "{label.name}" LABEL')
break
else:
is_intel_user = bool(pr.user.email and pr.user.email.endswith('@intel.com'))
label_to_add: str = ExternalPRLabels.ExternalIntelPR.name if is_intel_user else ExternalPRLabels.ExternalPR.name
pr.add_to_labels(label_to_add)
LOGGER.info(f'THE "{label_to_add}" LABEL WAS ADDED TO THE PR')
else:
LOGGER.info(
f'THE {pr.user.login} IS A COLLABORATOR, NO NEED TO ADD THE "External" LABEL'
)
github.close()

View File

@ -1 +0,0 @@
python-dateutil==2.9.0.post0

View File

@ -1,28 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import argparse
from pathlib import Path
def get_arguments() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument('-r', '--repository-name',
type=str,
required=True,
help='Repository name in the OWNER/REPOSITORY format')
parser.add_argument('--run-id',
type=int,
required=True,
help='Workflow Run ID')
parser.add_argument('--errors-to-look-for-file',
type=Path,
required=False,
help='.json file with the errors to look for in logs',
default=Path(__file__).resolve().parent.joinpath('errors_to_look_for.json'))
parser.add_argument('--dry-run',
required=False,
action='store_true',
help='Whether to run in dry mode and not actually retrigger the pipeline'
' and only collect and analyze logs')
return parser.parse_args()

View File

@ -1,20 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import logging
import os
GITHUB_TOKEN = os.environ.get('GITHUB_TOKEN')
def init_logger():
LOGLEVEL = os.environ.get('LOGLEVEL', 'INFO').upper()
logging.basicConfig(level=LOGLEVEL,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d-%Y %H:%M:%S')
init_logger()
LOGGER = logging.getLogger('rerunner')

View File

@ -1,62 +0,0 @@
[
{
"error_text": "This is a problem related to network connectivity",
"ticket": 135929
},
{
"error_text": "Unable to make request",
"ticket": 135715
},
{
"error_text": "GnuTLS recv error",
"ticket": 131918
},
{
"error_text": "Connection was reset",
"ticket": 131818
},
{
"error_text": "Failed to connect to github.com",
"ticket": 131657
},
{
"error_text": "Could not resolve host: github.com",
"ticket": 131546
},
{
"error_text": "retrieving gpg key timed out",
"ticket": 131538
},
{
"error_text": "Retry limit has been reached for chunk",
"ticket": 131537
},
{
"error_text": "fatal error: downloading",
"ticket": 131424
},
{
"error_text": "Failure when receiving data from the peer",
"ticket": 137121
},
{
"error_text": "Network is unreachable",
"ticket": 130955
},
{
"error_text": "connection timed out",
"ticket": 130955
},
{
"error_text": "The requested URL returned error: 500",
"ticket": 139384
},
{
"error_text": "Unable to fetch some archives",
"ticket": 130965
},
{
"error_text": "status_string: \"Timeout was reached\"",
"ticket": 142653
}
]

View File

@ -1,137 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import json
import re
import tempfile
from pathlib import Path
from typing import TypedDict
from zipfile import ZipFile
from workflow_rerun.constants import LOGGER
class LogFile(TypedDict):
file_name: str
path: Path
class ErrorData(TypedDict):
error_text: str
ticket: int
class LogAnalyzer:
def __init__(self,
path_to_log_archive: Path,
path_to_errors_file: Path) -> None:
self._path_to_log_archive = path_to_log_archive
self._path_to_errors_file = path_to_errors_file
self._errors_to_look_for: list[ErrorData] = []
self._collect_errors_to_look_for()
self._log_dir = tempfile.TemporaryDirectory().name
self._log_files: list[LogFile] = []
self._collect_log_files()
all_txt_log_files_pretty = '\n'.join(map(lambda item: str(item['path']), self._log_files))
LOGGER.info(f'ALL .txt LOG FILES: \n{all_txt_log_files_pretty}')
self.found_matching_error = False
self.found_error_ticket = None
def _collect_errors_to_look_for(self) -> None:
with open(file=self._path_to_errors_file,
mode='r',
encoding='utf-8') as errors_file:
errors_data = json.load(errors_file)
for error_data in errors_data:
self._errors_to_look_for.append(
ErrorData(error_text=error_data['error_text'],
ticket=error_data['ticket'])
)
def _collect_log_files(self) -> None:
"""
Collects the .txt log files from the log archive
The GitHub Actions pipeline logs archive should have the following structure:
> Job_name_0
> step_name_0.txt
> step_name_1.txt
...
> Job_name_1
> step_name_0.txt
> step_name_1.txt
...
> Job_name_2
...
...
We need to only analyze the `*.txt` files
"""
with ZipFile(file=self._path_to_log_archive,
mode='r') as zip_file:
zip_file.extractall(self._log_dir)
for _file in Path(self._log_dir).iterdir():
if _file.is_dir():
for log_file in _file.iterdir():
self._log_files.append(LogFile(file_name=log_file.name,
path=log_file.resolve()))
def _is_error_in_log(self,
error_to_look_for: str,
log_file_path: Path) -> bool:
"""
Searches for the error in the provided log
"""
error_to_look_for = self._clean_up_string(error_to_look_for)
with open(file=log_file_path,
mode='r',
encoding='utf-8') as log_file:
for line in log_file:
if error_to_look_for in self._clean_up_string(line):
return True
return False
@staticmethod
def _clean_up_string(string: str) -> str:
"""
Replaces special characters with spaces in the string, strips it from leading and following spaces,
and lowers it
for "Could not resolve host: github.com" returns "could not resolve host github com"
This cleanup is applied to both errors to look for and logs themselves for matching
"""
return re.sub(r'[^A-Za-z0-9]+', ' ', string).lower().strip()
def analyze(self) -> None:
"""
Iterates over the known errors and tries to find them in the collected log files
"""
for error in self._errors_to_look_for:
LOGGER.info(f'LOOKING FOR "{error["error_text"]}" ERROR...')
for log_file in self._log_files:
if self._is_error_in_log(error_to_look_for=error['error_text'],
log_file_path=log_file['path']):
LOGGER.info(f'FOUND "{error["error_text"]}" ERROR IN {log_file["path"]}. TICKET: {error["ticket"]}')
self.found_matching_error = True
self.found_error_ticket = error['ticket']
return
if __name__ == '__main__':
# Usage example
log_analyzer = LogAnalyzer(path_to_log_archive=Path('/tmp/logs/log.zip'),
path_to_errors_file=Path('/tmp/errors_to_look_for.json'))
log_analyzer.analyze()
if log_analyzer.found_matching_error:
print('found matching error, see logs above')

View File

@ -1,24 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
from pathlib import Path
import requests
from github.WorkflowRun import WorkflowRun
from workflow_rerun.constants import GITHUB_TOKEN, LOGGER
def collect_logs_for_run(run: WorkflowRun,
log_archive_path: Path) -> Path:
"""
Collects log archive for a pipeline
"""
with open(file=log_archive_path,
mode='wb') as log_archive:
LOGGER.info(f'STARTED LOG COLLECTION FOR {run.id} IN {log_archive_path}')
# PyGitHub does not expose the "/repos/{owner}/{repo}/actions/runs/{run_id}/logs" endpoint so we have to use requests
log_archive.write(requests.get(url=run.logs_url,
headers={'Authorization': f'Bearer {GITHUB_TOKEN}'}).content)
LOGGER.info(f'COLLECTED LOGS FOR {run.id} IN {log_archive_path}')
return log_archive_path

View File

@ -1,71 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import os
import sys
import tempfile
from pathlib import Path
from github import Github, Auth
from workflow_rerun.argument_parser import get_arguments
from workflow_rerun.constants import GITHUB_TOKEN, LOGGER
from workflow_rerun.log_analyzer import LogAnalyzer
from workflow_rerun.log_collector import collect_logs_for_run
if __name__ == '__main__':
args = get_arguments()
run_id = args.run_id
repository_name = args.repository_name
errors_file = args.errors_to_look_for_file
is_dry_run = args.dry_run
if is_dry_run:
LOGGER.info('RUNNING IN DRY RUN MODE. IF ERROR WILL BE FOUND, WILL NOT RETRIGGER')
github = Github(auth=Auth.Token(token=GITHUB_TOKEN))
gh_repo = github.get_repo(full_name_or_id=repository_name)
run = gh_repo.get_workflow_run(id_=run_id)
LOGGER.info(f'CHECKING IF RERUN IS NEEDED FOR {run.html_url} RUN IN {repository_name}.')
# Check if the run has already been retriggered
# we do not want to fall into a loop with retriggers
if run.run_attempt > 1:
LOGGER.info(f'THERE ARE {run.run_attempt} ATTEMPTS ALREADY. NOT CHECKING LOGS AND NOT RETRIGGERING. EXITING')
sys.exit(0)
log_archive_path = Path(tempfile.NamedTemporaryFile(suffix='.zip').name)
collect_logs_for_run(
run=run,
log_archive_path=log_archive_path,
)
log_analyzer = LogAnalyzer(
path_to_log_archive=log_archive_path,
path_to_errors_file=errors_file
)
log_analyzer.analyze()
if log_analyzer.found_matching_error:
LOGGER.info(f'FOUND MATCHING ERROR, RETRIGGERING {run.html_url}')
if is_dry_run:
LOGGER.info(f'RUNNING IN DRY RUN MODE, NOT RETRIGGERING, EXITING')
sys.exit(0)
status = run.rerun()
if status:
LOGGER.info(f'RUN RETRIGGERED SUCCESSFULLY: {run.html_url}')
else:
LOGGER.info(f'RUN WAS NOT RETRIGGERED, SEE ABOVE')
# Needed to run a step after for statistics
with open(file=os.environ['GITHUB_ENV'],
mode='a') as fh:
fh.write('PIPELINE_RETRIGGERED=true\n')
fh.write(f'FOUND_ERROR_TICKET={log_analyzer.found_error_ticket}\n')
# "status" is True (which is 1) if everything is ok, False (which is 0) otherwise
sys.exit(not status)
else:
LOGGER.info(f'NO ERROR WAS FOUND, NOT RETRIGGERING')

View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:894d636bcf156a7f3fae09f3c1d61df6b3db89117a917a3079995805c29115b3
size 89247

View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3f094a737d7ea40dba8d3fb13493275cae243d08e5f1dabce90c316c951a6ac2
size 52047

View File

@ -1,55 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""
Integration tests
"""
import unittest
from pathlib import Path
from github import Github, Auth
import os
import tempfile
from workflow_rerun.log_analyzer import LogAnalyzer
from workflow_rerun.log_collector import collect_logs_for_run
class IntegrationTest(unittest.TestCase):
"""
A class for testing integration between LogAnalyzer and log_collection
"""
def setUp(self) -> None:
print(f'\nIn test: "{self._testMethodName}"', flush=True)
self._cwd = Path(__file__).parent
self.errors_to_look_for_file = self._cwd.parent.joinpath(
'errors_to_look_for.json'
)
self.github = Github(auth=Auth.Token(token=os.environ.get('GITHUB_TOKEN')))
self.gh_repo = self.github.get_repo(full_name_or_id='openvinotoolkit/openvino')
# Even if we use "failure" for status we cannot guarantee logs containing any of the known error
# So these tests use the logs of the most recent successfull pipeline
self.wf_run = self.gh_repo.get_workflow_runs(status='success')[0]
print(f'Workflow run for testing: {self.wf_run}', flush=True)
def test_log_collection_and_analysis(self) -> None:
"""
Ensure logs collected by collect_logs_for_run are analyzed by LogAnalyzer
"""
log_archive_path = Path(tempfile.NamedTemporaryFile(suffix='.zip').name)
collect_logs_for_run(run=self.wf_run,
log_archive_path=log_archive_path)
analyzer = LogAnalyzer(
path_to_log_archive=log_archive_path,
path_to_errors_file=self.errors_to_look_for_file,
)
analyzer.analyze()
self.assertFalse(analyzer.found_matching_error)
def tearDown(self) -> None:
self.github.close()

View File

@ -1,104 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""
LogAnalyzer tests
"""
import unittest
from pathlib import Path
from workflow_rerun.log_analyzer import LogAnalyzer
class LogAnalyzerTest(unittest.TestCase):
"""
A class for testing LogAnalyzer
"""
def setUp(self) -> None:
print(f'\nIn test: "{self._testMethodName}"', flush=True)
self._cwd = Path(__file__).parent
self.log_archive_with_error = self._cwd.joinpath("data").joinpath(
'log_archive_with_error.zip'
)
self.log_archive_wo_error = self._cwd.joinpath("data").joinpath(
'log_archive_wo_error.zip'
)
self.errors_to_look_for_file = self._cwd.parent.joinpath(
'errors_to_look_for.json'
)
def test_log_analyzer_instantiation(self) -> None:
"""
Ensure LogAnalyzer is instantiated correctly.
"""
analyzer = LogAnalyzer(
path_to_log_archive=self.log_archive_wo_error,
path_to_errors_file=self.errors_to_look_for_file,
)
self.assertTrue(
hasattr(analyzer, '_errors_to_look_for'),
'Analyzer should have _errors_to_look_for',
)
self.assertTrue(
hasattr(analyzer, '_log_files'), 'Analyzer should have _log_files'
)
for error_data in analyzer._errors_to_look_for:
self.assertTrue(
error_data['error_text'], 'Each error_data should have text'
)
self.assertTrue(error_data['ticket'], 'Each error_data should have ticket')
for log_file in analyzer._log_files:
self.assertTrue(
log_file['file_name'], 'Each log_file should have file_name'
)
self.assertTrue(log_file['path'], 'Each log_file should have path')
def test_string_cleanup(self) -> None:
"""
Ensure log cleanup function returns correct results
"""
analyzer = LogAnalyzer(
path_to_log_archive=self.log_archive_wo_error,
path_to_errors_file=self.errors_to_look_for_file,
)
data = (
'Connection was reset',
'Failed to connect to github.com',
'Could not resolve host: github.com',
)
expected = (
'connection was reset',
'failed to connect to github com',
'could not resolve host github com',
)
for input_str, expected_str in zip(data, expected):
self.assertEqual(analyzer._clean_up_string(string=input_str), expected_str)
def test_analyzer_with_error(self) -> None:
"""
Ensure LogAnalyzer can find an error
"""
analyzer = LogAnalyzer(
path_to_log_archive=self.log_archive_with_error,
path_to_errors_file=self.errors_to_look_for_file,
)
analyzer.analyze()
self.assertTrue(analyzer.found_matching_error)
def test_analyzer_wo_error(self) -> None:
"""
Ensure LogAnalyzer does not find an error in the log files w/o errors
"""
analyzer = LogAnalyzer(
path_to_log_archive=self.log_archive_wo_error,
path_to_errors_file=self.errors_to_look_for_file,
)
analyzer.analyze()
self.assertFalse(analyzer.found_matching_error)

View File

@ -1,41 +0,0 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""
log collector tests
"""
import os
import unittest
import tempfile
from pathlib import Path
from github import Github, Auth
from workflow_rerun.log_collector import collect_logs_for_run
class LogCollectorTest(unittest.TestCase):
"""
A class for testing log collection
"""
def setUp(self) -> None:
print(f'\nIn test: "{self._testMethodName}"', flush=True)
self._cwd = Path(__file__).parent
self.github = Github(auth=Auth.Token(token=os.environ.get('GITHUB_TOKEN')))
self.gh_repo = self.github.get_repo(full_name_or_id='openvinotoolkit/openvino')
# Use the logs of the most recent successfull pipeline
self.wf_run = self.gh_repo.get_workflow_runs(status='success')[0]
print(f'Workflow run for testing: {self.wf_run}', flush=True)
def test_log_collection(self) -> None:
"""
Ensure log collection is working
"""
log_archive_path = Path(tempfile.NamedTemporaryFile(suffix='.zip').name)
collect_logs_for_run(run=self.wf_run, log_archive_path=log_archive_path)
self.assertTrue(Path(log_archive_path).exists())
def tearDown(self) -> None:
self.github.close()

View File

@ -2,47 +2,32 @@ name: Android ARM64 with vcpkg
on:
workflow_dispatch:
pull_request:
merge_group:
paths-ignore:
- '**/docs/**'
- 'docs/**'
- '**/**.md'
- '**.md'
- '**/layer_tests_summary/**'
- '**/conformance/**'
push:
paths-ignore:
- '**/docs/**'
- 'docs/**'
- '**/**.md'
- '**.md'
- '**/layer_tests_summary/**'
- '**/conformance/**'
branches:
# - master
- 'releases/**'
- master
concurrency:
# github.ref is not unique in post-commit
group: ${{ github.event_name == 'push' && github.run_id || github.ref }}-android-arm64-vcpkg
cancel-in-progress: true
permissions: read-all
jobs:
Smart_CI:
runs-on: ubuntu-latest
outputs:
affected_components: "${{ steps.smart_ci.outputs.affected_components }}"
skip_workflow: "${{ steps.smart_ci.outputs.skip_workflow }}"
steps:
- name: checkout action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
sparse-checkout: .github/actions/smart-ci
- name: Get affected components
id: smart_ci
uses: ./.github/actions/smart-ci
with:
repository: ${{ github.repository }}
pr: ${{ github.event.number }}
commit_sha: ${{ github.sha }}
ref_name: ${{ github.ref_name }}
component_pattern: "category: (.*)"
repo_token: ${{ secrets.GITHUB_TOKEN }}
skip_when_only_listed_labels_set: 'docs'
skip_when_only_listed_files_changed: '*.md,*.rst,*.png,*.jpg,*.svg,*/layer_tests_summary/*,*/conformance/*'
Build:
needs: Smart_CI
timeout-minutes: 150
timeout-minutes: 15
defaults:
run:
shell: bash
@ -50,17 +35,13 @@ jobs:
container:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount:/mount
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
CMAKE_GENERATOR: 'Ninja'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
SCCACHE_ERROR_LOG: /__w/openvino/sccache_log.txt
SCCACHE_LOG: warn
OPENVINO_REPO: '/__w/openvino/openvino/openvino'
VCPKG_ROOT: '/__w/openvino/openvino/vcpkg'
BUILD_DIR: '/__w/openvino/openvino/build'
@ -71,13 +52,12 @@ jobs:
VCPKG_DEFAULT_BINARY_CACHE: '/mount/caches/ccache/android_arm64/vcpkg_cache'
VCPKG_FORCE_SYSTEM_BINARIES: '1'
SCCACHE_AZURE_KEY_PREFIX: android_arm64
if: "!needs.smart_ci.outputs.skip_workflow"
steps:
- name: Install git
run: apt-get update && apt-get install --assume-yes --no-install-recommends git ca-certificates
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
path: 'openvino'
@ -89,14 +69,13 @@ jobs:
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/json
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/gtest
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/gflags
git submodule update --init -- ${OPENVINO_REPO}/thirdparty/open_model_zoo
popd
- name: Clone vcpkg
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
repository: 'microsoft/vcpkg'
# Keep in sync with <root>/vcpkg.json <builtin-baseline>
ref: '7ba0ba7334c3346e7eee1e049ba85da193a8d821'
path: 'vcpkg'
fetch-depth: '0'
@ -117,13 +96,13 @@ jobs:
apt --assume-yes install ccache scons ninja-build build-essential python3-pip
# vcpkg requires cmake 3.19 or later
python3 -m pip install -U pip cmake~=3.28.0
python3 -m pip install -U pip cmake
# vcpkg's tool dependencies
apt --assume-yes install curl zip unzip tar
# vcpkg 'python3' port dependencies
apt --assume-yes install autoconf libtool autoconf-archive
# vcpkg tree of dependencies require extra packages
apt --assume-yes install pkgconf linux-libc-dev
apt --assume-yes install pkg-config linux-libc-dev
# Install Android SDK, NDK and Tools
apt -y --no-install-recommends install unzip wget default-jdk
@ -132,9 +111,9 @@ jobs:
echo "yes" | ./cmdline-tools/bin/sdkmanager --sdk_root=${ANDROID_TOOLS} --install "ndk-bundle" "platform-tools" "platforms;android-${{ env.ANDROID_SDK_VERSION }}"
- name: Install sccache
uses: mozilla-actions/sccache-action@2e7f9ec7921547d4b46598398ca573513895d0bd # v0.0.4
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.7.5"
version: "v0.5.4"
#
# Build
@ -172,35 +151,10 @@ jobs:
-B ${BUILD_DIR}
- name: Clean ccache stats
run: ${SCCACHE_PATH} --zero-stats
run: sccache --zero-stats
- name: Cmake - build
run: cmake --build ${BUILD_DIR} --parallel
- name: Show ccache stats
run: ${SCCACHE_PATH} --show-stats
#
# Upload build logs
#
- name: Upload build logs
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: build_logs
path: ${{ env.SCCACHE_ERROR_LOG }}
if-no-files-found: 'ignore'
Overall_Status:
name: ci/gha_overall_status_android
needs: [Smart_CI, Build]
if: ${{ always() }}
runs-on: ubuntu-latest
steps:
- name: Check status of all jobs
if: >-
${{
contains(needs.*.result, 'failure') ||
contains(needs.*.result, 'cancelled')
}}
run: exit 1

View File

@ -6,8 +6,6 @@ on:
- created
- edited
permissions: read-all
jobs:
take-issue:
name: Take issue
@ -17,7 +15,7 @@ jobs:
timeout-minutes: 10
steps:
- name: take an issue
uses: bdougie/take-action@1439165ac45a7461c2d89a59952cd7d941964b87 # v1.6.1
uses: bdougie/take-action@v1.6.1
with:
message: Thank you for looking into this issue! Please let us know if you have any questions or require any help.
issueCurrentlyAssignedMessage: Thanks for being interested in this issue. It looks like this ticket is already assigned to a contributor. Please communicate with the assigned contributor to confirm the status of the issue.

View File

@ -1,34 +1,32 @@
name: Documentation
on:
pull_request:
merge_group:
env:
DOXY_VER: '1.9.6'
DOXYREST_VER: '2.1.3'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
cancel-in-progress: true
permissions: read-all
jobs:
Build_Doc:
runs-on: ubuntu-20.04
steps:
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
submodules: 'true'
lfs: 'true'
- name: Install apt-get dependencies
uses: awalsh128/cache-apt-pkgs-action@a6c3917cc929dd0345bfb2d3feaf9101823370ad # v1.4.2
uses: awalsh128/cache-apt-pkgs-action@v1.3.1
with:
packages: graphviz texlive liblua5.2-0 libclang1-9 libclang-cpp9
version: 3.0
- uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
- uses: actions/setup-python@v4
id: cp310
with:
python-version: '3.10'
@ -41,28 +39,35 @@ jobs:
run: |
python3 -m pip install -r docs/requirements.txt
(cd docs/openvino_sphinx_theme && python3 setup.py install)
python3 -m pip install docs/openvino_custom_sphinx_sitemap
- name: Download and install doxygen
- name: Download and install doxygen && doxyrest
run: |
# install doxyrest
wget https://github.com/vovkos/doxyrest/releases/download/doxyrest-$DOXYREST_VER/doxyrest-$DOXYREST_VER-linux-amd64.tar.xz
tar -xf doxyrest-$DOXYREST_VER-linux-amd64.tar.xz
echo "$(pwd)/doxyrest-$DOXYREST_VER-linux-amd64/bin/" >> $GITHUB_PATH
# install doxygen
wget https://www.doxygen.nl/files/doxygen-$DOXY_VER.linux.bin.tar.gz
tar -xzf doxygen-$DOXY_VER.linux.bin.tar.gz
echo "$(pwd)/doxygen-$DOXY_VER/bin/" >> $GITHUB_PATH
- name: Build docs
run: |
rm -rf build && mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=ON
cmake --build . --target sphinx_docs
- name: CMake configure
run: cmake -DENABLE_DOCS=ON -B build
- name: Cache documentation
id: cache_sphinx_docs
uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2
uses: actions/cache@v3
with:
path: build/docs/_build/.doctrees
key: sphinx-docs-cache
- name: Get number of CPU cores
uses: SimenB/github-actions-cpu-cores@v2
id: cpu-cores
- name: Build docs
run: cmake --build build --target sphinx_docs --parallel ${{ steps.cpu-cores.outputs.count }}
- name: Archive docs HTML
run: (cd build/docs && zip -r openvino_docs_html.zip _build)
@ -71,31 +76,40 @@ jobs:
PR_NUMBER=$(echo $GITHUB_REF | awk 'BEGIN { FS = "/" } ; { print $3 }')
echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_ENV
- name: 'Upload doxygen.log'
uses: actions/upload-artifact@v3
with:
name: doxygen_build_log_${{ env.PR_NUMBER }}.log
path: build/docs/doxygen.log
- name: 'Upload sphinx.log'
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: sphinx_build_log_${{ env.PR_NUMBER }}.log
path: build/docs/sphinx.log
- name: 'Upload docs html'
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_docs_html_${{ env.PR_NUMBER }}.zip
path: build/docs/openvino_docs_html.zip
- name: Run Pytest
run: |
pytest --sphinx="./build/docs/sphinx.log" \
--suppress-warnings="./docs/scripts/tests/suppress_warnings.txt" \
pytest --doxygen="./build/docs/doxygen.log" \
--include_pot \
--sphinx="./build/docs/sphinx.log" \
--suppress-warnings="./docs/suppress_warnings.txt" \
--confcutdir="./docs/scripts/tests/" \
--html="./build/docs/_artifacts/doc-generation.html" \
--sphinx-strip="$(pwd)/build/docs/sphinx_source" \
--xfail="./docs/scripts/tests/xfail.txt" \
--doxygen-strip="$(pwd)" \
--sphinx-strip="$(pwd)/build/docs/rst" \
--doxygen-xfail="./docs/doxygen-xfail.txt" \
--self-contained-html ./docs/scripts/tests/test_docs.py
- name: 'Upload test results'
if: failure()
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_docs_pytest
path: build/docs/_artifacts/

View File

@ -1,14 +1,12 @@
name: PR Commits
on: [pull_request]
permissions: read-all
jobs:
Checks:
runs-on: ubuntu-22.04
steps:
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
- name: Install dependencies
run: python3 -m pip install -r ./.github/github_org_control/requirements.txt

View File

@ -1,57 +0,0 @@
name: Cleanup caches
on:
workflow_dispatch:
schedule:
# at 00:00 on the 1st day of every month
- cron: '0 0 1 * *'
permissions: read-all
jobs:
Cleanup_PIP:
runs-on: aks-linux-2-cores-8gb
container:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount:/mount
env:
PIP_CACHE_PATH: /mount/caches/pip
steps:
- name: Pre-Collecting Cache Info
run: |
echo "Cache info: "
du -h -d2 ${PIP_CACHE_PATH}
- name: Cleanup cache
run: |
echo "Delete cache files if they have not been used in over 30 days"
[ ! -z "${PIP_CACHE_PATH}" ] && find ${PIP_CACHE_PATH} ! -type d -atime +30 -delete
- name: Post-Collecting Cache Info
run: |
echo "Cache info: "
du -h -d2 ${PIP_CACHE_PATH}
Cleanup_CCACHE:
runs-on: aks-linux-2-cores-8gb
container:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount:/mount
env:
CCACHE_PATH: /mount/caches/ccache/ubuntu22_riscv64
steps:
- name: Checkout cach action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
sparse-checkout: .github/actions/cache
- name: Cleanup ccache
uses: ./.github/actions/cache/cleanup
with:
cache-size: 10
max-cache-size: 50
cache-path: ${{ env.CCACHE_PATH }}
key: '.'

31
.github/workflows/cleanup_pip_cache.yml vendored Normal file
View File

@ -0,0 +1,31 @@
name: Cleanup PIP caches
on:
schedule:
# at 00:00 on the 1st day of every month
- cron: '0 0 1 * *'
jobs:
Cleanup_PIP_Caches:
runs-on: aks-linux-2-cores
container:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount/caches:/mount/caches
env:
PIP_CACHE_PATH: /mount/caches/pip
steps:
- name: Pre-Collecting Cache Info
run: |
echo "Cache info: "
du -h -d2 ${PIP_CACHE_PATH}
- name: Cleanup cache
run: |
echo "Delete cache files if they have not been used in over 30 days"
[ ! -z "${PIP_CACHE_PATH}" ] && find ${PIP_CACHE_PATH} ! -type d -atime +30 -delete
- name: Post-Collecting Cache Info
run: |
echo "Cache info: "
du -h -d2 ${PIP_CACHE_PATH}

View File

@ -16,8 +16,6 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions: read-all
jobs:
Build:
strategy:
@ -27,23 +25,23 @@ jobs:
runs-on: ${{ matrix.os }}
steps:
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
submodules: 'true'
- name: Install OpenCL
uses: awalsh128/cache-apt-pkgs-action@a6c3917cc929dd0345bfb2d3feaf9101823370ad # v1.4.2
uses: awalsh128/cache-apt-pkgs-action@v1.3.1
if: runner.os == 'Linux'
with:
packages: ocl-icd-opencl-dev opencl-headers
version: 3.0
- name: Install scons
if: runner.os == 'macOS'
run: brew install scons
- name: CMake configure
run: cmake -DCMAKE_BUILD_TYPE=Release -DTHREADING=SEQ -B build
- name: Get number of CPU cores
uses: SimenB/github-actions-cpu-cores@v2
id: cpu-cores
- name: Build snippets
run: cmake --build build --target openvino_docs_snippets --parallel
run: cmake --build build --target ie_docs_snippets --parallel ${{ steps.cpu-cores.outputs.count }}

View File

@ -5,22 +5,27 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions: read-all
jobs:
clang-format:
runs-on: ubuntu-22.04
runs-on: ubuntu-20.04
permissions:
pull-requests: write
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@v4
with:
submodules: 'true'
- name: Install clang-format-15
- name: Install clang-format-9
run: |
sudo apt update
sudo apt --assume-yes install clang-format-15
sudo apt --assume-yes install clang-format-9
- name: Install dependencies
run: |
python3 -m pip install --upgrade pip
python3 -m pip install -r ./src/bindings/python/requirements.txt
# Add for -DENABLE_PYTHON=ON, no cython
python3 -m pip install -r ./src/bindings/python/src/compatibility/openvino/requirements-dev.txt
# Run cmake with -DENABLE_PROFILING_ITT=ON -DSELECTIVE_BUILD=COLLECT in order to enable codestyle check for ITT collector
- name: CMake configure
@ -31,7 +36,7 @@ jobs:
- name: suggester / clang-format
if: startsWith(github.event_name, 'pull_request')
uses: reviewdog/action-suggester@9e1cd88b79ba3c0023c94e44accd72344f032093 # v1.13.0
uses: reviewdog/action-suggester@v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
level: warning
@ -42,7 +47,7 @@ jobs:
permissions:
pull-requests: write
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@v4
with:
submodules: 'true'
@ -60,7 +65,7 @@ jobs:
# always provide suggestions even for skipped scripts in ov_shellcheck tagret
- name: ShellCheck action
if: always()
uses: reviewdog/action-shellcheck@6e3a862f231c6895fbd335b70adef8f9243d5762 # v1.21.0
uses: reviewdog/action-shellcheck@v1
with:
level: style
reporter: github-pr-review
@ -73,7 +78,7 @@ jobs:
NamingConventionCheck:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@v4
with:
submodules: 'true'

View File

@ -5,8 +5,6 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions: read-all
jobs:
Coverage:
runs-on: ${{ matrix.config.os }}
@ -18,19 +16,19 @@ jobs:
steps:
- name: Setup python
uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
uses: actions/setup-python@v4
with:
python-version: '3.10.10'
architecture: 'x64'
- name: Setup ccache
uses: hendrikmuhs/ccache-action@c92f40bee50034e84c763e33b317c77adaa81c92 # v1.2.13
uses: hendrikmuhs/ccache-action@v1.2
with:
max-size: 50G
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
submodules: 'true'
@ -42,6 +40,9 @@ jobs:
python3 -m pip install --upgrade pip
python3 -m pip install -r ${{ github.workspace }}/src/bindings/python/wheel/requirements-dev.txt
python3 -m pip install -r ${{ github.workspace }}/src/bindings/python/requirements.txt
# For running Python API tests
python3 -m pip install -r ${{ github.workspace }}/src/bindings/python/src/compatibility/openvino/requirements-dev.txt
# For running Paddle frontend unit tests
python3 -m pip install -r ${{ github.workspace }}/src/frontends/paddle/tests/requirements.txt
# For running ONNX frontend unit tests
@ -56,8 +57,12 @@ jobs:
python3 -m pip install -r ${{ github.workspace }}/tools/mo/requirements_tf2.txt
python3 -m pip install -r ${{ github.workspace }}/tools/mo/requirements_dev.txt
- name: Get number of CPU cores
uses: SimenB/github-actions-cpu-cores@v2
id: cpu-cores
- name: Build OpenVINO with CMake
uses: ashutoshvarma/action-cmake-build@ade188313bc7eaa6f14349569a64d8bc716342ff # master
uses: ashutoshvarma/action-cmake-build@master
with:
build-dir: ${{ github.workspace }}/build
cc: ${{ matrix.config.cc }}
@ -79,6 +84,7 @@ jobs:
-DCMAKE_CXX_LINKER_LAUNCHER=ccache
-DENABLE_SYSTEM_SNAPPY=ON
build-type: Release
parallel: ${{ steps.cpu-cores.outputs.count }}
- name: Install wheel packages
run: cmake -DCOMPONENT=python_wheels -DCMAKE_INSTALL_PREFIX=${{ github.workspace }}/install_pkg -P '${{ github.workspace }}/build/cmake_install.cmake'
@ -99,7 +105,7 @@ jobs:
run: ${{ github.workspace }}/bin/intel64/Release/ov_proxy_plugin_tests
- name: Run OV Hetero Func tests
run: ${{ github.workspace }}/bin/intel64/Release/ov_hetero_func_tests --gtest_filter="*smoke*"
run: ${{ github.workspace }}/bin/intel64/Release/ov_hetero_func_tests
- name: Run IR frontend tests
run: ${{ github.workspace }}/bin/intel64/Release/ov_ir_frontend_tests
@ -114,7 +120,7 @@ jobs:
run: ${{ github.workspace }}/bin/intel64/Release/ov_tensorflow_frontend_tests --gtest_filter=-*IE_GPU*
- name: Build coverage with CMake
uses: ashutoshvarma/action-cmake-build@ade188313bc7eaa6f14349569a64d8bc716342ff # master
uses: ashutoshvarma/action-cmake-build@master
with:
build-dir: ${{ github.workspace }}/coverage
cc: ${{ matrix.config.cc }}
@ -126,6 +132,7 @@ jobs:
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
-DCMAKE_C_LINKER_LAUNCHER=ccache
-DCMAKE_CXX_LINKER_LAUNCHER=ccache
parallel: ${{ steps.cpu-cores.outputs.count }}
- name: Print info
@ -137,6 +144,6 @@ jobs:
lcov --capture --directory ${{ github.workspace }}/. --output-file coverage.info
genhtml coverage.info --output-directory coverage-report
- name: Collect coverage
uses: codecov/codecov-action@125fc84a9a348dbcf27191600683ec096ec9021c # v4.4.1
uses: codecov/codecov-action@v4
with:
verbose: true

View File

@ -1,155 +0,0 @@
name: Coverity (Ubuntu 20.04, Python 3.11)
on:
workflow_dispatch:
inputs:
openvinoRef:
description: 'Branch, tag or commit hash to clone openvino from. Taken from event context if not set'
type: string
schedule:
# run daily at 00:00
- cron: '0 0 * * *'
concurrency:
# github.ref is not unique in post-commit
group: ${{ github.event_name == 'push' && github.run_id || github.ref }}-linux-coverity
cancel-in-progress: true
permissions: read-all
env:
PIP_CACHE_PATH: /mount/caches/pip/linux
PYTHON_VERSION: '3.11'
jobs:
Build:
timeout-minutes: 150
defaults:
run:
shell: bash
runs-on: aks-linux-16-cores-32gb
container:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
CMAKE_BUILD_TYPE: 'Release'
CMAKE_GENERATOR: 'Ninja Multi-Config'
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
OPENVINO_CONTRIB_REPO: /__w/openvino/openvino/openvino_contrib
BUILD_DIR: /__w/openvino/openvino/openvino_build
COVERITY_TOOL_DIR: /__w/openvino/openvino/coverity_tool
steps:
- name: Install git
run: |
apt-get update
apt-get install --assume-yes --no-install-recommends git ca-certificates
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
path: ${{ env.OPENVINO_REPO }}
submodules: 'true'
ref: ${{ inputs.openvinoRef }}
- name: Clone OpenVINO Contrib
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
repository: 'openvinotoolkit/openvino_contrib'
path: ${{ env.OPENVINO_CONTRIB_REPO }}
submodules: 'true'
ref: 'master'
#
# Dependencies
#
- name: Install build dependencies
run: |
bash ${OPENVINO_REPO}/install_build_dependencies.sh
# default-jdk - Java API
apt install --assume-yes --no-install-recommends default-jdk
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: ./openvino/.github/actions/setup_python
with:
version: ${{ env.PYTHON_VERSION }}
pip-cache-path: ${{ env.PIP_CACHE_PATH }}
should-setup-pip-paths: 'true'
self-hosted-runner: 'true'
#
# Build
#
- name: CMake configure - OpenVINO
run: |
cmake \
-G "${{ env.CMAKE_GENERATOR }}" \
-DENABLE_CPPLINT=OFF \
-DENABLE_STRICT_DEPENDENCIES=OFF \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-DENABLE_FASTER_BUILD=OFF \
-S ${OPENVINO_REPO} \
-B ${BUILD_DIR}
- name: Install Coverity tool
run: |
rm -rf ${COVERITY_TOOL_DIR} && mkdir -p ${COVERITY_TOOL_DIR}
pushd ${COVERITY_TOOL_DIR}
wget https://scan.coverity.com/download/linux64 --progress=bar:force:noscroll --post-data "token=${{ secrets.COVERITY_TOKEN }}&project=openvino" -O coverity_tool.tgz
tar xvf coverity_tool.tgz && rm coverity_tool.tgz
popd
- name: Cmake build - OpenVINO with Coverity
run: ${COVERITY_TOOL_DIR}/cov-analysis*/bin/cov-build --dir ${BUILD_DIR}/cov-int cmake --build ${BUILD_DIR} --parallel --config ${{ env.CMAKE_BUILD_TYPE }}
- name: Pack Artefacts
run: |
pushd ${BUILD_DIR}
tar -C ${BUILD_DIR} -czvf openvino.tgz cov-int
popd
- name: Submit artefacts
run: |
apt-get update && apt-get install -y curl jq
pushd ${BUILD_DIR}
curl -X POST -d token=${{ secrets.COVERITY_TOKEN }} \
-d email=${{ secrets.COVERITY_USER }} \
-d file_name="openvino.tgz" \
-d version="${{ github.sha }}" \
-d description="https://github.com/openvinotoolkit/openvino/runs/${{ github.run_number }}" \
https://scan.coverity.com/projects/21921/builds/init | tee response
upload_url=$(jq -r '.url' response)
build_id=$(jq -r '.build_id' response)
curl -X PUT \
--header 'Content-Type: application/json' \
--upload-file openvino.tgz \
$upload_url
curl -X PUT \
-d token=${{ secrets.COVERITY_TOKEN }} \
https://scan.coverity.com/projects/21921/builds/$build_id/enqueue
popd
- name: Show Coverity configure logs
continue-on-error: true
run: ${COVERITY_TOOL_DIR}/cov-analysis*/bin/cov-configure -c ${COVERITY_TOOL_DIR}/cov-analysis-linux64-2023.6.2/config/coverity_config.xml -lscc text
- name: Upload Coverity build log
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: coverity_logs
path: ${{ env.BUILD_DIR }}/cov-int/build-log.txt
if-no-files-found: 'error'
- name: Upload Coverity build archive
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: coverity_archive
path: ${{ env.BUILD_DIR }}/openvino.tgz
if-no-files-found: 'error'

View File

@ -1,18 +1,17 @@
name: 'Dependency Review'
on: [pull_request, merge_group]
on: [pull_request]
permissions: read-all
permissions:
contents: read
jobs:
dependency-review:
runs-on: ubuntu-latest
steps:
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
- name: Dependency Review
uses: actions/dependency-review-action@72eb03d02c7872a771aacd928f3123ac62ad6d3a # v4.3.3
uses: actions/dependency-review-action@v3
with:
config-file: './.github/dependency_review.yml'
base-ref: ${{ github.pull_request.base.sha || github.event.merge_group.base_ref }}
head-ref: ${{ github.pull_request.head.sha || github.ref }}

View File

@ -2,10 +2,23 @@ name: Fedora (RHEL), Python 3.9
on:
workflow_dispatch:
pull_request:
merge_group:
paths-ignore:
- '**/docs/**'
- 'docs/**'
- '**/**.md'
- '**.md'
- '**/layer_tests_summary/**'
- '**/conformance/**'
push:
paths-ignore:
- '**/docs/**'
- 'docs/**'
- '**/**.md'
- '**.md'
- '**/layer_tests_summary/**'
- '**/conformance/**'
branches:
# - master
- master
- 'releases/**'
concurrency:
@ -13,35 +26,8 @@ concurrency:
group: ${{ github.event_name == 'push' && github.run_id || github.ref }}-fedora33
cancel-in-progress: true
permissions: read-all
jobs:
Smart_CI:
runs-on: ubuntu-latest
outputs:
affected_components: "${{ steps.smart_ci.outputs.affected_components }}"
skip_workflow: "${{ steps.smart_ci.outputs.skip_workflow }}"
steps:
- name: checkout action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
sparse-checkout: .github/actions/smart-ci
- name: Get affected components
id: smart_ci
uses: ./.github/actions/smart-ci
with:
repository: ${{ github.repository }}
pr: ${{ github.event.number }}
commit_sha: ${{ github.sha }}
ref_name: ${{ github.ref_name }}
component_pattern: "category: (.*)"
repo_token: ${{ secrets.GITHUB_TOKEN }}
skip_when_only_listed_labels_set: 'docs'
skip_when_only_listed_files_changed: '*.md,*.rst,*.png,*.jpg,*.svg,*/layer_tests_summary/*,*/conformance/*'
Build:
needs: Smart_CI
timeout-minutes: 150
defaults:
run:
@ -50,30 +36,25 @@ jobs:
container:
image: fedora:33
volumes:
- /mount:/mount
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
CMAKE_BUILD_TYPE: 'Release'
CMAKE_GENERATOR: 'Ninja'
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
SCCACHE_IGNORE_SERVER_IO_ERROR: 1
SCCACHE_SERVER_PORT: 35555
SCCACHE_ERROR_LOG: /__w/openvino/sccache_log.txt
SCCACHE_LOG: warn
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
INSTALL_DIR: /__w/openvino/openvino/openvino_install
INSTALL_TEST_DIR: /__w/openvino/openvino/tests_install
BUILD_DIR: /__w/openvino/openvino/openvino_build
SCCACHE_AZURE_KEY_PREFIX: fedora33_x86_64_Release
if: "!needs.smart_ci.outputs.skip_workflow"
steps:
- name: Install git
run: yum update -y && yum install -y git
- name: Clone OpenVINO
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
uses: actions/checkout@v4
with:
path: ${{ env.OPENVINO_REPO }}
submodules: 'true'
@ -93,15 +74,16 @@ jobs:
run: bash ${OPENVINO_REPO}/install_build_dependencies.sh
- name: Install sccache
uses: mozilla-actions/sccache-action@2e7f9ec7921547d4b46598398ca573513895d0bd # v0.0.4
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.7.5"
version: "v0.5.4"
- name: Install python dependencies
run: |
python3 -m pip install -U pip
# For Python API: build and wheel packaging
python3 -m pip install -r ${OPENVINO_REPO}/src/bindings/python/wheel/requirements-dev.txt
python3 -m pip install -r ${OPENVINO_REPO}/src/bindings/python/src/compatibility/openvino/requirements-dev.txt
# For running ONNX frontend unit tests
python3 -m pip install --force-reinstall -r ${OPENVINO_REPO}/src/frontends/onnx/tests/requirements.txt
@ -127,6 +109,7 @@ jobs:
-DENABLE_NCC_STYLE=OFF \
-DENABLE_TESTS=ON \
-DENABLE_STRICT_DEPENDENCIES=OFF \
-DENABLE_SYSTEM_TBB=ON \
-DENABLE_SYSTEM_OPENCL=ON \
-DENABLE_PYTHON_PACKAGING=ON \
-DCPACK_GENERATOR=TGZ \
@ -161,27 +144,18 @@ jobs:
- name: Build RPM packages
run: |
cmake -UTBB* \
-DCPACK_GENERATOR=RPM \
-DENABLE_SYSTEM_TBB=ON \
cmake -DCPACK_GENERATOR=RPM \
-DENABLE_TESTS=OFF \
${BUILD_DIR}
cmake --build ${BUILD_DIR} --parallel --target package --verbose
#
# Upload build artifacts and logs
# Upload build artifacts
#
- name: Upload build logs
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: build_logs
path: ${{ env.SCCACHE_ERROR_LOG }}
if-no-files-found: 'ignore'
- name: Upload openvino package
if: ${{ always() }}
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_package
path: ${{ env.BUILD_DIR }}/openvino_package.tar.gz
@ -189,7 +163,7 @@ jobs:
- name: Upload openvino RPM packages
if: ${{ always() }}
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_rpm_packages
path: ${{ env.BUILD_DIR }}/*.rpm
@ -197,7 +171,7 @@ jobs:
- name: Upload openvino tests package
if: ${{ always() }}
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
uses: actions/upload-artifact@v3
with:
name: openvino_tests
path: ${{ env.BUILD_DIR }}/openvino_tests.tar.gz
@ -205,7 +179,7 @@ jobs:
RPM_Packages:
needs: Build
timeout-minutes: 10
timeout-minutes: 5
defaults:
run:
shell: bash
@ -217,7 +191,7 @@ jobs:
steps:
- name: Download OpenVINO RPM packages
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
uses: actions/download-artifact@v3
with:
name: openvino_rpm_packages
path: ${{ env.RPM_PACKAGES_DIR }}
@ -250,24 +224,10 @@ jobs:
python3 /usr/share/openvino/samples/python/hello_query_device/hello_query_device.py
python3 -c 'from openvino import Core; Core().get_property("CPU", "AVAILABLE_DEVICES")'
python3 -c 'from openvino import Core; Core().get_property("GPU", "AVAILABLE_DEVICES")'
python3 -c 'from openvino import Core; Core().get_property("AUTO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("MULTI", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("HETERO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("BATCH", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("AUTO", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("MULTI", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("HETERO", "SUPPORTED_METRICS")'
python3 -c 'from openvino import Core; Core().get_property("BATCH", "SUPPORTED_METRICS")'
python3 -c 'from openvino.frontend import FrontEndManager; assert len(FrontEndManager().get_available_front_ends()) == 6'
benchmark_app --help
ovc --help
Overall_Status:
name: ci/gha_overall_status_fedora
needs: [Smart_CI, Build, RPM_Packages]
if: ${{ always() }}
runs-on: ubuntu-latest
steps:
- name: Check status of all jobs
if: >-
${{
contains(needs.*.result, 'failure') ||
contains(needs.*.result, 'cancelled')
}}
run: exit 1

View File

@ -5,13 +5,11 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions: read-all
jobs:
Check_Files_Size:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@v4
- name: git ls-tree
run: git ls-tree -r -t -l --full-name HEAD | sort -n -r -k 4

View File

@ -1,124 +0,0 @@
name: CPU functional tests
on:
workflow_call:
inputs:
runner:
description: 'Machine on which the tests would run'
type: string
required: true
image:
description: 'Docker image in which the tests would run'
type: string
required: false
default: null
permissions: read-all
jobs:
CPU_Functional_Tests:
name: CPU functional tests
timeout-minutes: 30
runs-on: ${{ inputs.runner }}
container:
image: ${{ inputs.image }}
defaults:
run:
shell: bash
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
PARALLEL_TEST_SCRIPT: ${{ github.workspace }}/install/tests/functional_test_utils/layer_tests_summary/run_parallel.py
PARALLEL_TEST_CACHE: ${{ github.workspace }}/install/tests/test_cache.lst
steps:
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
run: |
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
echo "PARALLEL_TEST_SCRIPT=$GITHUB_WORKSPACE/install/tests/functional_test_utils/layer_tests_summary/run_parallel.py" >> "$GITHUB_ENV"
echo "PARALLEL_TEST_CACHE=$GITHUB_WORKSPACE/install/tests/test_cache.lst" >> "$GITHUB_ENV"
- name: Extract OpenVINO packages
run: |
pushd $INSTALL_DIR
tar -xzf openvino_package.tar.gz -C $INSTALL_DIR
popd
pushd $INSTALL_TEST_DIR
tar -xzf openvino_tests.tar.gz -C $INSTALL_DIR
popd
- name: Fetch setup_python action
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
sparse-checkout: |
.github/actions/setup_python/action.yml
sparse-checkout-cone-mode: false
path: 'openvino'
- name: Setup Python 3.11
uses: ./openvino/.github/actions/setup_python
with:
version: '3.11'
should-setup-pip-paths: 'false'
self-hosted-runner: ${{ runner.os == 'Linux' }}
- name: Install python dependencies for run_parallel.py
run: python3 -m pip install -r ${INSTALL_TEST_DIR}/functional_test_utils/layer_tests_summary/requirements.txt
- name: Restore tests execution time
uses: actions/cache/restore@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2
with:
path: ${{ env.PARALLEL_TEST_CACHE }}
key: ${{ runner.os }}-${{ runner.arch }}-tests-functional-cpu-stamp-${{ github.sha }}
restore-keys: |
${{ runner.os }}-${{ runner.arch }}-tests-functional-cpu-stamp
- name: Intel CPU plugin func tests (parallel)
run: |
# Needed as the Linux CC does not require setupvars to work
if [[ -f "${INSTALL_DIR}/setupvars.sh" ]]; then
source ${INSTALL_DIR}/setupvars.sh
fi
# Needed as ze_loader.so is under INSTALL_TEST_DIR
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${INSTALL_TEST_DIR}
python3 ${PARALLEL_TEST_SCRIPT} -e ${INSTALL_TEST_DIR}/ov_cpu_func_tests -c ${PARALLEL_TEST_CACHE} -w ${INSTALL_TEST_DIR} -s suite -rf 0 -- --gtest_print_time=1 --gtest_filter=*smoke*
timeout-minutes: 25
- name: Save tests execution time
uses: actions/cache/save@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2
if: github.ref_name == 'master'
with:
path: ${{ env.PARALLEL_TEST_CACHE }}
key: ${{ runner.os }}-${{ runner.arch }}-tests-functional-cpu-stamp-${{ github.sha }}
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: test-results-functional-cpu
path: |
${{ env.INSTALL_TEST_DIR }}/temp/*.log
${{ env.INSTALL_TEST_DIR }}/logs/*.log
${{ env.INSTALL_TEST_DIR }}/logs/failed/*.log
${{ env.INSTALL_TEST_DIR }}/logs/crashed/*.log
${{ env.INSTALL_TEST_DIR }}/logs/hanged/*.log
${{ env.INSTALL_TEST_DIR }}/logs/interapted/*.log
${{ env.INSTALL_TEST_DIR }}/logs/hash_table.csv
${{ env.PARALLEL_TEST_CACHE }}
if-no-files-found: 'error'

View File

@ -1,263 +0,0 @@
name: C++ Unit Tests
on:
workflow_call:
inputs:
runner:
description: 'Machine on which the tests would run'
type: string
required: true
image:
description: 'Docker image in which the tests would run'
type: string
required: false
default: null
affected-components:
description: 'Components that are affected by changes in the commit defined by the Smart CI Action'
type: string
required: true
permissions: read-all
jobs:
CXX_Unit_Tests:
name: C++ unit tests
timeout-minutes: 30
runs-on: ${{ inputs.runner }}
container:
image: ${{ inputs.image }}
defaults:
run:
shell: bash
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
steps:
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
run: |
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
- name: Extract OpenVINO packages
run: |
pushd $INSTALL_DIR
tar -xzf openvino_package.tar.gz -C $INSTALL_DIR
popd
pushd $INSTALL_TEST_DIR
tar -xzf openvino_tests.tar.gz -C $INSTALL_DIR
popd
#
# Tests
#
- name: OpenVINO Core Unit Tests
if: fromJSON(inputs.affected-components).Core.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_core_unit_tests --gtest_print_time=1 --gtest_filter=-*IE_GPU* \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-OVCoreUT.xml
- name: OpenVINO Inference Functional Tests
if: fromJSON(inputs.affected-components).inference.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_inference_functional_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-InferenceFunc.xml
- name: OpenVINO Inference Unit Tests
if: fromJSON(inputs.affected-components).inference.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_inference_unit_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-InferenceUnit.xml
- name: Low Precision Transformations Tests
if: fromJSON(inputs.affected-components).LP_transformations.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_lp_transformations_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-LpTransformations.xml
- name: OpenVINO Conditional compilation tests
if: fromJSON(inputs.affected-components).Core.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_conditional_compilation_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ConditionalCompilation.xml
- name: IR frontend tests
if: fromJSON(inputs.affected-components).IR_FE.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_ir_frontend_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-IRFrontend.xml
- name: PaddlePaddle frontend tests
if: fromJSON(inputs.affected-components).PDPD_FE.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/paddle_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-PaddleTests.xml
- name: ONNX frontend tests
if: ${{ fromJSON(inputs.affected-components).ONNX_FE.test && runner.arch != 'ARM64' }} # Ticket for macOS ARM64: 122663, for Linux ARM64: 126280
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_onnx_frontend_tests --gtest_print_time=1 \
--gtest_filter=-*IE_GPU* \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ONNXFrontend.xml
- name: TensorFlow Common frontend tests
if: fromJSON(inputs.affected-components).TF_FE.test ||
fromJSON(inputs.affected-components).TFL_FE.test &&
(runner.os != 'macOS' && runner.arch != 'ARM64')
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_tensorflow_common_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-TensorFlowCommonFrontend.xml
- name: TensorFlow frontend tests
if: fromJSON(inputs.affected-components).TF_FE.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_tensorflow_frontend_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-TensorFlowFrontend.xml
- name: TensorFlow Lite frontend tests
if: fromJSON(inputs.affected-components).TFL_FE.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_tensorflow_lite_frontend_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-TensorFlowLiteFrontend.xml
- name: Transformations func tests
if: ${{ fromJSON(inputs.affected-components).transformations.test && runner.arch != 'ARM64' }} # Ticket: 126281
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_transformations_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-Transformations.xml
- name: Common test utils tests
if: ${{ runner.os != 'macOS' }} # Ticket: 134469
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_util_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-CommonUtilTests.xml
- name: Snippets func tests
if: fromJSON(inputs.affected-components).CPU.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_snippets_func_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-SnippetsFuncTests.xml
- name: CPU plugin unit tests
if: fromJSON(inputs.affected-components).CPU.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_cpu_unit_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-CPUUnitTests.xml
- name: ov_subgraphs_dumper_tests tests
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_subgraphs_dumper_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ov_subgraphs_dumper_tests.xml
- name: Template OpImpl tests
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_op_conformance_tests --gtest_print_time=1 --device=TEMPLATE --gtest_filter=*OpImpl*\
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-OpImplTests.xml
- name: AUTO unit tests
if: fromJSON(inputs.affected-components).AUTO.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_auto_unit_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ov_auto_unit_tests.xml
- name: AUTO func Tests
if: fromJSON(inputs.affected-components).AUTO.test
run: |
source ${{ env.INSTALL_DIR }}/setupvars.sh
${{ env.INSTALL_TEST_DIR }}/ov_auto_func_tests --gtest_print_time=1 \
--gtest_output=xml:${{ env.INSTALL_TEST_DIR }}/TEST-ov_auto_func_tests.xml
- name: Template plugin func tests
if: fromJSON(inputs.affected-components).TEMPLATE.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_template_func_tests --gtest_print_time=1 \
--gtest_filter=*smoke* \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-TemplateFuncTests.xml
- name: OV utils unit tests
if: ${{ runner.os != 'macOS' }} # Ticket: 134469
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_util_tests --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ov_util_tests.xml
- name: OpenVINO C API tests
if: fromJSON(inputs.affected-components).C_API.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_capi_test --gtest_print_time=1 \
--gtest_output=xml:${INSTALL_TEST_DIR}/TEST-OpenVINOCAPITests.xml
- name: AutoBatch unit tests
if: fromJSON(inputs.affected-components).AUTO_BATCH.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_auto_batch_unit_tests --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ov_auto_batch_unit_tests.xml
- name: AutoBatch func tests
if: fromJSON(inputs.affected-components).AUTO_BATCH.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_auto_batch_func_tests --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ov_auto_batch_func_tests.xml --gtest_filter="*smoke*"
- name: Proxy Plugin func tests
if: fromJSON(inputs.affected-components).PROXY.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_proxy_plugin_tests --gtest_print_time=1 --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-OVProxyTests.xml
- name: Hetero unit tests
if: fromJSON(inputs.affected-components).HETERO.test
run: |
source ${{ env.INSTALL_DIR }}/setupvars.sh
${{ env.INSTALL_TEST_DIR }}/ov_hetero_unit_tests --gtest_print_time=1 --gtest_output=xml:${{ env.INSTALL_TEST_DIR }}/TEST-OVHeteroUnitTests.xml
- name: Hetero func tests
if: fromJSON(inputs.affected-components).HETERO.test
run: |
source ${INSTALL_DIR}/setupvars.sh
${INSTALL_TEST_DIR}/ov_hetero_func_tests --gtest_print_time=1 --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-OVHeteroFuncTests.xml --gtest_filter="*smoke*"
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: ${{ !cancelled() }}
with:
name: test-results-cpp
path: ${{ env.INSTALL_TEST_DIR }}/TEST*.xml
if-no-files-found: 'warn'

View File

@ -1,93 +0,0 @@
name: Debian Packages
on:
workflow_call:
inputs:
runner:
description: 'Machine on which the tests would run'
type: string
required: true
image:
description: 'Docker image in which the tests would run'
type: string
required: false
default: null
permissions: read-all
jobs:
Debian_Packages:
name: Debian Packages
runs-on: ${{ inputs.runner }}
container:
image: ${{ inputs.image }}
defaults:
run:
shell: bash
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
DEBIAN_PACKAGES_DIR: ${{ github.workspace }}/packages
steps:
- name: Set apt retries
if: runner.os == 'Linux'
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries
- name: Download OpenVINO debian packages
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_debian_packages
path: ${{ env.DEBIAN_PACKAGES_DIR }}
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
run: echo "DEBIAN_PACKAGES_DIR=$GITHUB_WORKSPACE/packages" >> "$GITHUB_ENV"
- name: Install debian packages & check conflicts
run: |
apt-get update -y
if [[ "${{ runner.arch }}" == "X64" ]]; then
# Install debian packages from previous release
apt-get install --no-install-recommends -y gnupg wget ca-certificates
wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
echo "deb https://apt.repos.intel.com/openvino/2024 ubuntu20 main" | tee /etc/apt/sources.list.d/intel-openvino-2024.list
apt-get update -y
apt-get install -y openvino
fi
# install our local one and make sure the conflicts are resolved
apt-get install --no-install-recommends -y dpkg-dev
dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz
echo "deb [trusted=yes] file:${DEBIAN_PACKAGES_DIR} ./" | tee /etc/apt/sources.list.d/openvino-local.list
apt-get update -y
apt-get install openvino -y
working-directory: ${{ env.DEBIAN_PACKAGES_DIR }}
- name: Test debian packages
run: |
/usr/share/openvino/samples/cpp/build_samples.sh
/usr/share/openvino/samples/c/build_samples.sh
[[ "${{ runner.arch }}" == "X64" ]] && path_by_arch="intel64" || path_by_arch="aarch64"
~/openvino_cpp_samples_build/$path_by_arch/Release/hello_query_device
# check integrity of OpenVINO Python API installation
apt-get install python3-pip -y
python3 -m pip check
python3 /usr/share/openvino/samples/python/hello_query_device/hello_query_device.py
python3 -c 'from openvino import Core; Core().get_property("CPU", "AVAILABLE_DEVICES")'
if [[ "${{ runner.arch }}" == "X64" ]]; then
python3 -c 'from openvino import Core; Core().get_property("GPU", "AVAILABLE_DEVICES")'
fi
python3 -c 'from openvino import Core; Core().get_property("AUTO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("MULTI", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("HETERO", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino import Core; Core().get_property("BATCH", "SUPPORTED_PROPERTIES")'
python3 -c 'from openvino.frontend import FrontEndManager; assert len(FrontEndManager().get_available_front_ends()) == 6'
benchmark_app --help
opt_in_out --help
ovc --help

View File

@ -1,134 +0,0 @@
name: GPU
on:
workflow_call:
inputs:
test_type:
description: 'Type of tests to execute'
type: string
required: true
device:
description: 'Device name (igpu or dgpu)'
type: string
required: true
runner:
description: 'Runner labels by which the runner will be chosen. Example: [ "self-hosted", "igpu" ]'
type: string
required: true
container:
description: 'JSON to be converted to the value of the "container" configuration for the job'
type: string
required: false
default: '{"image": null}'
jobs:
GPU:
timeout-minutes: 80
runs-on: ${{ fromJSON(inputs.runner) }}
container: ${{ fromJSON(inputs.container) }}
defaults:
run:
shell: bash
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
GTEST_PARALLEL_SCRIPT: ${{ github.workspace }}/gtest_parallel.py
steps:
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: 'openvino_package'
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: 'openvino_tests'
path: ${{ env.INSTALL_TEST_DIR }}
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
run: |
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
echo "GTEST_PARALLEL_SCRIPT=$GITHUB_WORKSPACE/gtest_parallel.py" >> "$GITHUB_ENV"
- name: Extract OpenVINO packages
run: |
pushd $INSTALL_DIR
tar -xzf openvino_package.tar.gz -C $INSTALL_DIR
popd
pushd $INSTALL_TEST_DIR
tar -xzf openvino_tests.tar.gz -C $INSTALL_DIR
popd
- name: Install dependencies (Linux)
run: |
$INSTALL_DIR/install_dependencies/install_openvino_dependencies.sh -c=core -c=dev -c=gpu -y
apt-get update && apt-get install -y wget software-properties-common ca-certificates gpg-agent tzdata clinfo
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
TZ: "Europe/London" # to prevent tzdata from waiting user input
- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Get gtest-parallel script
run: wget https://raw.githubusercontent.com/google/gtest-parallel/master/gtest_parallel.py
- name: Install compute runtime drivers
run: |
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.15985.7/intel-igc-core_1.0.15985.7_amd64.deb
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.15985.7/intel-igc-opencl_1.0.15985.7_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/intel-level-zero-gpu-dbgsym_1.3.28454.6_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/intel-level-zero-gpu_1.3.28454.6_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/intel-opencl-icd-dbgsym_24.05.28454.6_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/intel-opencl-icd_24.05.28454.6_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.05.28454.6/libigdgmm12_22.3.11_amd64.deb
dpkg -i *.deb
- name: Install media & display runtimes
if: ${{ inputs.device == 'dgpu' }}
run: |
apt-get update && apt-get install -y \
libegl-mesa0 libegl1-mesa libegl1-mesa-dev libgbm1 libgl1-mesa-dev libgl1-mesa-dri \
libglapi-mesa libgles2-mesa-dev libglx-mesa0 libigdgmm11 libxatracker2 mesa-va-drivers \
mesa-vdpau-drivers mesa-vulkan-drivers va-driver-all
- name: Verify devices
run: clinfo
#
# Tests
#
- name: OpenVINO GPU ${{ inputs.test_type }} Tests
id: run_tests
run: |
source ${INSTALL_DIR}/setupvars.sh
TEST_RESULTS_DIR="${{ inputs.device }}_${{ inputs.test_type }}_tests"
echo "test_results_dir=$TEST_RESULTS_DIR" >> $GITHUB_OUTPUT
rm -rf ${INSTALL_TEST_DIR}/${TEST_RESULTS_DIR} && mkdir -p ${INSTALL_TEST_DIR}/${TEST_RESULTS_DIR}
test_filter=''
if [[ "${{ inputs.test_type }}" == "unit" ]]; then
# Ticket: 138018
test_filter='-*scatter_nd_update_gpu.dynamic_padded_output*:*border_gpu.basic_zero_input*:*bicubic_zeros_no_align_data1x1*:*bicubic_border_align_batches*:*bilinear_zeros_no_align_data1x1*:*non_zero_gpu.empty_input*:*mark_shape_of_subgraphs.concat_with_empty_tensor_inputs*:*concat_cpu_impl.dynamic_4d_f*:*border_gpu.basic_zero_input_dynamic*:*network_test.model_with_empty_input_is_not_dynamic*:*bicubic_zeros_align_data1x1*'
else
test_filter='*smoke*'
fi
python3 ${GTEST_PARALLEL_SCRIPT} ${INSTALL_TEST_DIR}/ov_gpu_${{ inputs.test_type }}_tests --dump_json_test_results=${INSTALL_TEST_DIR}/${TEST_RESULTS_DIR}/ov_gpu_${{ inputs.test_type }}_tests.json -- --report_unique_name --gtest_filter=$test_filter
- name: Upload Test Results
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
if: always()
with:
name: test-results-${{ inputs.test_type }}-${{ inputs.device }}
path: ${{ env.INSTALL_TEST_DIR }}/${{ steps.run_tests.outputs.test_results_dir }}
if-no-files-found: 'error'

View File

@ -1,101 +0,0 @@
name: ONNX Models tests
on:
workflow_call:
inputs:
runner:
description: 'Machine on which the tests would run'
type: string
required: true
container:
description: 'JSON to be converted to the value of the "container" configuration for the job'
type: string
required: false
default: '{"image": null}'
permissions: read-all
jobs:
ONNX_Models_tests:
name: ONNX Models tests
timeout-minutes: 60
runs-on: ${{ inputs.runner }}
container: ${{ fromJSON(inputs.container) }}
defaults:
run:
shell: bash
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
OPENVINO_REPO: ${{ github.workspace }}/openvino
INSTALL_DIR: ${{ github.workspace }}/install
INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
ONNX_MODELS_PATH: ${{ github.workspace }}/onnx_test_models
# instead of using static MODELS_SHARE_PATH
# choose one of the replicas dynamically instead
# depending on GITHUB_RUN_NUMBER variable
NUMBER_OF_REPLICAS: 2
ONNX_MODEL_ZOO_SHA: "5faef4c33eba0395177850e1e31c4a6a9e634c82"
if: ${{ github.event_name != 'merge_group' }}
steps:
- name: Download OpenVINO package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}
- name: Download OpenVINO tests package
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: openvino_tests
path: ${{ env.INSTALL_TEST_DIR }}
# Needed as ${{ github.workspace }} is not working correctly when using Docker
- name: Setup Variables
run: |
echo "OPENVINO_REPO=$GITHUB_WORKSPACE/openvino" >> "$GITHUB_ENV"
echo "INSTALL_DIR=$GITHUB_WORKSPACE/install" >> "$GITHUB_ENV"
echo "INSTALL_TEST_DIR=$GITHUB_WORKSPACE/install/tests" >> "$GITHUB_ENV"
echo "MODELS_SHARE_PATH=/mount/testdata$((GITHUB_RUN_NUMBER % NUMBER_OF_REPLICAS))" >> "$GITHUB_ENV"
echo $MODELS_SHARE_PATH
- name: Extract OpenVINO packages
run: |
pushd ${INSTALL_DIR}
tar -xzf openvino_package.tar.gz -C ${INSTALL_DIR}
popd
pushd ${INSTALL_TEST_DIR}
tar -xzf openvino_tests.tar.gz -C ${INSTALL_DIR}
popd
- name: Fetch model_zoo_preprocess script
uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
sparse-checkout: |
src/frontends/onnx/tests/tests_python/model_zoo_preprocess.sh
sparse-checkout-cone-mode: false
path: 'openvino'
- name: Update Models
run: bash ${OPENVINO_REPO}/src/frontends/onnx/tests/tests_python/model_zoo_preprocess.sh -d ${MODELS_SHARE_PATH} -o -s "${{ env.ONNX_MODEL_ZOO_SHA }}"
- name: Install OpenVINO Python wheels
run: |
# Install the core OV wheel
python3 -m pip install ${INSTALL_DIR}/tools/openvino-*.whl
extras_to_install="onnx"
# Find and install OV dev wheel
pushd ${INSTALL_DIR}/tools
ov_dev_wheel_name=$(find . -name 'openvino_dev*.whl')
python3 -m pip install $ov_dev_wheel_name[$extras_to_install]
popd
- name: Install Python tests dependencies
run: |
# To enable pytest parallel features
python3 -m pip install pytest-xdist[psutil] pytest-forked
- name: ONNX Models Tests
run: python3 -m pytest --backend="CPU" --model_zoo_dir="${MODELS_SHARE_PATH}" ${INSTALL_TEST_DIR}/onnx/tests/tests_python/test_zoo_models.py -v -n auto --forked -k 'not _cuda' --model_zoo_xfail

Some files were not shown because too many files have changed in this diff Show More