Commit Graph

43 Commits

Author SHA1 Message Date
Alexey Suhov a79830cb55
Update year to 2022 in copyright notice (#9755) 2022-01-19 01:07:49 +03:00
Victor Kuznetsov e212a1ac83
[TimeTests] Disable VPU option, rename cache option (#9114)
* add performance hint to time infer

* Update time tests with options

* fix mistake in help

* update test conf with GPU models

* move ie configs from load plugin timer
2021-12-17 13:06:56 +03:00
Victor Kuznetsov 0f1c545a19
[Time tests] Change work with perf_hint (#9090)
* add performance hint to time infer

* upd_time_tests
2021-12-09 10:29:55 +03:00
Victor Kuznetsov 55f64a505a
Expand time tests to cover CPU caching feature (#8938)
* add performance hint to time infer

* add options for vpu, cpu cache, perf hint

* change VPUX condition

* add perf hint verify

* upd perf hint usage. set as non required
2021-12-08 15:22:44 +03:00
Victor Kuznetsov 8a46ead91b
add pytest-timeout (#8352) 2021-11-02 13:44:02 +03:00
Victor Kuznetsov 445292b079
change pr type (#8143) 2021-10-26 11:29:57 +03:00
Victor Kuznetsov 6d9a7a108e
Stress, Memory tests update to comply with new OMZ tools layout (#7886)
* work with omz: change hardcoded paths for memory_tests, fix paths for memcheck get_testdata.py script

* change paths to omz tools

* revert mo changes

* remove mo from arguments

* remove extra var from cmd string

* remove is_db_used check

* add timeout to proc_exec

* merge master
2021-10-13 10:42:00 +03:00
Alina Kladieva fda3f5d237
[requirements] Set TF to 2.5.0 (#6620)
* [requirements] Set TF to 2.5.0

* [ngraph|paddlepaddle|requirements] Gast to 0.4.0

(required for TensorFlow 2.5)

* [ stress_tests/scripts/requirements] Upd versions

* [time_tests/test_runner/requirements] Upd versions

* [ngraph|paddle|requirements] Paddle strict major

* Gast version to 0.3.3

* Update requirements_dev.txt
2021-09-14 18:05:18 +03:00
Ilya Sharikov f44369ce4e
Updated requirements (#7397) 2021-09-13 08:50:23 +03:00
Victor Kuznetsov 4d377901bf
Move all utils to common folder (#7303)
* move all utils to common folder

* add utils install in CMakeLists.txt

* revert plugin utils

* add log info: test

* add logging info: test

* test imports

* change imports: test

* fix file name
2021-09-07 23:02:10 +03:00
Victor Kuznetsov 1f4664162a
install necessary dirs for tests (#7044)
* install necessary dirs to tests

* rem RUNTIME from install step

* fix paths

* fix install paths

* fix install paths: add destination dirs

* add pandas

* fix requirements conflict - change pytest version to ~5

* remove comment from requirements.txt

* upd numpy version
2021-08-24 00:39:12 +03:00
Ilya Sharikov 26c8b29b35
Added missing modules for timetests (#6896) 2021-08-03 12:19:40 +03:00
Vitaliy Urusovskij 31907e51e9
Fixed several issues in time_tests (#6481)
* Fixed return in `run_timetest`

* Fixed `parse_stats()` argument type notation

* Pass message from `timetest` exe to test error msg to be uploaded to DB
2021-07-01 16:37:23 +03:00
Victor Kuznetsov a1eb5b1fd5
Change target branch field (fix null) (#6433) 2021-06-30 12:53:17 +03:00
Victor Kuznetsov 089f76cc7f
Add `branch` field to submit to DB in MemCheck/TimeTests (#6073) 2021-06-11 12:57:01 +03:00
Victor Kuznetsov 4790c79eb4
Add time_tests dir path to sys.path (#5498) 2021-05-06 14:57:50 +03:00
Olesya Martinyuk d16c215fde
Add CPU info to time tests (#5451) 2021-05-04 14:49:10 +03:00
Vitaliy Urusovskij db2de2fb19
Fix time_tests runs when models_cache enabled (#5265)
* Set ``"additionalProperties": true` to enable 'use_model_cache' in time_tests

* Limit tests scope in desktop_test_config_cache.yml
2021-04-17 00:12:08 +03:00
Vitaliy Urusovskij c852634055
Remove comparison with references from time_tests as unused (#5248)
* Remove comparison with references from time_tests as unused

* Remove `--dump_refs` from time_tests as unused
2021-04-15 22:24:45 +03:00
Vitaliy Urusovskij a70d13f9e2
Add support of IE model cache in time_tests (#5192) 2021-04-15 00:11:48 +03:00
Alexander Shchepetov e864972f83
Add experiments timings to data to submit to DB(#5108) 2021-04-12 09:29:10 +03:00
Victor Kuznetsov 69a06f5977
Remove time-tests outliers using Inter Quartile Range (IQR) (#5014)
Change number of times to execute binary to aggregate statistics from 3 to 10.
2021-04-02 11:35:38 +03:00
Alexey Suhov 6478f1742a
Align copyright notice in python scripts (CVS-51320) (#4974)
* Align copyright notice in python scripts (CVS-51320)
2021-03-26 17:54:28 +03:00
Victor Kuznetsov a6d9842152
mv getBatchSize to cnnNetwork scope (#4231) 2021-02-09 10:47:44 +03:00
Andrey Somsikov 359c2ca018
Upgrade pyyaml to the latest (#4134) 2021-02-03 07:07:01 +03:00
Victor Kuznetsov 1217d26f2f
rename time test config (#3923) 2021-01-21 12:32:15 +03:00
Vitaliy Urusovskij 1e858830e2
Change `json.loads` to `json.load` for `--db_metadata` (#3871) 2021-01-19 18:33:00 +03:00
Vitaliy Urusovskij fe229ba19d
Move time_tests to new manifest version (#3759) 2020-12-30 17:28:22 +03:00
Vitaliy Urusovskij e910902c9d
Fix paths for `squeezenet1.1` in time_tests config (#3415) 2020-11-30 18:45:08 +03:00
Vitaliy Urusovskij 364bf08920
Add several new models to `tgl_test_config.yml` in time_tests (#3268)
* Fix wrong path for `yolo-v2-tiny-ava-0001` for time_tests

* Add several new models to `tgl_test_config.yml` in time_tests
2020-11-24 13:26:16 +03:00
Vitaliy Urusovskij dee00cdc92
Add new model to `tgl_test_config.yml` (#3236) 2020-11-20 13:33:57 +03:00
Vitaliy Urusovskij b5930eb58e
Fix "results" uploading to DB in time_tests (#3215) 2020-11-20 10:49:31 +03:00
Vitaliy Urusovskij 070fc17087
Extend information to submit to a DB in time_tests (#3018)
* Optimize imports in time_tests conftest.py

* Extend information to submit to a DB in time_tests:
1. Add `--manifest` and `--db_metadata` CLI keys
2. Add `prepare_db_info` fixture

* Move `validate_test_case` fixture to a const schema, remove extra checks

* Add `manifest_metadata` fixture to parse and validate manifest only once

* Add OS utils to submit os info to DB
2020-11-17 12:07:40 +03:00
Vitaliy Urusovskij e1f4585cb8
Add `validate_test_case` fixture with using of `jsonschema`. Specify all required fields for test cases (#2821) 2020-10-29 00:11:01 +03:00
Vitaliy Urusovskij 12b690952c
Time tests improvements (#2642)
* Remove extra functions from run_timetest.py

* Add `log.debug` of raw and aggregated statistics in run_timetest.py

* Implement storing of models locally for test_timetest.py
2020-10-14 20:05:52 +03:00
Vitaliy Urusovskij a4fe59ba0a
Specify `cl_cache_dir` env var: (#2581)
1. General behaviour doesn't work for some compute-runtime versions.
Specification of `cl_cache_dir` fixes it
2020-10-08 15:02:31 +03:00
Vitaliy Urusovskij 474dcc277f
Prepare new test configuration for test_timetest.py (#2561)
* Add logging of DB upload to tconf with refs saving

* Prepare `.automation/tgl_test_config.yml` with models from VPUX package

* Add restriction on `exe` and `model` existence in run_timetest.py
2020-10-08 10:30:43 +03:00
Vitaliy Urusovskij c3ff2948a4
Timetests test_runner improvements (#2552)
* Remove `generate_tmp_path` as unnecessary after refactoring

* Add `check_positive_int` check for `-niter` CLI key

* Replace `TestConfDumper` with number of fixtures:
1. Save all test info in global `request` and `pytestconfig` fixtures
2. Add `test_info` fixture for retrieving test info from test
3. Add `prepare_tconf_with_refs` fixture for test conf dump
2020-10-07 17:15:02 +03:00
Vitaliy Urusovskij 6949d93e42
Add `"cl_cache"` preparation in test_timetest.py (#2490) 2020-10-06 14:16:50 +03:00
Vitaliy Urusovskij 639052d0b8
Support references in test_timetest.py (#2431) 2020-10-01 11:35:57 +03:00
Vitaliy Urusovskij c8530201dd
Support DB reporting for test_timetest.py (#2359) 2020-10-01 08:39:19 +03:00
Vitaliy Urusovskij d48d2109dc
Test timetest MVP (#2345) 2020-09-21 21:33:42 +03:00
Vitaliy Urusovskij 20266276ec
New layout for `time_tests` (#2321)
* Change folders' layout for `time_tests`

* Refactoring of namings in `time_tests`
2020-09-18 18:53:49 +03:00