openvino/tests/stress_tests

Stress Tests Suite

This test suite contains tests evaluating the behavior of various OpenVINO use cases under stress conditions:

• MemCheckTests measuring memory required for the use cases and fail when memory usage exceeds a pre-defined level.

• StressMemLeaksTests ensure that the use cases does not increase memory levels when executing continuously.

• StressUnitTests executing various Inference Engine use cases in parallel threads and processes.

Each test refers to configuration files located in <test dir>\.automation folder.

Getting Started

Stress tests are based on the googletest framework. You can filter tests with --gtest_filter and explore tests available with --gtest_list_tests options.

Tests measuring memory have a temporary limitation - those need to be executed one at a time to mitigate memory statistics pollution. You can use gtest-parallel for massive tests execution.

Pre-requisites

• Linux OS to build the tests.

• gtest-parallel to execute tests.

Building Tests

To build the tests, you need to have OpenVINO™ installed or build from source. Before build the tests, open a terminal, set OpenVINO™ environment, and after that run the commands below:

source <OpenVINO_install_dir>/setupvars.sh
mkdir build && cd build
cmake .. && make -j$(nproc)

Preparing Test Data

Stress tests may work with models from Open Model Zoo. To use it, download and convert models to IRs using ./scripts/get_testdata.py script. Script will update test config file with data required for OMZ models execution.

From Intel network you can use models from cache at vdp_tests file share. Refer to VDP shared folders on using file shares.

Running Tests

gtest-parallel <openvino_bin>/StressMemLeaksTests

For MemCheckTests preferable way is:

python ./scripts/run_memcheck.py --gtest_parallel <gtest_parallel_py_path> 
<openvino_bin>/MemCheckTests -- --test_conf=<test_conf_path> --refs_conf=<refs_conf_path>

MemCheckTests logs can be used to gather reference values based on current memory consumption:

mkdir -p MemCheckTests-logs && \
gtest-parallel -d ./MemCheckTests-logs ./MemCheckTests && \
grep -rh ./MemCheckTests-logs -e ".*<model " | sed -e "s/.*<model /<model /" | sort