openvino/tests/stress_tests

Stress Tests Suite

This test suite contains tests evaluating the behavior of various OpenVINO use cases under stress conditions:

• MemCheckTests measuring memory required for the use cases and fail when memory usage exceeds a pre-defined level.

• StressMemLeaksTests ensure that the use cases does not increase memory levels when executing continuously.

• StressUnitTests executing various Inference Engine use cases in parallel threads and processes.

Each test refers to configuration files located in <test dir>\local_configs folder. The configuration files are installed along with tests on build time.

Getting Started

Stress tests are based on the googletest framework. You can filter tests with --gtest_filter and explore tests available with --gtest_list_tests options.

Tests measuring memory have a temporary limitation - those need to be executed one at a time to mitigate memory statistics pollution. You can use gtest-parallel for massive tests execution.

Pre-requisites

• Linux OS to build the tests.

• gtest-parallel to execute tests.

Building Tests

Stress tests should be built in 2 steps.

1. Build dldt

Build dldt as usual but with -DENABLE_TESTS=ON.

2. Build stress_tests

Stress tests depend from the Inference Engine Developer Package located in the dldt build directory.

In the command line snippet bellow, it is assumed that the Inference Engine Developer Package CMake module can be found in the directory build under dldt repository root.

(
export DLDT_BUILD_DIR=$(git rev-parse --show-toplevel)/build
mkdir -p build && cd build && \
cmake -DInferenceEngineDeveloperPackage_DIR=$DLDT_BUILD_DIR .. && make -j$(nproc) \
)

Preparing Test Data

Stress test use models from Open Model Zoo. Download and convert models to IRs using ./scripts/get_testdata.py script.

From Intel network you can use models from cache at vdp_tests file share. Refer to VDP shared folders on using file shares.

Running Tests

gtest-parallel ./MemCheckTests

gtest-parallel ./StressMemLeaksTests

There are the next available command-line keys:

1. --test_conf < path > - path to config with description of arguments used to parametrize tests
2. --test_conf < path > - path to config with definition of environment values (path to models etc.)
3. --refs_conf < path > (available for MemCheckTests only) - path to config with references used to compare with results of a run
4. --collect_results_only < bool > (available for MemCheckTests only) - boolean value that disable comparison and provide memory consumption results only

MemCheckTests logs can be used to gather reference values based on current memory consumption:

mkdir -p MemCheckTests-logs && \
gtest-parallel -d ./MemCheckTests-logs ./MemCheckTests && \
grep -rh ./MemCheckTests-logs -e ".*<model " | sed -e "s/.*<model /<model /" | sort