* Caching: pass global CACHE_DIR setting to plugin
This can be helpful for GPU - it doesn't support Import/Export but can
significantly speed up load time when CACHE_DIR is set for device only
* Ignore exception in 'DeviceSupportsConfigKey' if plugin doesn't support GetMetric at all
* Multi plugin - override loading network from file
When caching is enabled, MULTI plugin will check all devices
- For devices with caching supported - call LoadNetwork(modelPath, ...)
- For others - ReadNetwork once and then LoadNetwork(cnnNetwork) for each device
Caching unit test is added for both cases
Additional helper methods:
- ICore::ToExecutableNetwork - converts internal ExeNetwork to ExecutableNetwork
- ICore::DeviceSupportsImportExport - checks if device supports import and export functionality. Used by Hetero and Multi
* Updated according to review comments
* fixed sporadic failure of 'multi-device' test cases
Root cause:
Currently only one 'ExecutableNetwork' object is created for each LoadNetwork
For Multi-testing several threads could call simultaneously setNetworkInputs/Outputs/SetPointerToPlugin
It caused race condition and invalid data structures
* Fix build issues after rebase
* Multi: Set network inputs/outputs/pointerToPlugin for load-from-file case
Overloaded function doesn't call these methods, thus multi executable network was unusable
Added caching test verifying that inputs/outputs are copied now from first loaded device network
* Added LoadNetwork(filename) to AUTO
* Added more files
* So pointer can be used without loading
* Changed InferencePlugin, ICore to return internal interfaces
* Added SoPointers for InferRequest, ExecutableNetwork
* Fixed Windows
* Fixed KMB
* Fixes for KMB
* Removed dereference operator
* Play with include files
* Fixed compilation with older compilers
* Fixed comments
* Fixed win build
* Try to fix Windows
* Try to fix Windows 2
* Fixed windows
* Fixed windows
* Removed SOPointer as a base class
* Reverted back SOPointer split
* Code review
Co-authored-by: apankratovantonp <anton.pankratov@intel.com>
* Minimized legacy usage in tests
* Use legacy only for specific files
* Fixed code style
* Fixed linkage
* Removed old CPU / GPU tests binaries
* Test
* Disabled IB console
* Disabled test for AUTO QueryNetwork from parallel threads
It is a known internal issue in gtest when holding a shared_ptr to mocked object, which sometimes reports about memory leak
It is recommended to use Mock::VerifyAndClearExpectations at the end of each test when mock object is not needed anymore
After adding this, issue with incorrect TestThrowOnImport expectations is observed
* HotFix for ConvertPrecision tests
* [Caching] Don't propagate import error on plugin side
Instead, invalid cache will be silently removed and network will be loaded without cache
Co-authored-by: Ilya Churaev <ilya.churaev@intel.com>
On CI machines there is a chance to run various test executables at the same time.
When intensive CPU load occurs, 'sleeps' can last much longer than expected even in very pessimistic scenario
* Removed suppressions for IInferRequest deprecation
* Fixed Windows
* More fixes for Windows
* Fixed compilation on Windows
* Fixed comment in documentatipn
* Fixes for Andorid
* Fixes for old gcc 4.8
* WA for cross-compilations
* Fixed compilation
* Fixed HETERO plugin compilation for old compilers
* Flags
Co-authored-by: lab_ddpqa <lab_ddpqa@intel.com>
* Caching support of multi-device scenario
- IE_CORE: introduce CacheGuard which can create locks for specific cache identified by 'hash'
- Added functional tests for it
Fixes of Thread Sanitizer failures:
- ngraph::Serialize - m_ref[i] can create new element, casted to 'const' to avoid this
- ngraph::get_opset oprations: reworked to use std::call_once instead of double bool check
* Added docs for ie_cache_guard.hpp
* Fix Debian 9 compilation issue
* Fix build for CentOS 6
Added assert to verify that table of locked hashes is empty on destruction
* Fixed review comments
* [Hetero plugin] Model caching support
- Enable IMPORT_EXPORT_SUPPORT metric
- Introduce internal FORCE_DISABLE_CACHE flag to avoid caching of subnetworks
- Added test for hetero with plugins which don't support caching
- Added test for hetero with plugins supported different cache architecture
* Hetero plugin - support DEVICE_ARCHITECTURE
Test setup:
mock.1 - mock.9 returns "one" for DEVICE_ARCHITECTURE
mock.10 - mock.99 returns "two"
Test:
Load "HETERO:mock.1,mock.51".
Load "HETERO:mock.2,mock.52" - cache shall be reused
* Fixed review comments
Covered use case case
ie.SetConfig({{"TARGET_FALLBACK", "CPU"}}, "HETERO");
ie.LoadNetwork(network, "HETERO");
* Fixed more comments and failed tests
Don't propagate FORCE_DISABLE_CACHE to plugins as they can throw exception
Fixed case with set TARGET_FALLBACK from core with different architectures of one plugin
* Fix unit tests
Add 'FORCE_DISABLE_CACHE' config key only for LoadExeNetwork
It is not needed to have in in QueryNetwork and other places
* Attempt to fix failed func test on Windows
* Model caching support - Core part
Introducing model caching support
Use core.SetConfig({{CONFIG_KEY(CACHE_DIR), <dir>}}); to enable caching of models
OpenVINO will try to create caching folder if it doesn't exist, but it is recommended for client to create caching folder with necessary permissions before enabling cache in config
For caching, plugins shall support import/export functionality
Plugin requirements:
- Add METRIC_KEY(IMPORT_EXPORT_SUPPORT) in SUPPORTED_METRICS to support caching
If plugin has different device architectures with different caches, i.e.
For "GNA.0" - one cache, for "GNA.10" - another cache
In this case plugin shall support DEVICE_ARCHITECTURE metric and return different strings for different DEVICE_ID's
Added functional tests
* Fix CentOS build issues
* Few updates according to code review
* Revert unnecessary changes for Import/Export core implementation
These changes affect old behavior and may be undesired
For caching support these is no need to change anything in this area
If needed, such removal of 'Magic' usage can be done under separate task in future
* More tests:
1) Verify that Imported data from stream is the same as was exported
2) Verify that cache is not loaded when config in LoadNetwork is changed
3) Verify that if CNN Network is changed between ReadNetwork and LoadNetwork - cache is not loaded
* Update of NetworkCompilationContext
Put back functionality of calculating hash based on runtime information, weights
Implemented OstreamHashWrapper to avoid serialization to buffer
* Correction of CACHE_DIR key description
* Unit tests for compilation_context
Changes:
1) Improved handling of OstreamHashAdapter
2) Improved runtime info serialization (not just PrimitivesPriority and affinity)
3) Removed redundant weights hash calculation
* Fix GCC 4.8 build issues
* Compilation context updates
1) Use hash of sum of serialized data to get hash of network. It is more efficient comparing to weights sum calculation
2) CalculateFileInfo - convert path to absolute ("./test.blob" and "test.blob" shall give same hash)
* Hash - added more rt_info attributes + tests
- PrimitivesPriority
- FusedNames
- Dequantization
* Moved "get_absolute_path" macro to file_utils.h
* Make 'absoluteFilePath' a library API, not macro
* One more unit test for fileName hashing
* Fix compilation error after merge with latest master
* Allow tests to be executed in parallel (stress mode)
* More minor updates for stress testing
Now it allows to execute tests with '--repeat=100' option where one test is executed in multiple processes simultaneously
Example:
./gtest-parallel <openvino_dir>/bin/intel64/Debug/ieFuncTests --gtest_filter=CachingTest* --repeat=10
* Use absolute model file path for calculating blob name
* Added 'createDirectoryRecursive' API to plugin_api/file_utils
Ilya Lavrenov