e6beb94085
Bumps [jsonschema](https://github.com/python-jsonschema/jsonschema) from 4.17.0 to 4.22.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/python-jsonschema/jsonschema/releases">jsonschema's releases</a>.</em></p> <blockquote> <h2>v4.22.0</h2> <!-- raw HTML omitted --> <h2>What's Changed</h2> <ul> <li>Improve <code>best_match</code> (and thereby error messages from <code>jsonschema.validate</code>) in cases where there are multiple <em>sibling</em> errors from applying <code>anyOf</code> / <code>allOf</code> -- i.e. when multiple elements of a JSON array have errors, we now do prefer showing errors from earlier elements rather than simply showing an error for the full array (<a href="https://redirect.github.com/python-jsonschema/jsonschema/issues/1250">#1250</a>).</li> <li>(Micro-)optimize equality checks when comparing for JSON Schema equality by first checking for object identity, as <code>==</code> would.</li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/shinnar"><code>@shinnar</code></a> made their first contribution in <a href="https://redirect.github.com/python-jsonschema/jsonschema/pull/1224">python-jsonschema/jsonschema#1224</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/python-jsonschema/jsonschema/compare/v4.21.1...v4.22.0">https://github.com/python-jsonschema/jsonschema/compare/v4.21.1...v4.22.0</a></p> <h2>v4.21.1</h2> <!-- raw HTML omitted --> <ul> <li>Slightly speed up the <code>contains</code> keyword by removing some unnecessary validator (re-)creation.</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/python-jsonschema/jsonschema/compare/v4.21.0...v4.21.1">https://github.com/python-jsonschema/jsonschema/compare/v4.21.0...v4.21.1</a></p> <h2>v4.21.0</h2> <!-- raw HTML omitted --> <h2>What's Changed</h2> <ul> <li>Fix the behavior of <code>enum</code> in the presence of <code>0</code> or <code>1</code> to properly consider <code>True</code> and <code>False</code> unequal (<a href="https://redirect.github.com/python-jsonschema/jsonschema/issues/1208">#1208</a>).</li> <li>Special case the error message for <code>{min,max}{Items,Length,Properties}</code> when they're checking for emptiness rather than true length.</li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/otto-ifak"><code>@otto-ifak</code></a> made their first contribution in <a href="https://redirect.github.com/python-jsonschema/jsonschema/pull/1208">python-jsonschema/jsonschema#1208</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/python-jsonschema/jsonschema/compare/v4.20.0...v4.21.0">https://github.com/python-jsonschema/jsonschema/compare/v4.20.0...v4.21.0</a></p> <h2>v4.20.0</h2> <ul> <li>Properly consider items (and properties) to be evaluated by <code>unevaluatedItems</code> (resp. <code>unevaluatedProperties</code>) when behind a <code>$dynamicRef</code> as specified by the 2020 and 2019 specifications.</li> <li><code>jsonschema.exceptions.ErrorTree.__setitem__</code> is now deprecated. More broadly, in general users of <code>jsonschema</code> should never be mutating objects owned by the library.</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/python-jsonschema/jsonschema/compare/v4.19.2...v4.20.0">https://github.com/python-jsonschema/jsonschema/compare/v4.19.2...v4.20.0</a></p> <h2>v4.19.2</h2> <!-- raw HTML omitted --> <ul> <li>Fix the error message for additional items when used with heterogeneous arrays.</li> <li>Don't leak the <code>additionalItems</code> keyword into JSON Schema draft 2020-12, where it was replaced by <code>items</code>.</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/python-jsonschema/jsonschema/compare/v4.19.1...v4.19.2">https://github.com/python-jsonschema/jsonschema/compare/v4.19.1...v4.19.2</a></p> <h2>v4.19.1</h2> <!-- raw HTML omitted --> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/python-jsonschema/jsonschema/blob/main/CHANGELOG.rst">jsonschema's changelog</a>.</em></p> <blockquote> <h1>v4.22.0</h1> <ul> <li>Improve <code>best_match</code> (and thereby error messages from <code>jsonschema.validate</code>) in cases where there are multiple <em>sibling</em> errors from applying <code>anyOf</code> / <code>allOf</code> -- i.e. when multiple elements of a JSON array have errors, we now do prefer showing errors from earlier elements rather than simply showing an error for the full array (<a href="https://redirect.github.com/python-jsonschema/jsonschema/issues/1250">#1250</a>).</li> <li>(Micro-)optimize equality checks when comparing for JSON Schema equality by first checking for object identity, as <code>==</code> would.</li> </ul> <h1>v4.21.1</h1> <ul> <li>Slightly speed up the <code>contains</code> keyword by removing some unnecessary validator (re-)creation.</li> </ul> <h1>v4.21.0</h1> <ul> <li>Fix the behavior of <code>enum</code> in the presence of <code>0</code> or <code>1</code> to properly consider <code>True</code> and <code>False</code> unequal (<a href="https://redirect.github.com/python-jsonschema/jsonschema/issues/1208">#1208</a>).</li> <li>Special case the error message for <code>{min,max}{Items,Length,Properties}</code> when they're checking for emptiness rather than true length.</li> </ul> <h1>v4.20.0</h1> <ul> <li>Properly consider items (and properties) to be evaluated by <code>unevaluatedItems</code> (resp. <code>unevaluatedProperties</code>) when behind a <code>$dynamicRef</code> as specified by the 2020 and 2019 specifications.</li> <li><code>jsonschema.exceptions.ErrorTree.__setitem__</code> is now deprecated. More broadly, in general users of <code>jsonschema</code> should never be mutating objects owned by the library.</li> </ul> <h1>v4.19.2</h1> <ul> <li>Fix the error message for additional items when used with heterogeneous arrays.</li> <li>Don't leak the <code>additionalItems</code> keyword into JSON Schema draft 2020-12, where it was replaced by <code>items</code>.</li> </ul> <h1>v4.19.1</h1> <ul> <li>Single label hostnames are now properly considered valid according to the <code>hostname</code> format. This is the behavior specified by the relevant RFC (1123). IDN hostname behavior was already correct.</li> </ul> <h1>v4.19.0</h1> <ul> <li>Importing the <code>Validator</code> protocol directly from the package root is deprecated. Import it from <code>jsonschema.protocols.Validator</code> instead.</li> <li>Automatic retrieval of remote references (which is still deprecated) now properly succeeds even if the retrieved resource does not declare which version of JSON Schema it uses. Such resources are assumed to be 2020-12 schemas. This more closely matches the pre-referencing library behavior.</li> </ul> <h1>v4.18.6</h1> <ul> <li>Set a <code>jsonschema</code> specific user agent when automatically retrieving remote references (which is deprecated).</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|---|---|---|
| .. | ||
| common | ||
| pipelines | ||
| test_utils | ||
| CMakeLists.txt | ||
| README.md | ||
| __init__.py | ||
| base_test_rules.yml | ||
| collect_refs.py | ||
| config.py | ||
| conftest.py | ||
| env_config_local.yml | ||
| pytest.ini | ||
| requirements.txt | ||
| reshape_test_rules.yml | ||
| test_base.py | ||
| test_config_local.yml | ||
README.md
End-to-end Tests User Documentation
This folder contains a code to run end-to-end validation of OpenVINO on real models of different frameworks (PyTorch, TensorFlow, and ONNX)
The documentation provides necessary information about environment setup for e2e validation run, adding new model to the validation, and instructions to launch validation.
The following steps assume that your current working directory is:
tests/e2e_tests
Environment preparation:
- Install Python modules required for tests:
pip3 install -r requirements.txt
Add model from TensorFlow Hub repo to end-to-end validation:
To add new test for model from TF Hub repo just add new line into pipelines/production/tf_hub/precommit.yml This line should contain comma separated model name and its link
movenet/singlepose/lightning,https://www.kaggle.com/models/google/movenet/frameworks/tensorFlow2/variations/singlepose-lightning/versions/4
Main entry-point
There is one main testing entry-point which is responsible for test run - test_base.py. This script performs the following actions:
- Loads model from its source
- Infers original model through framework
- Converts original model through OVC convert model
- Infers converted model through OpenVINO
- Provides results of element-wise comparison of framework and OpenVINO inference
Launch tests
test_base.py is the main script to run end-to-end tests.
Run all end-to-end tests in pipelines/:
pytest test_base.py
test_base.py options:
--modules=MODULES [MODULES ...]- Paths to tests.-k TESTNAME [TESTNAME ...]- Test names.-s- Step-by-step logging.
Example:
pytest test_base.py -s --modules=pipelines/production/tf_hub
For full information on pytest options, run
pytest --helpor see the documentation