### Details:
Ported SDPA to PagedAttention transformation from python to C++ code.
the related PRs:
https://github.com/openvinotoolkit/openvino/pull/24127https://github.com/openvinotoolkit/openvino/pull/24177
Tested model scope:
- [x] "hf-internal-testing/tiny-random-BloomForCausalLM",
- [x] "hf-internal-testing/tiny-random-FalconForCausalLM",
- [x] "hf-internal-testing/tiny-random-Starcoder2ForCausalLM",
- [x] "hf-internal-testing/tiny-random-GPTJForCausalLM",
- [x] "hf-internal-testing/tiny-random-StableLmForCausalLM",
- [x] "hf-internal-testing/tiny-random-LlamaForCausalLM",
- [x] "hf-internal-testing/tiny-random-MistralForCausalLM",
- [x] "hf-internal-testing/tiny-random-OPTForCausalLM",
- [x] "hf-internal-testing/tiny-random-PhiForCausalLM",
- [x] "hf-internal-testing/tiny-random-StableLmForCausalLM",
- [x] "facebook/opt-125m",
- [x] "llama2",
- [x] "bigcode/starcoder2-7b"
- [ ] "mosaicml/mpt-7b-chat" (FAILED both py/c++) - acceptable for this
PR
Issue: RuntimeError: Check '(axis_range_min <= axis) && (axis <=
axis_range_max)' failed at src/core/src/validation_util.cpp:386:
Concat Parameter axis 2 out of the tensor rank range [0, 0].
- [x] _means, that the response to the dedicated prompt is the same for
the py and c++ transformations._
### Tickets:
- *CVS-138664*
---------
Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
Co-authored-by: Andrii Staikov <andrii.staikov@intel.com>
### Details:
- adding new struct for PCI information (aligned with GPU)
- adding new property to query PCI information (as a RO metric)
- implement query of pci information from driver in zero_device, with
backwards compatibility
### Tickets:
[ - *ticket-id*](https://jira.devtools.intel.com/browse/CVS-138767)
Updates the requirements on
[flake8-builtins](https://github.com/gforcada/flake8-builtins) to permit
the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/gforcada/flake8-builtins/blob/main/CHANGES.rst">flake8-builtins's
changelog</a>.</em></p>
<blockquote>
<h2>2.5.0 (2024-04-09)</h2>
<ul>
<li>Support detecting a shadowing folder as module name (as part of
<code>A005</code>).
[asfaltboy]</li>
</ul>
<h2>2.4.0 (2024-04-01)</h2>
<ul>
<li>Add rule for lambda argument shadowing (<code>A006</code>).
[cielavenir]</li>
</ul>
<h2>2.3.0 (2024-03-29)</h2>
<ul>
<li>Add rule for builtin module name shadowing (<code>A005</code>).
[asfaltboy]</li>
</ul>
<h2>2.2.0 (2023-11-03)</h2>
<ul>
<li>Use <code>tox</code> and <code>pre-commit</code> to ease project
maintenance.
[gforcada]</li>
</ul>
<h2>2.2.0a0 (2023-11-01)</h2>
<ul>
<li>
<p>Drop python 3.7 support.
[gforcada]</p>
</li>
<li>
<p>Test pre-release of python 3.12.
[gforcada]</p>
</li>
<li>
<p>Use <code>pyproject.toml</code> rather than <code>setup.py</code>.
[gforcada]</p>
</li>
<li>
<p>Switch from <code>setuptools</code> to <code>hatchling</code>.
[gforcada]</p>
</li>
<li>
<p>Add rule for import shadowing (<code>A004</code>).
[felixvd]</p>
</li>
<li>
<p>Switch to <code>main</code> branch.
[gforcada]</p>
</li>
</ul>
<h2>2.1.0 (2022-12-23)</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="ab45f4d565"><code>ab45f4d</code></a>
Preparing release 2.5.0</li>
<li><a
href="3890afbd5e"><code>3890afb</code></a>
Merge pull request <a
href="https://redirect.github.com/gforcada/flake8-builtins/issues/125">#125</a>
from asfaltboy/ps/import-a005-module-dir</li>
<li><a
href="b46b4d7b80"><code>b46b4d7</code></a>
chore: add change note for shadow dir module</li>
<li><a
href="74588e17d8"><code>74588e1</code></a>
feat(A005): support detecting a shadowing folder</li>
<li><a
href="5b6dbffe32"><code>5b6dbff</code></a>
Back to development: 2.4.1</li>
<li><a
href="90cd95b0db"><code>90cd95b</code></a>
Preparing release 2.4.0</li>
<li><a
href="19e04fab10"><code>19e04fa</code></a>
fix: revert release date</li>
<li><a
href="7793730ebf"><code>7793730</code></a>
chore: black</li>
<li><a
href="116ce7b766"><code>116ce7b</code></a>
Merge pull request <a
href="https://redirect.github.com/gforcada/flake8-builtins/issues/123">#123</a>
from cielavenir/raiseA002ForLambda</li>
<li><a
href="8702675502"><code>8702675</code></a>
update changes</li>
<li>Additional commits viewable in <a
href="https://github.com/gforcada/flake8-builtins/compare/0.1...2.5.0">compare
view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
### Details:
- *Add ScatterNDUpdate-15 to PyAPI*
- *Parametrize to test both ScatterND versions + add test for reduce
attribute*
### Tickets:
- *111092*
---------
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
### Details:
- *Extended the calls to as_node and as_nodes with the name parameter in
all the opsets.*
- *Added tests for the involved operations.*
### Tickets:
- #24030
- part of
[CVS-133139](https://jira.devtools.intel.com/browse/CVS-133139)
### Details:
Numpy started preparation to release 2.0 version.
It brings many breaking changes listed below.
Align with these changes it's a plan for further development
(CVS-138766), for today we want to secure users against unintentional
installation of 2.x version of numpy with OV.
This PR has to be ported for 2024/1 branch
- Preparing to restrict OV to not use release of Numpy 2.0 which may
introduce many breaking changes including:
- required bump of `pybind` to 2.12+
- API and namespaces clean-ups and deprecations
- behavior changes of keywords such as `copy`
- changes to existing and introduction of new dtypes (such as
`StringDType` and possibility of adding custom ones)
- more can be found here:
https://numpy.org/devdocs/release/2.0.0-notes.html
- and here:
https://numpy.org/devdocs/numpy_2_0_migration_guide.html#numpy-2-migration-guide
### Tickets:
- CVS-138838
---------
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
### Details:
- Constant's function `get_data` is extended to provide both copy and
view of the data.
- `get_data` can now cast date to desired dtype.
- Added testcases.
Example:
```python
In [1]: import openvino as ov
In [2]: import numpy as np
In [3]: arr = np.array([1.0, 1012.5, 0.0, 0.1, 0.5], dtype=np.float32)
In [4]: c = ov.runtime.opset13.constant(arr, dtype=np.float32)
In [5]: c.get_data(dtype=np.bool_)
Out[5]: array([ True, True, False, True, True])
In [6]: c = ov.runtime.opset13.constant(arr, dtype=ov.Type.bf16)
In [7]: c.get_data(dtype=np.float32)
Out[7]:
array([1.0000000e+00, 1.0120000e+03, 0.0000000e+00, 9.9609375e-02,
5.0000000e-01], dtype=float32)
In [8]: c.get_data(dtype=np.float32, copy=False)
Out[8]:
array([1.0000000e+00, 1.0120000e+03, 0.0000000e+00, 9.9609375e-02,
5.0000000e-01], dtype=float32)
```
### Tickets:
- CVS-129930
- CVS-127292
### Details:
- Removed internal conversion of all typed passed to `convert_model()`
to `np.dtype`, as some types do not have `numpy` analogs, which causes
errors. Also this step is redundant as we can use `ov.Type` instead.
### Tickets:
- 138755
### Details:
- Limit NF4 conversion to FP16 -> NF4 in convert operator as values are
correctly quantized, Not support NF4 conversion to/from other types.
### Tickets:
- [CVS-135304](https://jira.devtools.intel.com/browse/CVS-135304)
### Details:
- Automatically adding the node name to all autogenerated constants.
Example: `NodeName/Constant_X`
- Removed `apply_affix_on` approach.
- Adjusted testcases to check compatibility.
- The change is backward incompatible -- breaking cases unknown.
- To be added: extend other operations `as_node/as_nodes` to follow this
example, as this approach is not automatically applicable.
### Tickets:
- CVS-133139
---------
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
### Details:
- Initialization of OV Opset15
(Discussed internally and agreed to introduce new opset in 2024.2)
To provide more agile approach, this PR has been updated with an idea
(proposed by @mmikolajcz) to include only new operators within the new
Opset, and fill it up at the end of the development period (not
necessarily per release).
This should lead to infrequent introduction of further opsets.
\* Currently common opsets tests rely on Parameter/ShapeOf/Convert in
every opset, so they were added.
But this approach for opset testing should be revised.
### Tickets:
- 138268, 131683
### Details:
- *The PR adds RISCV64 arch for python wheel setup to fix the following
error during build on the target machine:*
```
[100%] Building Python wheel
openvino-2024.2.0-14978-cp310-cp310-manylinux_2_36_riscv64.whl
Processing /home/sipeed/openvino/openvino/src/bindings/python/wheel
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File
"/home/sipeed/openvino/openvino/src/bindings/python/wheel/setup.py",
line 50, in <module>
OV_RUNTIME_LIBS_DIR = os.getenv("OV_RUNTIME_LIBS_DIR",
f"runtime/{LIBS_DIR}/{ARCH}/{CONFIG}")
NameError: name 'ARCH' is not defined
[end of output]
note: This error originates from a subprocess, and is likely not a
problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
make[2]: ***
[src/bindings/python/wheel/CMakeFiles/ie_wheel.dir/build.make:177:
wheels/openvino-2024.2.0-14978-cp310-cp310-manylinux_2_36_riscv64.whl]
Error 1
make[1]: *** [CMakeFiles/Makefile2:2841:
src/bindings/python/wheel/CMakeFiles/ie_wheel.dir/all] Error 2
make: *** [Makefile:156: all] Error 2
```
### Tickets:
- *N/A*
PagedAttention operation exposed in Python API only for easier vLLM
openvino integration. It is not intended to be used outside our
integration work in vLLM and similar applications where we can use
PagedAttention. Exposed as a hidden part of API, will not be documented.
Connected to already existing implementation in CPU plugin.
Operation is not a part of any public opset.
### Details:
- Edited and overloaded the __getitem__ operation for `Shape` and
`PartialShape`
- Added tests for correct the implementation
### Tickets:
- 75641
- #21968
---------
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
Co-authored-by: Przemyslaw Wysocki <przemyslaw.wysocki@intel.com>
### Details:
- Support LLM inference
|device|
|--|
|HETERO:CPU|
|HETERO:GPU|
|HETERO:CPU,GPU|
|HETERO:GPU,CPU|
|HETERO:GPU.0,GPU.1|
|HETERO:GPU.0,GPU.1,CPU|
|HETERO:GPU.0,GPU.1,GPU.2|
- Use the `ov::hint::model_distribution_policy` property in
[PR23077](https://github.com/openvinotoolkit/openvino/pull/23077/)
- Use host memory in different subgraphs when input/output data exchange
- Mask supported nodes and unsupported nodes to Subgraph in graph, and
query model use subgraph, keep the model in query_model same as compile
- Add a property `ov::query_model_ratio` to set the percentage of the
model can be queried during query model
- Improve performance on some LLM with large parameter by split the
model to different devices, the number of split models should be
smaller, reduce communication usage between multiple devices
### Tickets:
- *CVS-133258*
---------
Co-authored-by: Nadezhda <nadezhda.ageeva@intel.com>
Co-authored-by: Shen, Wanglei <wanglei.shen@intel.com>
Co-authored-by: yanlan song <bell.song@intel.com>
### Details:
- Add Python API for compile_model from model and weight buffer in
memory
### Tickets:
- CVS-135799
---------
Co-authored-by: jiwaszki <jan.iwaszkiewicz@intel.com>
### Details:
- *Add support of ones_like and zeros in fx*
These two ops appear in LLaVA and AudioCraft models respectively.
### Tickets:
- *No ticket id*
---------
Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>
### Details:
- *use logger to print the logs in dynamo bindings.*
- *Add a warning of the case that openvino_execute fails when building
OpenVINOGraphModule*
### Tickets:
- *No ticket id*
### Description
Some modules of dynamo bindings of openvino use `print` directly to
print logs. I think it will be better if using `logging` to allow
control of the logs.
---------
Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>
### Details:
- *The first infer request for stable diffusion model might create
Tensor with shape and size 0, it must be handled to avoid attempt to
reshape array of size 1 into 0 shape*
---------
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
### Details:
- extend the `util::DictAttributeSerializer::on_adapter()` method,
making it compatible with `ov::PartialShape` and
`ov::op::util::Variable` types;
- add extra tests to test the correctness of `Node.get_attributes()`
### Tickets:
- #23455
---------
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
[TRANSFORMATIONS] Create python binding for pattern::Optional
Expose the C++ op::pattern::Optional to Python in order to
simplify patterns creation.
Cover the functionality with the dedicated tests.
### Tickets:
CVS-133523
Signed-off-by: Andrii Staikov <andrii.staikov@intel.com>
---------
Signed-off-by: Andrii Staikov <andrii.staikov@intel.com>
### Details:
- *Enable new property model_distribution_policy for CPU inference*
-- *Add C++ interface and test cases*
-- *Add Python interface and test cases*
### Tickets:
- *CVS-127844*
New op support for:
- torch.export updates
- benchmarking model support
- chatglm2 support
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: ynimmaga <yamini.nimmagadda@intel.com>
Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
### Details:
There were two problems with memory sharing, which affected HF models.
`value()` method of class `ResourceVariable` copies value of variable in
some cases. This results in sharing of memory with a copied value, which
is deallocated when `GraphIterator` finishes its work.
Replaced `value()` with `read_value_no_copy()` which should not copy
value in any case.
Another problem is that original model inference moves weights to new
location which leads to corruption of shared memory in OV model.
So the original model should be inferred after the OV inference to
prevent any changes in original model.
### Tickets:
- 134787, 134408
---------
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
Ivan Tikhonov
Rinne