### Details:
Ported SDPA to PagedAttention transformation from python to C++ code.
the related PRs:
https://github.com/openvinotoolkit/openvino/pull/24127https://github.com/openvinotoolkit/openvino/pull/24177
Tested model scope:
- [x] "hf-internal-testing/tiny-random-BloomForCausalLM",
- [x] "hf-internal-testing/tiny-random-FalconForCausalLM",
- [x] "hf-internal-testing/tiny-random-Starcoder2ForCausalLM",
- [x] "hf-internal-testing/tiny-random-GPTJForCausalLM",
- [x] "hf-internal-testing/tiny-random-StableLmForCausalLM",
- [x] "hf-internal-testing/tiny-random-LlamaForCausalLM",
- [x] "hf-internal-testing/tiny-random-MistralForCausalLM",
- [x] "hf-internal-testing/tiny-random-OPTForCausalLM",
- [x] "hf-internal-testing/tiny-random-PhiForCausalLM",
- [x] "hf-internal-testing/tiny-random-StableLmForCausalLM",
- [x] "facebook/opt-125m",
- [x] "llama2",
- [x] "bigcode/starcoder2-7b"
- [ ] "mosaicml/mpt-7b-chat" (FAILED both py/c++) - acceptable for this
PR
Issue: RuntimeError: Check '(axis_range_min <= axis) && (axis <=
axis_range_max)' failed at src/core/src/validation_util.cpp:386:
Concat Parameter axis 2 out of the tensor rank range [0, 0].
- [x] _means, that the response to the dedicated prompt is the same for
the py and c++ transformations._
### Tickets:
- *CVS-138664*
---------
Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
Co-authored-by: Andrii Staikov <andrii.staikov@intel.com>
### Details:
- Added ROIAlignRotated definition to core
- Refactored tests of ROIAlign shape inference info - to use the same
tests for ROIAlignRotated
### Tickets:
- *CVS-115297*
---------
Co-authored-by: Pawel Raasz <pawel.raasz@intel.com>
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
### Details:
- Limit NF4 conversion to FP16 -> NF4 in convert operator as values are
correctly quantized, Not support NF4 conversion to/from other types.
### Tickets:
- [CVS-135304](https://jira.devtools.intel.com/browse/CVS-135304)
### Details:
Updated ReadValue, Loop ref implementations
Added a new functionality to template plugin to collect Variables inside
Multisubgraph ops
Added a new functional test
### Tickets:
- *CVS-135198*
### Details:
- Add LP specializations for cast, fill, write data.
- Remove not used private members.
- Move public non-template members to cpp.
- Optimize bin-size (Ubuntu 20, Release, gcc11):
- No LTO around 421 KiB
- LTO around 10.6 KiB
- Add common elements byte size util function.
- Prepare to use new type like u2, u3, u6
### Tickets:
- Part of
[CVS-119224](https://jira.devtools.intel.com/browse/CVS-119224)
### Details:
- Initialization of OV Opset15
(Discussed internally and agreed to introduce new opset in 2024.2)
To provide more agile approach, this PR has been updated with an idea
(proposed by @mmikolajcz) to include only new operators within the new
Opset, and fill it up at the end of the development period (not
necessarily per release).
This should lead to infrequent introduction of further opsets.
\* Currently common opsets tests rely on Parameter/ShapeOf/Convert in
every opset, so they were added.
But this approach for opset testing should be revised.
### Tickets:
- 138268, 131683
### Details:
- Compressed int4 weight is now supported with OneDNN
- transpose.cpp change for int4 support from ngraph transform is a
temporal solution. To be handled from GPU plugin.
### Tickets:
- 124115
### Details:
Fix segfault in Constant when destroy not initialized string buffer:
- Initialize buffer in Constant before exception throw to avoid
segfault.
- Improve default ctor of` StringAlignedBuffer`.
- Set only required members in `SharedStringAlignedBuffer`'s dtor.
- Add unit test for `StringAlignedBuffer`.
- Use template `get_ptr` instead cast.
- Optimize assertion error message.
### Tickets:
- Relate to
[CVS-119224](https://jira.devtools.intel.com/browse/CVS-119224)
### Details:
- *Optimize second token latency for PagedAttention with multi-query
when running bf16:*
- change block_size from 1 to 16
- amx fast path for q*k
- fast path for k*v
- *depends: https://github.com/ilya-lavrenov/vllm/pull/21*
### Tickets:
- *ticket-id*
---------
Co-authored-by: Li, Tingqian <tingqian.li@intel.com>
Co-authored-by: dmitrygo <dmitry.gorokhov@intel.com>
### Details:
- Add u8 support in op evaluate of logical_not. It is called in cpu_impl
of GPU plugin and GPU plugin is using u8 as boolean from transformed
graph.
### Tickets:
- 124503
### Details:
- Add new types `u2, u3, u6` to Convert operator.
- Expand `element::Iterator` to support NF4 and `BitProxy` to support
NF4 conversion.
- Update tensor to calculate correctly byte size for `u3, u6` types.
- Fix NF4 <-> conversion to always use byte pack/unpack and quantization
when convert to/from floating point. In future the conversion for NF4
will be limited to f32 -> NF4.
### Tickets:
- [CVS-127000](https://jira.devtools.intel.com/browse/CVS-127000)
- [CVS-128024](https://jira.devtools.intel.com/browse/CVS-128024)
---------
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
### Details:
- Support LLM inference
|device|
|--|
|HETERO:CPU|
|HETERO:GPU|
|HETERO:CPU,GPU|
|HETERO:GPU,CPU|
|HETERO:GPU.0,GPU.1|
|HETERO:GPU.0,GPU.1,CPU|
|HETERO:GPU.0,GPU.1,GPU.2|
- Use the `ov::hint::model_distribution_policy` property in
[PR23077](https://github.com/openvinotoolkit/openvino/pull/23077/)
- Use host memory in different subgraphs when input/output data exchange
- Mask supported nodes and unsupported nodes to Subgraph in graph, and
query model use subgraph, keep the model in query_model same as compile
- Add a property `ov::query_model_ratio` to set the percentage of the
model can be queried during query model
- Improve performance on some LLM with large parameter by split the
model to different devices, the number of split models should be
smaller, reduce communication usage between multiple devices
### Tickets:
- *CVS-133258*
---------
Co-authored-by: Nadezhda <nadezhda.ageeva@intel.com>
Co-authored-by: Shen, Wanglei <wanglei.shen@intel.com>
Co-authored-by: yanlan song <bell.song@intel.com>
### Details:
- *Fixed CPU transformation that decomposes `Convert[..->bool]` to `Abs,
Ceiling, Convert[..->u8]` - added `Min(Abs, UINT8_MAX)` to avoid
overflows*
- *If `Convert[..->bool]` is inserted on output, the Plugin applies the
transformation which is described above and inserts `Convert[u8->bool]`
on output to align model output precisions. The `Convert[u8->bool]` is
executed by plugin, reference `cpu_convert`. The PR added `bool` lowest
and max bounds in `cpu_convert` to have normalized output data - [0, 1]
(not [0..256])*
- *Renamed `sequnce_generator.hpp` -> `sequence_generator.hpp` as
@itikhono asked 😃*
- *Added specific input data generation for `Convert[..->boolean]`*
- *Added `fill_data_boolean` to generate only 0, 1 values for boolean
input data*
### Tickets:
- *136130, 122321*
### Details:
- Improved Inverse testing suite for dynamic shapes in the CPU plugin
- Introduced random tests to PT FE Inerse testing suite
- Added LU decomposition pseudocode to spec
- Removed numpy import from opset/opset14.py
- Moved reference inverse outside inverse namespace and removed inverse
namespace
- Added label propagation tests
- Changed bf16 threshold restriction
- Removed dynamic rank check as it's already present in CPU plugin
- Simplified check for B => P[i] == j
- Did NOT use std::copy_n as ov::bfloat16 is missing difference_value to
use in std implementations
### Tickets:
- 133090
---------
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
Co-authored-by: Pawel Raasz <pawel.raasz@intel.com>
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
### Details:
Removed code that makes friendly names unique from Serialization and a
name uniqueness check from Deserializator.
Enabled the mode of ResolveNameCollisions transformation to uniqueize
all friendly names, not only autogenerated in Frontends
### Tickets:
- *CVS-131567*
---------
Co-authored-by: Evgenya Nugmanova <evgeniia.nugmanova@intel.com>
Co-authored-by: Andrei Kochin <andrei.kochin@intel.com>
[TRANSFORMATIONS] Create python binding for pattern::Optional
Expose the C++ op::pattern::Optional to Python in order to
simplify patterns creation.
Cover the functionality with the dedicated tests.
### Tickets:
CVS-133523
Signed-off-by: Andrii Staikov <andrii.staikov@intel.com>
---------
Signed-off-by: Andrii Staikov <andrii.staikov@intel.com>
[TRANSFORMATIONS] Fix Optional to match even with no inputs
### Details:
The Optional pattern type may create a wrong pattern to match if no
inputs are provided to the Optional node. If no inputs present to the
Optional type, it will not create an alternative branch(es) to check
against resulting in the incorrect matching.
Fix that by adding a check for the number of inputs being 0.
Do a minor refactoring/renaming for the readability purposes.
### Tickets:
CSV-133523
Signed-off-by: Andrii Staikov <andrii.staikov@intel.com>
---------
Signed-off-by: Andrii Staikov <andrii.staikov@intel.com>
### Details:
- Introduce new low precision types `u2`, `u3`, `u6`.
- Introduce `ov::element::Iterator` for low precision types like `u1,
u2, u3, u4, i4, u6`:
- Gives pointer like access to low precision values in Tensor,
containers etc.
- Can be used by STL algorithms to access data in unified algorithms for
data manipulation.
- Can be used in Constant, Convert operators to replace duplicate
implementations for accessing low precision data (bin-size reduction).
- Can be used for operator reference implementation or plugin if there
is no hardware specific solution.
### Tickets:
- [CVS-126998](https://jira.devtools.intel.com/browse/CVS-126998)
- Part of
[CVS-128024](https://jira.devtools.intel.com/browse/CVS-128024)
New PR due to merge
conflicts(https://github.com/openvinotoolkit/openvino/pull/22917)
Updated the copyright year from 2018-2023 to 2018-2024 in all openvino
files
Ref. to script: CVS-101144
Command used:
```bash
git grep -lz '2018-2023 Intel Corporation' | xargs -0 sed -i '' -e 's/2018-2023 Intel Corporation/2018-2024 Intel Corporation/g'
```
RandomUniform has constant_fold disabled and also it yields very
different results if given the same inputs with different types (e.g.
f32 vs. f16). Therefore it shouldn't have input precision converted.
Also make "requires_precision_conversion" attribute not copyable to make
sure it's not copied to other nodes during copy_runtime_info.
Ticket: CVS-134391
Ivan Tikhonov