**Details:** EliminateLoopInputsOutputs transformation helps to avoid an
isolated sub-graphs `Parameter->Result` in a body graph of `Loop` and
`TensorIterator`. Such sub-graphs are useless and can by-pass `Loop`
operation.
**Ticket:** 142440
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
Co-authored-by: Evgeny Kotov <evgeny.kotov@intel.com>
Co-authored-by: Tikhonov Ivan <ivan.tikhonov@intel.com>
### Details:
- Optimize RMS kernel by subgroup block IO
- Compute maximum possible LWS and the number of data in subgroups
- Add test cases which covers leftover handling and unaligned data shape
### Tickets:
- 141845
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
### Details:
- `ov::intel_npu::create_executor` is declared a supported property only
for the `intel_npu::CompiledModel` class. The current PR adds an entry
inside `intel_npu::Plugin` as well.
### Tickets:
- *CVS-143060*
### Details:
- Change the default softmax accumulator type from FP16 to FP32 (this
doesn't affect performance, as well as accuracy at first glance;
however, let's stick to higher precision here)
- Applied review comments from PR
https://github.com/openvinotoolkit/openvino/pull/24466
- Minor internal operations refactoring (SDPA and IndirectSDPA)
- Add new tests for GQA optimization and for indirect inputs
### Details:
- Changes needed to enable Squeeze (and models) with axes pointing to
non-squeezable dims (currently ov throws an error for such case) -
considered as backward compatible change
- If agreed to update functionality within the same version of the op,
actual changes are needed only in shape infer related functions to cover
**Template, CPU and GPU plugins**:
- src/core/shape_inference/include/squeeze_shape_inference.hpp
- src/plugins/intel_cpu/src/shape_inference/custom/reshape.cpp
- The rest of the changes are test related.
- Pytorch FE Squeeze test enabled - passed
### Tickets:
- 142387
**Details:** In pre-commit we observe sporadic failure of Rint test on
arm
**Tickets:** TBD
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
### Details:
- *introduce performance hint and threads scheduling in CPU inference*
### Tickets:
- *CVS-138834*
---------
Co-authored-by: Sun Xiaoxia <xiaoxia.sun@intel.com>
### Details:
- WA of out-of-bound access for fc kernel.
- Use OneDNN FC kernel for fp16 FC when possible. It hides the issue
because the target platform can use OneDNN.
- For shape-agnostic kernel, TILE_B=8 is chosen. It causes out-of-bound
memory access for LLM 2nd token.
### Tickets:
- 142277
### Details:
- *United `LoopEndStatic` and `LoopEndDynamic` into one node `LoopEnd`
to avoid extra conditions in the code and improve performance since some
pointer data shifts might be known and compiled in JIT code*
- *Removed dynamic aarch64 loop emitters since they don't work anyway
CVS-141550*
- *Added support dynamism to `IdentifyBuffers` and
`DefineBufferClusters`. It's not efficient algorithm since we don't know
exact values of data pointer shifts and cannot be sure that they will be
proportionally in runtime. It should be implemented as the separate
feature based on some judgments, for example*
### Tickets:
- *141268*
### Prerequisites:
- *https://github.com/openvinotoolkit/openvino/pull/21922*
### Details:
- SliceToStridedSlice was moved from MOC to Common transformations.
- Updated RoPE fusion patterns
- 2 new transformations: EliminateSlice, SliceSequenceToSingleSlice
- Updated EliminateStridedSlice transformation to support int32_max
case, deleted transformation duplicate from CPU
- Added new unit tests
Tested locally with these models:
- [x] Means that Model compilation passed, and all RoPE related
transformations were applied in the same order
- [x] hf-internal-testing/tiny-random-StableLmForCausalLM
- [x] hf-internal-testing/tiny-random-FalconForCausalLM
- [x] hf-internal-testing/tiny-random-Starcoder2ForCausalLM
- [x] hf-internal-testing/tiny-random-LlamaForCausalLM
- [x] hf-internal-testing/tiny-random-GPTNeoXForCausalLM
- [x] hf-internal-testing/tiny-random-GPTJForCausalLM
- [x] hf-internal-testing/tiny-random-CodeGenForCausalLM
- [x] hf-internal-testing/tiny-random-MistralForCausalLM
- [x] hf-internal-testing/tiny-random-PhiForCausalLM
- [x] Qwen/Qwen1.5-7B
- [x] THUDM/chatglm3-6b
- [x] EleutherAI/gpt-neox-20b
- [x] google/gemma-2b-it
- [x] EleutherAI/gpt-j-6b
- [x] meta-llama/Meta-Llama-3-8B
- [x] mistralai/Mistral-7B-v0.1
### Tickets:
- *CVS-126971*
### Details:
As a result, you can use this code for most of your needs:
- in a Jupiter notebook
- as a cmd tool
- with model weights being far away
```
def configure_visualizer(enable=True, max_const_elements=7):
# parameter max_const_elements manipulates the number of first elements of a constant that will be serialized
# to svg. Put 0 if the model is on a share
import os
envs_to_set = [
'OV_VISUALIZE_TREE_IO',
'OV_VISUALIZE_TREE_OUTPUT_SHAPES',
'OV_VISUALIZE_TREE_OUTPUT_TYPES',
'OV_VISUALIZE_TREE_EDGE_LABELS',
# symbolic information and partial values
'OV_VISUALIZE_APPLY_SYMBOLIC_PROPAGATION',
'OV_VISUALIZE_PARTIAL_VALUES_AND_LABELS'
]
for e in envs_to_set:
os.environ[e] = '1' if enable else '0'
os.environ["OV_VISUALIZE_TREE_CONST_MAX_ELEMENTS"] = str(max_const_elements) if enable else "7"
def serialize_model_to_svg(model, output_name='serialized_model.svg'):
from openvino.runtime.passes import VisualizeTree
configure_visualizer(True)
VisualizeTree(output_name).run_on_model(model)
configure_visualizer(False)
def convert_model_to_svg(input_name, output_name=None):
import openvino as ov
core = ov.Core()
model = core.read_model(input_name)
if output_name is None:
output_name = os.path.splitext(os.path.basename(input_name))[0] + '.svg'
serialize_model_to_svg(model, output_name)
if __name__ == "__main__":
import sys, os
arguments = sys.argv[1:]
if len(arguments) not in [1, 2]:
print(f"Usage {sys.executable} {__file__} /path/to/model [output_file_name.svg]")
sys.exit(1)
output_file = os.path.splitext(os.path.basename(arguments[0]))[0] + '.svg' if len(arguments) == 1 else arguments[1]
convert_model_to_svg(arguments[0], output_file)
```
SharedPtr mutual reference fix
### Details:
- use lambda function replacing "Return" function of Gmock in ON_CALL
for *mockIExeNet.get()
- use lambda function replacing "Return" function of Gmock in ON_CALL
for *mockIExeNetActual.get()
"Retrun" function would cause increment of use account of
inferReqInternal and inferReqInternalActual, thereby causing mutual
reference of SharedPtr between mockIExeNet and inferReqInternal or
mockIExeNetActual and inferReqInternalActual.
### Ticket:
- [CVS-134423](https://jira.devtools.intel.com/browse/CVS-134423)
### Details:
Add tests for the SPDAToPagedAttention transformation to Precommit
1) Check if PagedAttentionExtension node appeared in the model after the
transformation
2) Check 2 last dimensions of 'key_cache.' and 'value_cache.' inputs to
be static.
### Tickets:
- [CVS-138943](https://jira.devtools.intel.com/browse/CVS-138943)
Signed-off-by: Andrii Staikov <andrii.staikov@intel.com>
### Details:
- Windows std::wstring convert to std::string some unicode will be ???,
such as L"unicode_Яㅎあ" -> "unicode_???"
### Tickets:
- CVS-141444
- CVS-141729
---------
Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
### Details:
- Reduce the condition for not-using usm-host input buffer
- This is to reduce the overhead in next-gen GPU performance where
usm_host is the input buffer
### Tickets:
- 141493
### Details:
This PR is a copy of
https://github.com/openvinotoolkit/openvino/pull/24757 from release
branch, it includes:
- Enabled SDPA by default
- Updated SDPA decomposition rule to cover only well-checked cases
- Updated functional tests accordingly
ze_device_properties_t has pNext pointer and if it's not nullptr then
it's assumed that it points to a different structure. pNext being
uninitialized can lead to a crash during zeDeviceGetProperties.
### Details:
- Fixed Densify reading implementation
- Added test for Densify
- Added test for Dequantize
- Removed unnecessary model test
### Tickets:
- 127943
---------
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
Andrei Kashchikhin