### Details:
- *Complete the remaining work after the
[PR](https://github.com/openvinotoolkit/openvino/pull/24297) is merged.*
- *Add test case.*
### Tickets:
- *141542*
---------
Signed-off-by: xipingya <xiping.yan@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
### Details:
- Aligned build jobs with runner configuration to speed up the build
- added temporary variable data_type to resolve internal compilation
error with the new msvc (19.40)
https://jira.devtools.intel.com/browse/CVS-142993
### Tickets:
- *ticket-id*
### Details:
- *This PR fixes geomean level performance issue caused by
https://github.com/openvinotoolkit/openvino/pull/23180 on platforms with
sufficient cores.*
- *In the PR mensioned above, `parallel_nt` is used for executing
subgraph instead of `parallel_for5d`, becuase the former has performance
advantage for the case where #tasks is more than #threads*. That is
becuase using `parallel_nt`, we can initilize parameters for each
thread, while parameters are initilized for each task in using
`parallel_for5d`. So `parallel_nt` has less overheads.
- *However, for the case where #tasks is less than #threads,
`parallel_nt` brings performance regression. Because compared with
`parallel_for5d`, `parallel_nt` doesn't apply the argument
`tbb::static_partitioner{}` when calling `tbb::parallel_for`. Here we
use `parallel_nt_static` instead to fix the performance regression.*
### Details:
- In case when scale tensor was used by multiple compressed FCs we had
an exception on attempt to insert reorder between them. This patch
changes reorder name so now multiple reorders are inserted for each
Scale+FC pair, and then on later stages we merge them if they do same
tensor transform.
**Details:** EliminateLoopInputsOutputs transformation helps to avoid an
isolated sub-graphs `Parameter->Result` in a body graph of `Loop` and
`TensorIterator`. Such sub-graphs are useless and can by-pass `Loop`
operation.
**Ticket:** 142440
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
Co-authored-by: Evgeny Kotov <evgeny.kotov@intel.com>
Co-authored-by: Tikhonov Ivan <ivan.tikhonov@intel.com>
### Details:
- Optimize RMS kernel by subgroup block IO
- Compute maximum possible LWS and the number of data in subgroups
- Add test cases which covers leftover handling and unaligned data shape
### Tickets:
- 141845
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
### Details:
- `ov::intel_npu::create_executor` is declared a supported property only
for the `intel_npu::CompiledModel` class. The current PR adds an entry
inside `intel_npu::Plugin` as well.
### Tickets:
- *CVS-143060*
### Details:
- Change the default softmax accumulator type from FP16 to FP32 (this
doesn't affect performance, as well as accuracy at first glance;
however, let's stick to higher precision here)
- Applied review comments from PR
https://github.com/openvinotoolkit/openvino/pull/24466
- Minor internal operations refactoring (SDPA and IndirectSDPA)
- Add new tests for GQA optimization and for indirect inputs
### Details:
- Changes needed to enable Squeeze (and models) with axes pointing to
non-squeezable dims (currently ov throws an error for such case) -
considered as backward compatible change
- If agreed to update functionality within the same version of the op,
actual changes are needed only in shape infer related functions to cover
**Template, CPU and GPU plugins**:
- src/core/shape_inference/include/squeeze_shape_inference.hpp
- src/plugins/intel_cpu/src/shape_inference/custom/reshape.cpp
- The rest of the changes are test related.
- Pytorch FE Squeeze test enabled - passed
### Tickets:
- 142387
**Details:** In pre-commit we observe sporadic failure of Rint test on
arm
**Tickets:** TBD
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
### Details:
- *introduce performance hint and threads scheduling in CPU inference*
### Tickets:
- *CVS-138834*
---------
Co-authored-by: Sun Xiaoxia <xiaoxia.sun@intel.com>
### Details:
- WA of out-of-bound access for fc kernel.
- Use OneDNN FC kernel for fp16 FC when possible. It hides the issue
because the target platform can use OneDNN.
- For shape-agnostic kernel, TILE_B=8 is chosen. It causes out-of-bound
memory access for LLM 2nd token.
### Tickets:
- 142277
### Details:
- *United `LoopEndStatic` and `LoopEndDynamic` into one node `LoopEnd`
to avoid extra conditions in the code and improve performance since some
pointer data shifts might be known and compiled in JIT code*
- *Removed dynamic aarch64 loop emitters since they don't work anyway
CVS-141550*
- *Added support dynamism to `IdentifyBuffers` and
`DefineBufferClusters`. It's not efficient algorithm since we don't know
exact values of data pointer shifts and cannot be sure that they will be
proportionally in runtime. It should be implemented as the separate
feature based on some judgments, for example*
### Tickets:
- *141268*
### Prerequisites:
- *https://github.com/openvinotoolkit/openvino/pull/21922*
### Details:
- SliceToStridedSlice was moved from MOC to Common transformations.
- Updated RoPE fusion patterns
- 2 new transformations: EliminateSlice, SliceSequenceToSingleSlice
- Updated EliminateStridedSlice transformation to support int32_max
case, deleted transformation duplicate from CPU
- Added new unit tests
Tested locally with these models:
- [x] Means that Model compilation passed, and all RoPE related
transformations were applied in the same order
- [x] hf-internal-testing/tiny-random-StableLmForCausalLM
- [x] hf-internal-testing/tiny-random-FalconForCausalLM
- [x] hf-internal-testing/tiny-random-Starcoder2ForCausalLM
- [x] hf-internal-testing/tiny-random-LlamaForCausalLM
- [x] hf-internal-testing/tiny-random-GPTNeoXForCausalLM
- [x] hf-internal-testing/tiny-random-GPTJForCausalLM
- [x] hf-internal-testing/tiny-random-CodeGenForCausalLM
- [x] hf-internal-testing/tiny-random-MistralForCausalLM
- [x] hf-internal-testing/tiny-random-PhiForCausalLM
- [x] Qwen/Qwen1.5-7B
- [x] THUDM/chatglm3-6b
- [x] EleutherAI/gpt-neox-20b
- [x] google/gemma-2b-it
- [x] EleutherAI/gpt-j-6b
- [x] meta-llama/Meta-Llama-3-8B
- [x] mistralai/Mistral-7B-v0.1
### Tickets:
- *CVS-126971*
### Details:
As a result, you can use this code for most of your needs:
- in a Jupiter notebook
- as a cmd tool
- with model weights being far away
```
def configure_visualizer(enable=True, max_const_elements=7):
# parameter max_const_elements manipulates the number of first elements of a constant that will be serialized
# to svg. Put 0 if the model is on a share
import os
envs_to_set = [
'OV_VISUALIZE_TREE_IO',
'OV_VISUALIZE_TREE_OUTPUT_SHAPES',
'OV_VISUALIZE_TREE_OUTPUT_TYPES',
'OV_VISUALIZE_TREE_EDGE_LABELS',
# symbolic information and partial values
'OV_VISUALIZE_APPLY_SYMBOLIC_PROPAGATION',
'OV_VISUALIZE_PARTIAL_VALUES_AND_LABELS'
]
for e in envs_to_set:
os.environ[e] = '1' if enable else '0'
os.environ["OV_VISUALIZE_TREE_CONST_MAX_ELEMENTS"] = str(max_const_elements) if enable else "7"
def serialize_model_to_svg(model, output_name='serialized_model.svg'):
from openvino.runtime.passes import VisualizeTree
configure_visualizer(True)
VisualizeTree(output_name).run_on_model(model)
configure_visualizer(False)
def convert_model_to_svg(input_name, output_name=None):
import openvino as ov
core = ov.Core()
model = core.read_model(input_name)
if output_name is None:
output_name = os.path.splitext(os.path.basename(input_name))[0] + '.svg'
serialize_model_to_svg(model, output_name)
if __name__ == "__main__":
import sys, os
arguments = sys.argv[1:]
if len(arguments) not in [1, 2]:
print(f"Usage {sys.executable} {__file__} /path/to/model [output_file_name.svg]")
sys.exit(1)
output_file = os.path.splitext(os.path.basename(arguments[0]))[0] + '.svg' if len(arguments) == 1 else arguments[1]
convert_model_to_svg(arguments[0], output_file)
```
SharedPtr mutual reference fix
### Details:
- use lambda function replacing "Return" function of Gmock in ON_CALL
for *mockIExeNet.get()
- use lambda function replacing "Return" function of Gmock in ON_CALL
for *mockIExeNetActual.get()
"Retrun" function would cause increment of use account of
inferReqInternal and inferReqInternalActual, thereby causing mutual
reference of SharedPtr between mockIExeNet and inferReqInternal or
mockIExeNetActual and inferReqInternalActual.
### Ticket:
- [CVS-134423](https://jira.devtools.intel.com/browse/CVS-134423)
### Details:
Add tests for the SPDAToPagedAttention transformation to Precommit
1) Check if PagedAttentionExtension node appeared in the model after the
transformation
2) Check 2 last dimensions of 'key_cache.' and 'value_cache.' inputs to
be static.
### Tickets:
- [CVS-138943](https://jira.devtools.intel.com/browse/CVS-138943)
Signed-off-by: Andrii Staikov <andrii.staikov@intel.com>
### Details:
- Windows std::wstring convert to std::string some unicode will be ???,
such as L"unicode_Яㅎあ" -> "unicode_???"
### Tickets:
- CVS-141444
- CVS-141729
---------
Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
Xiping Yan