### Details:
- By combining move and execute into single function, so
isExecutable() check is performed only once
- It turned out isExecutable() check is not that lightweight. For some
models it adds noticeable (about 5%+) latency overhead. So, the solution
is to group 'execute' and 'toNumaNode' into a single function and
perform the check only once.
- fixes overhead introduced by another fix:
https://github.com/openvinotoolkit/openvino/pull/23849
### Tickets:
- 138220
### Details:
- *Calls to external binary kernels (e.g. Brgemm) are normally backed-in
on the code emission stage*
- *Here we create a mechanism that allows to substitute external kernels
in runtime after the code emission stage.*
- *This is a prerequisite for dynamic MHA, since it allows to recompile
required Brgemm blocks in runtime*
### Tickets:
- *123273*
### Details:
- adding new struct for PCI information (aligned with GPU)
- adding new property to query PCI information (as a RO metric)
- implement query of pci information from driver in zero_device, with
backwards compatibility
### Tickets:
[ - *ticket-id*](https://jira.devtools.intel.com/browse/CVS-138767)
### Details:
- We found a performance issue when multiple `gemm_tiled_opt` kernels
are built as a single program.
- This PR adds a WA to lower the kernel batch size to four in the case
of dynamic models.
### Details:
- *MVN with rank size 3 always requires Reorder and Reshape. Due to this
pattern, too many Reorder may occur when used with Convolution, which
may performance degradation. It stand out in Stable-Diffusion Unet and
Decoder*
- *Convolution performance itself also has the gain in MTL.*
### Tickets:
- *137360*
---------
Signed-off-by: hyunback <hyunback.kim@intel.com>
### Details:
- The first requirement from the attached ticket: "Move call of
transformations to CompiledModel constructor. Since the CompiledModel
constructor requires input and output ports, such ports should be passed
before transformations are called, otherwise we can have issues when
transformation mistakenly changed names of tensors and we have issues
like https://jira.devtools.intel.com/browse/CVS-122932"
- Moving the compilation step inside the `intel_npu::CompiledModel`
constructor led to the divergence of the function signature required by
the two methods calling the constructor:
[`intel_npu::Plugin::compile_model`](60dec94dc4/src/plugins/intel_npu/src/plugin/src/plugin.cpp (L510-L516))
and
[`intel_npu::Plugin::import_model`](60dec94dc4/src/plugins/intel_npu/src/plugin/src/plugin.cpp (L582-L588)).
Thus, the current PR introduces an additional constructor.
### Tickets:
- *CVS-132132*
### Extra Validation:
- PR-10217, NPU plugin repository
### Details:
- *It adds support for handling batching on the plugin side only for
integrated platforms*
- *Handling batching by plugin means splitting batch size on the NPU
plugin side and using concurrent inferences for each size*
- *With this PR batching on the plugin is supported only when it is
found on the 0-th dimension and stateful models are also not supported
with batching on plugin*
- *BATCH_MODE is an internal property, we can choose which in which mode
we prefer to handle the batching. If AUTO is set it will try to handle
it on the plugin side but will fallback on the compiler if it doesn't
meet all the conditions for plugin batching. In the case of PLUGIN, it
will try to compile the networks(find batch and force it to 1 for the
compiler) for plugin batching. Otherwise, if COMPILER batching is used
it will work as until now. No other changes*
Diagram of the process flow:
### Tickets:
- *[E#103116]*
The commit eb025fad1e breaks the build
when `CXXFLAGS` or `CFLAGS` have a specific `-Werror` flag.
For example, when `CXXFLAGS` have `-Werror=format-security`, the build
fails with this message:
```
c++: warning: =format-security: linker input file unused because linking not done
c++: error: =format-security: linker input file not found: No such file or directory
```
That's because every occurrence of the `-Werror` string is being
replaced in the original `CXXFLAGS` and `CFLAGS` variables. So, if they
contain `-Werror=<something>`, it will become just `=<something>`, which
will be an unrecognized option.
The current commit fixes this by appending `-Wno-error` to `CXXFLAGS`
and `CFLAGS` to properly disable errors on warnings.
### Details:
- Simplify comparison operator for `layout` class
- Return layout by reference in more cases instead of copy
- Initialize primitive_inst users once to avoid primitives map access in
some runtime optimizations
- Up to 4ms host overhead reduction in some cases
### Tickets:
- *133954*
### Details:
- *Updated initialization of StreamsExecutor configuration by changing
some parameters*
- *Change the call of ov::threading::IStreamsExecutor::Config() in all
plugins*
### Tickets:
- *CVS-130373*
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
Co-authored-by: Wanglei Shen <wanglei.shen@intel.com>
SD1.5 weight compressed model's Perf issue in a770.
Even though static model, set formats likes dynamic mode. It cause huge
performance issues.
### Tickets:
- *138632*
Signed-off-by: hyunback <hyunback.kim@intel.com>
### Details:
- Update the usage of CPU resources for GPU compilation when CPU as
acceleration device
- Don't prioritize to load dynamic model to CPU, but follow the
device::priorities
- Add logs for compilation time and first inference time
### Tickets:
- CVS-138196
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
### Details:
- *Added brgemm blocking support for bf16 and int8 precisions: in this
case blocking loops are shared between BrgemmCopyB and BrgemmCPU nodes*
- *Reduced allocation shapes of input brgemm buffers in case of low
precision*
### Tickets:
- *CVS-115165*
### Details:
- Register BACKEND_COMPILATION_PARAMS in properties
### Tickets:
- *[EISW-112824](https://jira.devtools.intel.com/browse/EISW-112824)*
The option was added to compiler as part of this ticket. But we also
need this option registered to properties in order to start using it
with single image test
### Details:
- Generalize `UnsqueezeBroadcastReshapeMatmulFusion` pass for non-const
broadcast's input support
- Update test case for `UnsqueezeBroadcastReshapeMatmulFusion` pass
### Tickets:
- 136386
Dynamic convolution with padding run cldnn NOT onednn. cldnn test run
with onednn and it caused can't find kernel selection issue.
### Details:
- *item1*
- *...*
### Tickets:
- *ticket-id*
Signed-off-by: hyunback <hyunback.kim@intel.com>
_Dynamic convolutions with explicit padding run planar bfyx format in
clDNN.
But this limitation is only for clDNN NOT oneDNN. Because unexpected
format selection in dynamic convolution, SD1.5 in platform used dpas
always run clDNN and it caused bad performance._
### Tickets:
- *138632*
---------
Signed-off-by: hyunback <hyunback.kim@intel.com>
### Details:
- Correct tensor shape of AUTO before setting it to selected HW infer
request.
- Remove CPU shape.compatible checks when setting output tensor with
dynamic shape
### Tickets:
- CVS-136756
### Details:
- Currently, in dynamic shape, ofm size is fixed as 2 because it is is
tightly coupled by weight layout (i.e., tile_ofm == 2 requires
os_iyx_osv32 weight format)
- For Vec*Mat case (b==1, used for 2nd+ token generation in LLMs) with
small output size N, this configuration is not good because of the
limited # of gpu threads.
- To increase GPU threads, needed to 1) reduce the ofm size to 1 2) But
still share the weight format with first token case where ofm=2 is good.
- Previously the os_iyx_osv32 weight format was packing int4 weight with
the following order:
k0_f0f16 | k0_f1f17 | .... | k0_f15f31 || k1_f0f16 | k1_f1f17 | ... |
k1_f15f31
- To use common weight format for ofm=1 and ofm=2, implemented a new
weight format for int4 dtype, where the weight is stored int the
following order:
f0_k0k1 | f1_k0k1 | .... | f15_k0k1|| f16_k0k1 | f17_k0k1 | ... |
f31_k0k1
### Tickets:
- 138296
Egor Duplenskii