natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add note for GPU default BATCH (#22997) 

### Details:
- *Note that GPU may not trigger "auto_batch" if model or GPU memory
size are not allowed.*
 - *compile_model requires ov::model for GPU default BATCH*

### Tickets:
 - **

---------

Signed-off-by: Peter Chen <peter.chen@intel.com>
This commit is contained in:
Chen Peter 2024-02-21 23:09:58 +08:00 committed by GitHub
parent 5d3f37091a
commit 642fbc0757
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/automatic-batching.rst
View File

@ -202,6 +202,10 @@ The following are limitations of the current AUTO Batching implementations:
- When *forcing* Auto-batching via the "explicit" device notion, make sure that you validate the results for correctness.
- Performance improvements happen at the cost of the growth of memory footprint. However, Auto-batching queries the available memory (especially for dGPU) and limits the selected batch size accordingly.
.. note::
``BATCH`` device supports GPU by default, but GPU still may not trigger ``auto_batch`` in ``tput`` mode if model or GPU memory size are not allowed. Which means it is required to check ``supported_properties`` of GPU ``tput`` mode ``compiled_model`` before doing any actions (set/get) with ``ov::auto_batch_timeout`` property.<br/>
To make sure ``BATCH`` device supports GPU by default, ``ov::model`` is required for ``core.compile_model``. A string of model file path to ``core.compile_model`` will be passed to GPU Plugin directly due to performance consideration and without involving ``BATCH``.
Testing Performance with Benchmark_app
######################################