From 642fbc07575d2e5f8bb8987c8724ed97e99a96c7 Mon Sep 17 00:00:00 2001 From: Chen Peter Date: Wed, 21 Feb 2024 23:09:58 +0800 Subject: [PATCH] Add note for GPU default BATCH (#22997) ### Details: - *Note that GPU may not trigger "auto_batch" if model or GPU memory size are not allowed.* - *compile_model requires ov::model for GPU default BATCH* ### Tickets: - ** --------- Signed-off-by: Peter Chen --- .../inference-devices-and-modes/automatic-batching.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/automatic-batching.rst b/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/automatic-batching.rst index c3867da72af..784e8d11269 100644 --- a/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/automatic-batching.rst +++ b/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/automatic-batching.rst @@ -202,6 +202,10 @@ The following are limitations of the current AUTO Batching implementations: - When *forcing* Auto-batching via the "explicit" device notion, make sure that you validate the results for correctness. - Performance improvements happen at the cost of the growth of memory footprint. However, Auto-batching queries the available memory (especially for dGPU) and limits the selected batch size accordingly. +.. note:: + ``BATCH`` device supports GPU by default, but GPU still may not trigger ``auto_batch`` in ``tput`` mode if model or GPU memory size are not allowed. Which means it is required to check ``supported_properties`` of GPU ``tput`` mode ``compiled_model`` before doing any actions (set/get) with ``ov::auto_batch_timeout`` property.
+ To make sure ``BATCH`` device supports GPU by default, ``ov::model`` is required for ``core.compile_model``. A string of model file path to ``core.compile_model`` will be passed to GPU Plugin directly due to performance consideration and without involving ``BATCH``. + Testing Performance with Benchmark_app ######################################