natas_hw
/
openvino
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix grammar (#22457)

This commit is contained in:
Zlobin Vladimir 2024-01-30 01:21:31 +04:00 committed by GitHub
parent d9faf192a6
commit ec4f36b015
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/articles_en/openvino_workflow/gen_ai.rst
View File

@ -139,8 +139,8 @@ The optimized model can be saved as usual with a call to ``save_pretrained()``.
with `GPTQ <https://github.com/PanQiWei/AutoGPTQ>`__. In this case, there is no need for an additional model optimization step because model conversion will automatically preserve the INT4 optimization results, allowing model inference to benefit from it.
Another optimization that is applied by default when using ``OVModelForCausalLM`` class is transformation of the model to a stateful form.
This transformation further improves inference performance and decreases amount of allocated runtime memory in long running text generation scenarious.
It is achieved by hiding inputs and outputs of the model that represent past KV-cache tensors, and handling them inside the model in a more efficient way.
This transformation further improves inference performance and decreases amount of allocated runtime memory in long running text generation scenarios.
It is achieved by hiding inputs and outputs of the model that represent past KV-cache tensors and handling them inside the model in a more efficient way.
This feature is activated automatically for a wide range of supported text generation models, keeping not supported models in a regular, stateless form.
Model usage are identical for stateful and stateless models as long as Optimum-Intel API is used because KV-cache handling is an internal detail of the text-generation API of Transformers library.