diff --git a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training.rst b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training.rst index c1796c87113..c98c4547b8f 100644 --- a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training.rst +++ b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training.rst @@ -48,11 +48,6 @@ To install the latest released version via pip manager run the following command pip install nncf -.. note:: - - To install with specific frameworks, use the `pip install nncf[extras]` command, where extras is a list of possible extras, for example, `torch`, `tf`, `onnx`. - - To install the latest NNCF version from source, follow the instruction on `GitHub `__. .. note:: diff --git a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training-pytorch.rst b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training-pytorch.rst new file mode 100644 index 00000000000..91b405d43e9 --- /dev/null +++ b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training-pytorch.rst @@ -0,0 +1,72 @@ +Quantization-aware Training (QAT) with PyTorch +=============================================== + +Below are the steps required to integrate QAT from NNCF into a training script written with +PyTorch: + + +1. Apply Post Training Quantization to the Model +################################################## + +Quantize the model using the :doc:`Post-Training Quantization <../quantizing-models-post-training/basic-quantization-flow>` method. + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py + :language: python + :fragment: [quantize] + + +2. Fine-tune the Model +######################## + +This step assumes applying fine-tuning to the model the same way it is done for the baseline model. For QAT, it is required to train the model for a few epochs with a small learning rate, for example, 10e-5. +Quantized models perform all computations in floating-point precision during fine-tuning by modeling quantization errors in both forward and backward passes. + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py + :language: python + :fragment: [tune_model] + + +.. note:: + The precision of weights transitions to INT8 only after converting the model to OpenVINO Intermediate Representation. + You can expect a reduction in model footprint only for that format. + + +These steps outline the basics of applying the QAT method from the NNCF. However, in some cases, it is required to save/load model +checkpoints during training. Since NNCF wraps the original model with its own object, it provides an API for these needs. + +3. (Optional) Save Checkpoint +#################################### + +To save a model checkpoint, use the following API: + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py + :language: python + :fragment: [save_checkpoint] + + +4. (Optional) Restore from Checkpoint +################################################ + +To restore the model from checkpoint, use the following API: + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py + :language: python + :fragment: [load_checkpoint] + + +Deploying the Quantized Model +############################### + +The model can be converted into the OpenVINO Intermediate Representation (IR) if needed, compiled, and run with OpenVINO without any additional steps. + +.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_torch.py + :language: python + :fragment: [inference] + +For more details, see the corresponding :doc:`documentation <../../running-inference>`. + +Examples +#################### + +* `Quantization-aware Training of Resnet18 PyTorch Model `__ +* `Quantization-aware Training of STFPM PyTorch Model `__ diff --git a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training-tensorflow.rst b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training-tensorflow.rst new file mode 100644 index 00000000000..41a2ea61521 --- /dev/null +++ b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training-tensorflow.rst @@ -0,0 +1,112 @@ +Quantization-aware Training (QAT) with TensorFlow +=================================================== + +Below are the steps required to integrate QAT from NNCF into a training script written with TensorFlow: + +.. note:: + Currently, NNCF for TensorFlow supports optimization of the models created using Keras + `Sequential API `__ or + `Functional API `__. + +1. Import NNCF API +######################## + +Add NNCF-related imports in the beginning of the training script: + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py + :language: python + :fragment: [imports] + +2. Create NNCF Configuration +#################################### + +Define NNCF configuration which consists of model-related parameters (the ``"input_info"`` section) and parameters +of optimization methods (the ``"compression"`` section). For faster convergence, it is also recommended to register a dataset object +specific to the DL framework. The data object will be used at the model creation step to initialize quantization parameters. + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py + :language: python + :fragment: [nncf_congig] + + +3. Apply Optimization Methods +#################################### + +Wrap the original model object with the ``create_compressed_model()`` API using the configuration +defined in the previous step. This method returns a so-called compression controller and a wrapped model that can be used the +same way as the original model. Optimization methods are applied at this step, so that the model +undergoes a set of corresponding transformations and contains additional operations required for optimization. In case of QAT, the compression controller object is used for model export and, optionally, in distributed training as demonstrated below. + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py + :language: python + :fragment: [wrap_model] + + +4. Fine-tune the Model +#################################### + +This step assumes applying fine-tuning to the model the same way it is done for the baseline model. For QAT, it is required to train the model for a few epochs with a small learning rate, for example, 10e-5. In principle, +you can skip this step, meaning that the post-training optimization will be applied to the model. + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py + :language: python + :fragment: [tune_model] + + +5. Multi-GPU Distributed Training +#################################### + +In the case of distributed multi-GPU training (not DataParallel), call ``compression_ctrl.distributed()`` before fine-tuning. This informs optimization methods to make adjustments to function in the distributed mode. + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py + :language: python + :fragment: [distributed] + + +.. note:: + The precision of weights transitions to INT8 only after converting the model to OpenVINO Intermediate Representation. + You can expect a reduction in model footprint only for that format. + + +These steps outline the basics of applying the QAT method from the NNCF. However, in some cases, it is required to save/load model +checkpoints during training. Since NNCF wraps the original model with its own object, it provides an API for these needs. + +6. (Optional) Save Checkpoint +#################################### + +To save a model checkpoint, use the following API: + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py + :language: python + :fragment: [save_checkpoint] + + +7. (Optional) Restore from Checkpoint +################################################ + +To restore the model from checkpoint, use the following API: + +.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py + :language: python + :fragment: [load_checkpoint] + + +For more details on saving/loading checkpoints in the NNCF, see the corresponding `NNCF documentation `__. + +Deploying quantized model +######################### + +The model can be converted into the OpenVINO Intermediate Representation (IR) if needed, compiled and run with OpenVINO. +No extra steps or options are required. + +.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py + :language: python + :fragment: [inference] + +For more details, see the corresponding :doc:`documentation <../../running-inference>`. + +Examples +#################### + +* `Quantizing TensorFlow model with NNCF `__ + diff --git a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training.rst b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training.rst index cce63315939..f5ec455a7e2 100644 --- a/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training.rst +++ b/docs/articles_en/openvino-workflow/model-optimization-guide/compressing-models-during-training/quantization-aware-training.rst @@ -1,8 +1,12 @@ -.. {#qat_introduction} - Quantization-aware Training (QAT) ================================= +.. toctree:: + :maxdepth: 1 + :hidden: + + Quantization-aware Training with PyTorch + Quantization-aware Training with TensorFlow Introduction #################### @@ -12,223 +16,6 @@ degradation caused by quantization. In fact, this is the most accurate quantizat apply QAT from the Neural Network Compression Framework (NNCF) to get 8-bit quantized models. This assumes that you are knowledgeable in Python programming and familiar with the training code for the model in the source DL framework. -Using NNCF QAT -#################### - -Here, we provide the steps that are required to integrate QAT from NNCF into the training script written with -PyTorch or TensorFlow 2: - -.. note:: - Currently, NNCF for TensorFlow 2 supports optimization of the models created using Keras - `Sequential API `__ or - `Functional API `__. - -1. Import NNCF API -++++++++++++++++++++ - -In this step, you add NNCF-related imports in the beginning of the training script: - -.. tab-set:: - - .. tab-item:: PyTorch - :sync: pytorch - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py - :language: python - :fragment: [imports] - - .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py - :language: python - :fragment: [imports] - -2. Create NNCF configuration -++++++++++++++++++++++++++++ - -Here, you should define NNCF configuration which consists of model-related parameters (``"input_info"`` section) and parameters -of optimization methods (``"compression"`` section). For faster convergence, it is also recommended to register a dataset object -specific to the DL framework. It will be used at the model creation step to initialize quantization parameters. - -.. tab-set:: - - .. tab-item:: PyTorch - :sync: pytorch - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py - :language: python - :fragment: [nncf_congig] - - .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py - :language: python - :fragment: [nncf_congig] - - -3. Apply optimization methods -+++++++++++++++++++++++++++++ - -In the next step, you need to wrap the original model object with the ``create_compressed_model()`` API using the configuration -defined in the previous step. This method returns a so-called compression controller and a wrapped model that can be used the -same way as the original model. It is worth noting that optimization methods are applied at this step so that the model -undergoes a set of corresponding transformations and can contain additional operations required for the optimization. In -the case of QAT, the compression controller object is used for model export and, optionally, in distributed training as it -will be shown below. - -.. tab-set:: - - .. tab-item:: PyTorch - :sync: pytorch - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py - :language: python - :fragment: [wrap_model] - - .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py - :language: python - :fragment: [wrap_model] - - -4. Fine-tune the model -++++++++++++++++++++++ - -This step assumes that you will apply fine-tuning to the model the same way as it is done for the baseline model. In the -case of QAT, it is required to train the model for a few epochs with a small learning rate, for example, 10e-5. In principle, -you can skip this step which means that the post-training optimization will be applied to the model. - -.. tab-set:: - - .. tab-item:: PyTorch - :sync: pytorch - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py - :language: python - :fragment: [tune_model] - - .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py - :language: python - :fragment: [tune_model] - - - -5. Multi-GPU distributed training -+++++++++++++++++++++++++++++++++ - -In the case of distributed multi-GPU training (not DataParallel), you should call ``compression_ctrl.distributed()`` before -the fine-tuning that will inform optimization methods to do some adjustments to function in the distributed mode. - -.. tab-set:: - - .. tab-item:: PyTorch - :sync: pytorch - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py - :language: python - :fragment: [distributed] - - .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py - :language: python - :fragment: [distributed] - -6. Export quantized model -+++++++++++++++++++++++++ - -When fine-tuning finishes, the quantized model can be exported to the corresponding format for further inference: ONNX in -the case of PyTorch and frozen graph - for TensorFlow 2. - -.. tab-set:: - - .. tab-item:: PyTorch - :sync: pytorch - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py - :language: python - :fragment: [export] - - .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py - :language: python - :fragment: [export] - - -.. note:: - The precision of weights gets INT8 only after the step of model conversion to OpenVINO Intermediate Representation. - You can expect the model footprint reduction only for that format. - - -These were the basic steps to applying the QAT method from the NNCF. However, it is required in some cases to save/load model -checkpoints during the training. Since NNCF wraps the original model with its own object it provides an API for these needs. - -7. (Optional) Save checkpoint -+++++++++++++++++++++++++++++ - -To save model checkpoint use the following API: - -.. tab-set:: - - .. tab-item:: PyTorch - :sync: pytorch - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py - :language: python - :fragment: [save_checkpoint] - - .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py - :language: python - :fragment: [save_checkpoint] - - -8. (Optional) Restore from checkpoint -+++++++++++++++++++++++++++++++++++++ - -To restore the model from checkpoint you should use the following API: - -.. tab-set:: - - .. tab-item:: PyTorch - :sync: pytorch - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py - :language: python - :fragment: [load_checkpoint] - - .. tab-item:: TensorFlow 2 - :sync: tensorflow-2 - - .. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py - :language: python - :fragment: [load_checkpoint] - - -For more details on saving/loading checkpoints in the NNCF, see the following `documentation `__. - -Deploying quantized model -######################### - -The quantized model can be deployed with OpenVINO in the same way as the baseline model. No extra steps or options are -required in this case. For more details, see the corresponding :doc:`documentation <../../running-inference>`. - -Examples -#################### - -* `Quantizing PyTorch model with NNCF `__ - -* `Quantizing TensorFlow model with NNCF `__ +:doc:`Quantization-aware Training with PyTorch ` +:doc:`Quantization-aware Training with TensorFlow ` \ No newline at end of file diff --git a/docs/articles_en/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.rst b/docs/articles_en/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.rst index de0b0f96cc0..2b2136c6a25 100644 --- a/docs/articles_en/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.rst +++ b/docs/articles_en/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.rst @@ -106,7 +106,7 @@ See the `example section <#examples-of-how-to-apply-nncf-post-training-quantizat After that the model can be converted into the OpenVINO Intermediate Representation (IR) if needed, compiled and run with OpenVINO. -If you have not already installed OpenVINO developer tools, install it with ``pip install openvino-dev``. +If you have not already installed OpenVINO developer tools, install it with ``pip install openvino``. .. tab-set:: diff --git a/docs/articles_en/openvino-workflow/model-optimization.rst b/docs/articles_en/openvino-workflow/model-optimization.rst index 1203b99f648..f2e2eb2dc1d 100644 --- a/docs/articles_en/openvino-workflow/model-optimization.rst +++ b/docs/articles_en/openvino-workflow/model-optimization.rst @@ -23,7 +23,7 @@ Model optimization is an optional offline step of improving the final model perf .. note:: OpenVINO also supports optimized models (for example, quantized) from source frameworks such as PyTorch, TensorFlow, and ONNX (in Q/DQ; Quantize/DeQuantize format). No special steps are required in this case and optimized models can be converted to the OpenVINO Intermediate Representation format (IR) right away. -Post-training Quantization is the fastest way to optimize an arbitrary DL model and should be applied first, but it is limited in terms of achievable accuracy-performance trade-off. The recommended approach to obtain OpenVINO quantized model is to convert a model from original framework to ``ov.Model`` and ensure that the model works correctly in OpenVINO, for example, by calculating the model metrics. Then, ``ov.Model`` can be used as input for the ``nncf.quantize()`` method to get the quantized model (see the diagram below). +Post-training Quantization is the fastest way to optimize an arbitrary DL model and should be applied first, but it is limited in terms of achievable accuracy-performance trade-off. The recommended approach to obtain OpenVINO quantized model is to convert a model from original framework to ``ov.Model`` and ensure that the model works correctly in OpenVINO, for example, by calculating the model metrics. Then, ``ov.Model`` can be used as input for the ``nncf.quantize()`` method to get the quantized model or as input for the ``nncf.compress_weights()`` method to compress weights of Large Language Models (see the diagram below). In case of unsatisfactory accuracy or performance after Post-training Quantization, Training-time Optimization can be used as an option. diff --git a/docs/optimization_guide/nncf/code/qat_torch.py b/docs/optimization_guide/nncf/code/qat_torch.py index 121417e4ab0..f80a7e8f9ae 100644 --- a/docs/optimization_guide/nncf/code/qat_torch.py +++ b/docs/optimization_guide/nncf/code/qat_torch.py @@ -1,53 +1,29 @@ # Copyright (C) 2018-2024 Intel Corporation # SPDX-License-Identifier: Apache-2.0 -#! [imports] -import torch -import nncf # Important - should be imported right after torch -from nncf import NNCFConfig -from nncf.torch import create_compressed_model, register_default_init_args -#! [imports] - -#! [nncf_congig] -nncf_config_dict = { - "input_info": {"sample_size": [1, 3, 224, 224]}, # input shape required for model tracing - "compression": { - "algorithm": "quantization", # 8-bit quantization with default settings - }, -} -nncf_config = NNCFConfig.from_dict(nncf_config_dict) -nncf_config = register_default_init_args(nncf_config, train_loader) # train_loader is an instance of torch.utils.data.DataLoader -#! [nncf_congig] - -#! [wrap_model] +#! [quantize] model = TorchModel() # instance of torch.nn.Module -compression_ctrl, model = create_compressed_model(model, nncf_config) -#! [wrap_model] - -#! [distributed] -compression_ctrl.distributed() # call it before the training loop -#! [distributed] +model = nncf.quantize(model, ...) +#! [quantize] #! [tune_model] ... # fine-tuning preparations, e.g. dataset, loss, optimizer setup, etc. # tune quantized model for 5 epochs as the baseline for epoch in range(0, 5): - compression_ctrl.scheduler.epoch_step() # Epoch control API - for i, data in enumerate(train_loader): - compression_ctrl.scheduler.step() # Training iteration control API ... # training loop body #! [tune_model] #! [export] -compression_ctrl.export_model("compressed_model.onnx") -#! [export] +# example_input is an example input to make it possible to trace the model +torch.onnx.export(quantized_model, example_input, './compressed_model.onnx') +#! [export] #! [save_checkpoint] checkpoint = { 'state_dict': model.state_dict(), - 'compression_state': compression_ctrl.get_compression_state(), + 'nncf_config': model.nncf.get_config(), ... # the rest of the user-defined objects to save } torch.save(checkpoint, path_to_checkpoint) @@ -55,8 +31,8 @@ torch.save(checkpoint, path_to_checkpoint) #! [load_checkpoint] resuming_checkpoint = torch.load(path_to_checkpoint) -compression_state = resuming_checkpoint['compression_state'] -compression_ctrl, model = create_compressed_model(model, nncf_config, compression_state=compression_state) -state_dict = resuming_checkpoint['state_dict'] +nncf_config = resuming_checkpoint['nncf_config'] +quantized_model = nncf.torch.load_from_config(model, nncf_config, example_input) +state_dict = resuming_checkpoint['state_dict'] model.load_state_dict(state_dict) #! [load_checkpoint]