openvino/docs/optimization_guide/nncf_introduction.md

Neural Network Compression Framework

The Neural Network Compression Framework (NNCF) aims at optimizing Deep Neural Networks (DNN) by means of methods such as quantization and pruning. It provides in-training optimization capabilities, which means that the optimization methods require model fine-tuning or even re-training.

NNCF is distributed as a separate tool but is closely aligned with OpenVINO in terms of supported optimization features and models. It is open source and available on GitHub. The diagram below shows the model optimization workflow, using NNCF.

Features

• Support for optimization of PyTorch and TensorFlow 2.x models.
• Support for various optimization algorithms, applied during a model fine-tuning process to achieve a better trade-off between performance and accuracy:

Compression algorithm	PyTorch	TensorFlow 2.x
8- bit quantization	Supported	Supported
Filter pruning	Supported	Supported
Sparsity	Supported	Supported
Mixed-precision quantization	Supported	Not supported
Binarization	Supported	Not supported

• Stacking of optimization methods, for example: 8-bit quaNtization + Filter Pruning.
• Support for Accuracy-Aware model training pipelines via the Adaptive Compression Level Training and Early Exit Training.
• Automatic, configurable model graph transformation to obtain the compressed model.

  NOTE: Only models created using Sequential or Keras Functional API are supported. Support for TensorFlow models is limited.

• GPU-accelerated layers for faster compressed model fine-tuning.
• Distributed training support.
• Examples of configuration files for each supported compression algorithm.
• Exporting PyTorch compressed models to ONNX checkpoints and TensorFlow compressed models to SavedModel or Frozen Graph format, ready to use with OpenVINO toolkit.
• Git patches for prominent third-party repositories (huggingface-transformers) demonstrating the process of integrating NNCF into custom training pipelines.

Installation

NNCF provides the packages available for installation through the PyPI repository. To install the latest version via pip manager, run the following command:

pip install nncf

Usage examples

NNCF provides various examples and tutorials that demonstrate usage of optimization methods:

Tutorials

• Quantization-aware training of PyTorch model
• Quantization-aware training of TensorFlow model

Samples

• PyTorch:

  • Image Classification sample
  • Object Detection sample
  • Semantic segmentation sample

• TensorFlow samples:

  • Image Classification sample
  • Object Detection sample
  • Instance Segmentation sample

Additional Resources

• Compressed Model Zoo
• NNCF in HuggingFace Optimum
• Post-training optimization