How to choose the quantization method

[camhpj]

I am trying to do int8 quantization on a pytorch model and am confused as to how to select the accuracy aware method and provide the max accuracy drop. Any help with this would be appreciated.

[alexsu52]

Hi @camhpj,

nncf.quantize_with_accuracy_control does not yet support the PyTorch model directly, instead you can export a PyTorch model to OpenVINO or ONNX and run nncf.quantize_with_accuracy_control on the exported model.

OpenVINO example: https://github.com/openvinotoolkit/nncf/tree/develop/examples/post_training_quantization/openvino/yolov8_quantize_with_accuracy_control
ONNX example: https://github.com/openvinotoolkit/nncf/tree/develop/examples/post_training_quantization/onnx/yolov8_quantize_with_accuracy_control

nncf.quantize_with_accuracy_control controls of accuracy metric by keeping the most impactful operations within the model in the original precision (OpenVINO documentation). This means that some operation will be executed at the original precision and performance of the model will be lower than that of a fully quantized model. I would recommend to use quantization aware training (QAT) for PyTorch model. It allows you to get the best accuracy of quantized model via model fine-tuning without performance degradation. But this method requires a training pipeline: QAT example for PyTorch model

[camhpj]

Thanks!