생성 모델을 이용한 데이터 프리 양자화를 위한 Bit-width Aware Generator와 채널 어텐션 기반 중간 레이어 지식 증류
Bit-width Aware Generator and Intermediate Layer Knowledge Distillation using Channel-wise Attention for Generative Data-Free Quantization
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 논문에서는 생성 모델을 이용한 데이터 프리 양자화에서 발생할 수 있는 지식 격차를 줄이기위하여 BAG (Bit-width Aware Generator)와 채널 어텐션 기반 중간 레이어 지식 증류를 제안한다. 생성 모델을 이용한 데이터 프리 양자화의 생성자는 오직 원본 네트워크의 피드백에만 의존하여 학습하기 때문에, 양자화된 네트워크의 낮은 bit-width로 인한 감소된 수용 능력 차이를 학습에 반영하지못한다. 제안한 BAG는 양자화된 네트워크와 동일한 bit-width로 양자화하여, 양자화된 네트워크에 맞는 합성 이미지를 생성하여 이러한 문제를 완화한다. 또한, 양자화된 네트워크와 원본 모델 간의 지식 격차를 줄이는 것 역시 양자화에서 매우 중요한 문제이다. 이를 완화하기 위해 제안한 채널 어텐션 기반 중간 레이어 지식 증류는 학생 모델이 교사 모델로부터 어떤 채널에 더 집중해서 학습해야하는지를 가르친다. 제안한 기법의 효율성을 보이기 위해, CIFAR-100에서 학습한 원본 네트워크를가중치와 활성값을 각각 3-bit로 양자화하여 학습을 수행하였다. 그 결과 56.14%의 Top-1 Accuracy를달성하였으며, 베이스라인 모델인 AdaDFQ 대비 3.4% 정확도를 향상했다.
Abstract
In this paper, we propose the BAG (Bit-width Aware Generator) and the Intermediate Layer Knowledge Distillation using Channel-wise Attention to reduce the knowledge gap between a quantized network, a full-precision network, and a generator in GDFQ (Generative Data-Free Quantization). Since the generator in GDFQ is only trained by the feedback from the full-precision network, the gap resulting in decreased capability due to low bit-width of the quantized network has no effect on training the generator. To alleviate this problem, BAG is quantized with same bit-width of the quantized network, and it can generate synthetic images, which are effectively used for training the quantized network. Typically, the knowledge gap between the quantized network and the full-precision network is also important. To resolve this, we compute channel-wise attention of outputs of convolutional layers, and minimize the loss function as the distance of them. As the result, the quantized network can learn which channels to focus on more from mimicking the full-precision network. To prove the efficiency of proposed methods, we quantize the network trained on CIFAR-100 with 3 bit-width weights and activations, and train it and the generator with our method. As the result, we achieve 56.14% Top-1 Accuracy and increase 3.4% higher accuracy compared to our baseline AdaDFQ.
참고문헌 (37)
- Jacob, Benoit, "Quantization and training of neural networks for efficient integer-arithmetic-only inference", Proceedings of the IEEE, 2018, 2704~2713
- Gray, Robert M., "Quantization", IEEE transactions on information theory, 44(6), pp.7308~7316 (1998)[DOI]
- Yang, Jiwei, "Quantization networks", Proceedings of the IEEE, 2018, 2704~2713
- Nagel, Markus, "Data-free quantization through weight equalization and bias correction", Proceedings of the IEEE, 2019, 1325~1334
- Liu, Xingchao, "Post-training quantization with multiple points: Mixed precision without mixed precision", Proceedings of the AAAI conference on artificial intelligence, 35(10), pp.8697~8705 (2021)
- Li, Yuhang, "Brecq: Pushing the limit of post-training quantization by block reconstruction", Proceedings of 9th International Conference on Learning Representations (2021)
- Wang, Kuan, "Haq: Hardware-aware automated quantization with mixed precision", Proceedings of the IEEE, 2019, 8612~8620
- Esser, Steven K., "Learned step size quantization", Proceedings of 8th International Conference on Learning Representations (2020)
- Xu, Shoukai, "Generative low-bitwidth data free quantization", Computer Vision–ECCV 2020: 16th European Conference, 12357, pp.1~17 (2021)
- Liu, Yuang, "Zero-shot adversarial quantization", Proceedings of the IEEE, 2021, 1512~1521
- Qian, Biao, "Adaptive data-free quantization", Proceedings of the IEEE, 2023, 7960~7968
- Romero, Adriana, "FitNets: Hints for Thin Deep Nets", Proceedings of 8th International Conference on Learning Representations (2015)
- Chen, Defang, "Cross-layer distillation with semantic calibration", Proceedings of the AAAI conference on artificial intelligence, 35(8), pp.7028~7036 (2021)
- Goodfellow, Ian, "Generative adversarial nets", Advances in Neural Information Processing Systems, pp.2672~2680 (2014)
- Radford, Alec, "Unsupervised representation learning with deep convolutional generative adversarial networks", Proceedings of 4th International Conference on Learning Representations (2016)
- Odena, Augustus, "Conditional image synthesis with auxiliary classifier gans", International conference on machine learning, 70, pp.2642~2651 (2017)
- Li, Shaojie, "Revisiting discriminator in gan compression : A generator-discriminator cooperative compression scheme", Advances in Neural Information Processing Systems, pp.28560~28572 (2021)
- Metz, Luke, "Unrolled generative adversarial networks", Proceedings of 5th International Conference on Learning Representations (2017)
- He, Kaiming, "Deep residual learning for image recognition", Proceedings of the IEEE, 2016, 770~778
- Learning multiple layers of features from tiny images, "Alex Krizhevsky", 1~60 (2009)
- Cai, Yaohui, "Zeroq: A novel zero shot quantization framework", Proceedings of the IEEE, 2020, 13166~13175
- Li, Huantong, "Hard sample matters a lot in zero-shot quantization", Proceedings of the IEEE, 2023, 24417~24426
- Choi, Kanghyun, "Qimera: Data-free quantization with synthetic boundary supporting samples", Advances in Neural Information Processing Systems, pp.14835~14847 (2021)
- Zhang, Xiangguo, "Diversifying sample generation for accurate data-free quantization", Proceedings of the IEEE, 2021, 15658~15667
- Choi, Kanghyun, "It's all in the teacher: Zero-shot quantization brought closer to the teacher", Proceedings of the IEEE, 2022, 8301~8311
- S. Zagoruyko, "Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer", Proceedings of 5th International Conference on Learning Representations (2017)
- Ioffe, Sergey, "Batch normalization:Accelerating deep network training by reducing internal covariate shift", International conference on machine learning, 37, pp.448~456 (2015)
- Maas, Andrew L., "Rectifier nonlinearities improve neural network acoustic models", International conference on machine learning, 30, pp.3~8 (2013)
- Kingma, Diederik P., "Adam: A method for stochastic optimization", Proceeding of 3rd International Conference on Learning Representations (2015)
- Zhu, Baozhou, "Autorecon: Neural architecture search-based reconstruction for data-free compression", Proceedings of the AAAI conference on artificial intelligence, pp.3470~3476 (2021)
- Zhong, Yunshan, "Intraq: Learning synthetic images with intra-class heterogeneity for zero-shot network quantization", Proceedings of the IEEE, 2022, 12329~12338
- Qian, Biao, "Rethinking data-free quantization as a zero-sum game", Proceedings of the AAAI conference on artificial intelligence, 37(8), pp.9489~9497 (2023)
- Chen, Xinrui, "TexQ: Zero-shot Network Quantization with Texture Feature Distribution Calibration", Advances in Neural Information Processing Systems (2023)
- Li, Jixing, "ACQ: Improving generative data-free quantization via attention correction", Pattern Recognition, 152, pp.110444~ (2024)[DOI]
- I. P. Durugkar, "Generative multi-adversarial networks", Proceedings of 5th International Conference on Learning Representations (2017)