초소형 IoT 장치에 구현 가능한 딥러닝 양자화 기술 분석
Analysis of Deep learning Quantization Technology for Micro-sized IoT devices
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
많은 연산량을 가진 딥러닝은 초소형 IoT 장치나 모바일 장치에 구현하기가 어렵다. 최근에는 이러한 장치에서도 딥러닝을 구현할 수 있도록 모델의 연산량을 줄이는 딥러닝 경량화 기술이 소개되었다. 양자화는 연속적인 분포를가지는 파라미터 값들을 고정된 비트의 이산 값으로 표현하여 모델의 메모리 및 크기 등을 줄여 효율적으로 사용할수 있는 경량화 기법이다. 그러나 양자화로 인한 이산 값 표현으로 인해 모델의 정확도가 낮아지게 된다. 본 논문에서는정확도를 개선할 수 있는 다양한 양자화 기술을 소개한다. 먼저 기존 양자화 기술 중 APoT와 EWGS를 선택하여 동일한 환경에서 실험을 통해 결과를 비교 분석하였다. 선택된 기술은 ResNet모델에서 CIFAR-10 또는 CIFAR-100 데이터 세트로 훈련되고 테스트 되었다. 실험 결과 분석을 통해 기존 양자화 기술의 문제점을 파악하고 향후 연구에 대한방향성을 제시하였다.
Abstract
Deep learning with large amount of computations is difficult to implement on micro-sized IoT devices or moblie devices. Recently, lightweight deep learning technologies have been introduced to make sure that deep learning can be implemented even on small devices by reducing the amount of computation of the model. Quantization is one of lightweight techniques that can be efficiently used to reduce the memory and size of the model by expressing parameter values with continuous distribution as discrete values of fixed bits. However, the accuracy of the model is reduced due to discrete value representation in quantization. In this paper, we introduce various quantization techniques to correct the accuracy. We selected APoT and EWGS from existing quantization techniques, and comparatively analyzed the results through experimentations The selected techniques were trained and tested with CIFAR-10 or CIFAR-100 datasets in the ResNet model. We found out problems with them through experimental results analysis and presented directions for future research.
참고문헌 (25)
- Howard, Andrew G., "Mobilenets : Efficient convolutional neural networks for mobile vision applications", arXiv preprint arXiv : 1704. 04861, ~ (2017)
- Blalock, Davis, "What is the state of neural network pruning?", Proceedings of machine learning and systems, 2, pp.129~146 (2020)
- Hinton, Geoffrey, "Distilling the knowledge in a neural network", arXiv preprint arXiv : 1503. 02531, ~ (2015)
- Itay Hubara, "Binarized neural networks", Advances in neural information processing systems, 29 (2016)
- Raghuraman Krishnamoorthi, "Quantizing deep convolutional networks for efficient inference : A whitepaper", arXiv preprint arXiv : 1806. 08342, ~ (2018)
- Benoit Jacob, "Quantization and training of neural networks for efficient integer-arithmetic-only inference", Proceedings of the IEEE conference on computer vision and pattern recognition, pp.2704~2713 (2018)
- Hao Wu, "Integer quantization for deep learning inference : Principles and empirical evaluation", arXiv preprint arXiv : 2004. 09602, ~ (2020)
- Song Han, "Deep compression : Compressing deep neural networks with pruning, trained quantization and Huffman coding", arXiv preprint arXiv : 1510. 00149, ~ (2015)
- Zhaohui Yang, "Searching for low-bit weights in quantized neural networks", Advances in neural information processing systems, 33, pp.4091~4102 (2020)
- Kohei Yamamoto, "Learnable companding quantization for accurate low-bit neural networks", Proceedings of the IEEE, 2021, 5029~5038
- Yunchao Gong, "Compressing deep convolutional networks using vector quantization", arXiv preprint arXiv : 1412. 6115, ~ (2014)
- Yang, Jiwei, "Quantization networks", Proceedings of the IEEE, 2019, 7308~7316
- Gong, Ruihao, "Differentiable soft quantization : Bridging full-precision and low-bit neural networks", Proceedings of the IEEE, 2019, 4852~4861
- Kim, Dohyung, "Distance-aware quantization", Proceedings of the IEEE, 2021, 5271~5280
- Aojun Zhou, "Incremental network quantization : Towards lossless cnns with low-precision weights", arXiv preprint arXiv : 1702. 03044, ~ (2017)
- Yuhang Li, "Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks", International Conference on Learning Representations (2020)
- Lee, Junghyup, "Network quantization with element-wise gradient scaling", Proceedings of the IEEE, 2021, 6448~6457
- Yoshua Bengio, "Estimating or propagating gradients through stochastic neurons for conditional computation", arXiv preprint arXiv : 1308. 3432, ~ (2013)
- Avron, Haim, "Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix", Journal of the ACM(JACM), 58(2), pp.1~34 (2011)
- Itay Hubara, "Improving post training neural quantization : Layer-wise calibration and integer programming", arXiv preprint arXiv : 2006. 10518, ~ (2020)
- Markus Nagel, "Data-free quantization through weight equalization and bias correction", Proceedings of the IEEE, 2019, 1325~1334
- Li, Yuhang, "Brecq : Pushing the limit of post-training quantization by block reconstruction", arXiv preprint arXiv : 2102. 05426, ~ (2021)
- Kaiming He, "Deep residual learning for image recognition", Proceedings of the IEEE conference on computer vision and pattern recognition, pp.770~778 (2016)
- Alex Krizhevsky, "Learning multiple layers of features from tiny images", ~ (2009)
- Lee, Junghyup, "Sfnet : Learning object-aware semantic correspondence", Proceedings of the IEEE, 2019, 2278~2287