DNN을 위한 비트 단위 파라미터 조작 프레임워크 및 파라미터와 정확도 간의 상호 연관성 분석
Bit-level Parameter Manipulation Framework for DNNs and Analysis of Mutual Relationship Between Parameters and Accuracy
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
최근 DNN이 다양한 산업에 확산되면서 IoT 기기 및 엣지 컴퓨팅에 적합한 경량 모델에 관한 연구가 급증하고 있다. 본 논문에서는 기존에 없던 딥러닝 모델의 파라미터를 1 비트 단위로 조작할 수 있는 자동화 프레임워크를 개발하며 파라미터 비트와 모델 정확도 사이의 관계를 실험 및 연구한다. 본 연구는 제안된 프레임워크를 사용하여 ImageNet 데이터셋으로 사전 학습된 DNN 모델 중 CNN 모델들의 파라미터를 하위 n-bit를 0, 1 또는 랜덤한 값으로 치환하는 3가지 방법을 통해 각각 정보 손실 발생시키면서 파라미터와 정확도 간의 강인성을 비트 단위로 실험하였다. 주요 모델로는 InceptionV3, InceptionResnetV2, ResNet50, Xception, DenseNet121, MobileNetV1, MobileNetV2 을 사용하였다. 실험 결과, 성능이 낮은 모델일수록 하위 비트의 정보 손실에 민감하여 성능이 좋은 모델보다 정확도를 유지하는 비트 수가 적다는 것을 실험적으로 확인했고, 파라미터와 정확도 간의 강인성이 높다는 것을 확인하였다. 이러한 실험을 바탕으로 모델별 유효 파라미터 비트를 설정하여 파라미터를 줄이며 정확도를 유지할 수 있다.
Abstract
Recently, with the proliferation of DNNs in various industries, there has been a surge in research on lightweight models suitable for IoT devices and edge computing. In this paper, we propose an automated framework that enables manipulation of deep learning model parameters at a 1-bit level, a capability not previously available. We investigate the relationship between parameter bits and model accuracy. Using the developed framework, we systematically experimented with the parameters of CNN models pre-trained on the ImageNet dataset by setting the lower n-bit to 0, 1, or a random value while each method inducing distinct information loss. The primary models evaluated include InceptionV3, InceptionResnetV2, ResNet50, Xception, DenseNet121, MobileNetV1, and MobileNetV2. Experimental results show that models with lower performance are more sensitive to information loss in the lower bits, requiring fewer bits to maintain accuracy compared to high-performing models. This concludes a high robustness between parameters and accuracy.
참고문헌 (19)
- Deep learning model lightweight technology analysis, "K. Lee", Korea Institute of Science and Technology Information, ~ (2020)
- S. Yoo, "Trends in Deep Learning Inference Engines for Embedded Systems", Electronics and Telecommunications Trends, 34(4), pp.23~31 (2019), ETRI
- J. W. Choi, "A Study on the Performance Evaluation of Edge Device according to the Compression Method of Deep Learning Model", Journal of the Institute of Electronics and Information Engineers, 58(6), pp.50~60 (2021)
- D. Chang, "Lightweight of ONNX using Quantization-based Model Compression", The Journal of The Institute of Internet, Broadcasting and Communication, 21(1), pp.93~98 (2021)
- M. Courbariaux, "BinaryConnect : Training Deep Neural Networks with binary weights during propagations", Advances in Neural Information Processing Systems, pp.3123~3131 (2015)
- S. Han, "Deep Compression : Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding", Proceedings of the International Conference on Learning Representations (2016)
- D. Wang, "Compressing 3DCNNs based on tensor train decomposition", Neural Networks, 131, pp.215~230 (2020)
- O. Russakovsky, "ImageNet Large Scale Visual Recognition Challenge", International journal of computer vision, 115, pp.211~252 (2015)
- A. Krizhevsky, "ImageNet Classification with Deep Convolutional Neural Networks", Advances in Neural Information Processing Systems, pp.1097~1105 (2012)
- K. He, "Deep Residual Learning for Image Recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.770~778 (2016)
- G. Huang, "Densely Connected Convolutional Networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.4700~4708 (2017)
- C. Szegedy, "Going Deeper with Convolutions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1~9 (2015)
- C. Szegedy, "Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning", Thirty-First AAAI Conference on Artificial Intelligence, pp.4278~4284 (2016)
- F. Chollet, "Xception : Deep Learning with Depthwise Separable Convolutions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1251~1258 (2017)
- A. G. Howard, "MobileNet : Efficient Convolutional Neural Networks for Mobile Vision Applications", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1382~1391 (2017)
- B. Zoph, "NasNet : Learning Transferable Architectures for Scalable Image Recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.2117~2126 (2018)
- A. Dosovitskiy, "An Image Is Worth 16x16 Words : Transformers for Image Recognition at Scale", Proceedings of the International Conference on Learning Representations (2021)
- A. Vaswani, "Attention Is All You Need", Advances in Neural Information Processing Systems, pp.30~38 (2017)