임베디드 시스템에서의 객체 분류를 위한 인공 신경망 경량화 연구
Neural Network Model Compression Algorithms for Image Classification in Embedded Systems
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This paper introduces model compression algorithms which make a deep neural network smaller and faster for embedded systems. The model compression algorithms can be largely categorized into pruning, quantization and knowledge distillation. In this study, gradual pruning, quantization aware training, and knowledge distillation which learns the activation boundary in the hidden layer of the teacher neural network are integrated. As a large deep neural network is compressed and accelerated by these algorithms, embedded computing boards can run the deep neural network much faster with less memory usage while preserving the reasonable accuracy. To evaluate the performance of the compressed neural networks, we evaluate the size, latency and accuracy of the deep neural network, DenseNet201, for image classification with CIFAR-10 dataset on the NVIDIA Jetson Xavier.
Abstract
This paper introduces model compression algorithms which make a deep neural network smaller and faster for embedded systems. The model compression algorithms can be largely categorized into pruning, quantization and knowledge distillation. In this study, gradual pruning, quantization aware training, and knowledge distillation which learns the activation boundary in the hidden layer of the teacher neural network are integrated. As a large deep neural network is compressed and accelerated by these algorithms, embedded computing boards can run the deep neural network much faster with less memory usage while preserving the reasonable accuracy. To evaluate the performance of the compressed neural networks, we evaluate the size, latency and accuracy of the deep neural network, DenseNet201, for image classification with CIFAR-10 dataset on the NVIDIA Jetson Xavier.
참고문헌 (22)
- A. Krizhevsky, "ImageNet Classification with Deep Convolutional Neural Networks", Communications of the ACM, 60(6), pp.84~90 (2017)[DOI]
- J. Deng, "ImageNet : A Large-scale Hierarchical Image Database", 2009IEEE Conference on Computer Vision and Pattern Recognition (2009)
- K. He, "Deep Residual Learning for Image Recognition", 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
- A. Dosovitskiy, "An Image Is Worth 16x16Words: Transformers for Image Recognition at Scale", International Conference on Learning Representations (2021)
- 오선택, "드론의 변전소 애자 이상현상 탐지를 위한 심층신경망 경량화 모델 기반 알고리즘 개발 연구", 제어.로봇.시스템학회 논문지, 26(11), pp.884~890 (2020), 제어·로봇·시스템학회[DOI]
- "Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications" (2017)[URL]
- T. Uetsuki, "CNN-based End-to-end Autonomous Driving on FPGA Using TVM and VTA", 2021IEEE 14th International Symposium on Embedded Multicore, 2021, ~
- "Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding" (2015)[URL]
- Y. LeCun, "Optimal Brain Damage", Advances in Neural Information Processing Systems (1989)
- B. Hassibi, "Second Order Derivatives for Network Pruning: Optimal Brain Surgeon", Advances in Neural Information Processing Systems (1992)
- "Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation" (2020)[URL]
- G. E. Hinton, "Distilling the Knowledge in a Neural Network", Advances in Neural Information Processing Systems (2014)
- M. Zhu, "To prune, or not to Prune: Exploring the Efficacy of Pruning for Model Compression", International Conference on Learning Representations Workshop (2018)
- B. Heo, "Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons", AAAI Conference on Artificial Intelligence (2019)
- Y. Lecun, "Gradient-based Learning Applied to Document Recognition", IEEE, 86(11), pp.2278~2324 (1998)[DOI]
- K. Simonyan, "Very Deep Convolutional Networks for Large Scale Image Recognition", International Conference on Learning Representations (2015)
- G. Huang, "Densely Connected Convolutional Networks", 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
- "Learning Both Weights and Connections for Efficient Neural Networks" (2015)[URL]
- A. See, "Compression of Neural Machine Translation via Pruning", The 20th Signll Conference on Computational Natural Language Learning (2016)
- "Exploring Sparsity in Recurrent Neural Networks" (2017)[URL]
- "Fitnets: Hints for Thin Deep Nets" (2015)[URL]
- S. Lim, "Fast Autoaugment", Advances in Neural Information Processing Systems, 32 (2019)