심층 신경망을 위한 클러스터링 기반 모델 압축 기법
Clustering-based Model Compression Method for Deep Neural Networks
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
온-디바이스 머신러닝은 비용 효율성, 데이터 프라이버시, 응답성 측면에서의 강점으로 인해 점차 인기를 얻고 있다. 그러나 소형 임베디드시스템에서는 메모리 용량이 제한적이기 때문에 심층 신경망 모델을 처리하기 쉽지 않다. 이전 연구에서는 양자화나 가지치기와 같은 다양한 모델압축 기법들을 제안하였다. 그러나 이러한 기법들은 일반적으로 압축으로 인한 정확도 손실을 최소화하기 위해 적절한 데이터 샘플을 사용한 세심한미세 조정을 필요로 한다. 본 연구에서는 유사한 컨볼루션 커널을 클러스터링하고 가지치기하여 입력 모델을 압축하는 새로운 훈련 후 모델 압축방법을 제안한다. 본 연구에서 제안된 방법은 커널 간의 유사성만을 고려하므로 데이터 샘플을 필요로하지 않는다. 본 연구는 대표적인 신경망모델을 사용하여 제안된 방법을 평가하고 적은 정확도 손실로도 메모리 사용량을 효과적으로 줄일 수 있다는 것을 입증한다.
Abstract
On-device machine learning is becoming more popular for its strengths in cost efficiency, data privacy, and responsiveness. However,processing deep neural network models on small embedded systems is challenging due to their limited memory capacity. Previous workhas proposed various model compression techniques, such as quantization and pruning. However, the techniques generally require carefulfine-tuning with proper data samples to minimize accuracy loss from compression. This work proposes a new post-training modelcompression method that compresses the input model by clustering and pruning similar convolution kernels. The proposed method doesnot require data samples because it considers the similarity between kernels only. This work evaluates the proposed method withrepresentative neural network models and demonstrates that the method can effectively reduce memory usage on average with smallaccuracy loss.
참고문헌 (15)
- O. D. Incel, "On-device deep learning for mobile and wearable sensing applications : A review", IEEE Sensors Journal, 23(6), pp.5501~5512 (2023)
- A. Albanese, "Tiny machine learning for high accuracy product quality inspection", IEEE Sensors Journal, 23(2), pp.1575~1583 (2023)
- I. Gim, "Memory-efficient dnn training on mobile devices", MobiSys ’22, pp.464~476 (2022), Association for Computing Machinery
- Y. Choi, "Towards the limit of network quantization", Proceedings of the 5th International Conference on Learning Representations, ICLR ’17 (2017)
- S. Gupta, "Deep learning with limited numerical precision", Proceedings of the 32nd International Conference on International Conference on Machine Learn-ing(ICML), ICML’15 (2015)
- S. Han, "Learning both weights and connections for efficient neural networks", Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15 (2015), MIT Press
- H. Li, "Pruning filters for efficient convnets", Proceedings of the 5th International Conference on Learning Representations, ICLR’17 (2017)
- L. Liu, "Group fisher pruning for practical network compression", International Conference on Machine Learning, pp.7021~7032 (2021), PMLR
- M. Cho, "Differentiable k-means clustering layer for neural network compression", ICLR (2022)
- S. Bai, "Unified datafree compression : Pruning and quantization without finetuning", 2023 IEEE, 2023, 5853~5862
- T. Liang, "Pruning and quantization for deep neural network acceleration : A survey", Neurocomputing, 461, pp.370~403 (2021)
- G. Jo, "OpenCL framework for arm processors with neon support", Proceedings of the 2014 Workshop on Programming Models for SIMD, 2014, 33~40[DOI]
- "TensorFlow Lite - ML for Mobile and Edge Devices" (2024)[URL]
- K. Simonyan, "Very deep convolutional networks for large-scale image recognition", 3rd International Conference on Learning Representations(ICLR) (2015)
- S. Han, "Deep compression : Compressing deep neural network with pruning, trained quantization and huffman coding", 4th International Conference on Learning Representations(ICLR) (2016)