모델 내부/외부 특징량 상관 학습을 통한 지식 증류
Knowledge Distillation based-on Internal/External Correlation Learning
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 논문에서는 이종 모델의 특징맵 간 상관관계인 외부적 상관관계와 동종 모델 내부 특징맵간 상관관계인 내부적 상관관계를 활용하여 교사 모델로부터 학생 모델로 지식을 전이하는Internal/External Knowledge Distillation (IEKD)를 제안한다. 두 상관관계를 모두 활용하기 위하여 특징맵을 시퀀스 형태로 변환하고, 트랜스포머를 통해 내부적/외부적 상관관계를 고려하여 지식 증류에 적합한 새로운 특징맵을 추출한다. 추출된 특징맵을 증류함으로써 내부적 상관관계와 외부적 상관관계를 함께 학습할 수 있다. 또한 추출된 특징맵을 활용하여 feature matching을 수행함으로써 학생 모델의 정확도 향상을 도모한다. 제안한 지식 증류 방법의 효과를 증명하기 위해, CIFAR-100 데이터 셋에서 “ResNet-32×4/VGG-8” 교사/학생 모델 조합으로 최신 지식 증류 방법보다 향상된 76.23% Top-1 이미지 분류 정확도를 달성하였다.
Abstract
In this paper, we propose an Internal/External Knowledge Distillation (IEKD), which utilizes both external correlations between feature maps of heterogeneous models and internal correlations between feature maps of the same model for transferring knowledge from a teacher model to a student model. To achieve this, we transform feature maps into a sequence format and extract new feature maps suitable for knowledge distillation by considering internal and external correlations through a transformer. We can learn both internal and external correlations by distilling the extracted feature maps and improve the accuracy of the student model by utilizing the extracted feature maps with feature matching. To demonstrate the effectiveness of our proposed knowledge distillation method, we achieved 76.23% Top-1 image classification accuracy on the CIFAR-100 dataset with the “ResNet-32×4/VGG-8” teacher and student combination and outperformed the state-of-the-art KD methods.
참고문헌 (28)
- He, Kaiming, "Deep residual learning for image recognition", Proceedings of the IEEE conference on computer vision and pattern recognition, pp.770~778 (2016)
- Shaoqing Ren, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), pp.1137~1149 (2017), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- Han, Song, "Deep compression:Compressing deep neural networks with pruning, trained quantization and huffman coding" (2016), International Conference on Learning Representations
- Mohammad Rastegari, "Xnor-net: Imagenet classification using binary convolutional neural networks", Computer Vision–ECCV 2016:14th European Conference, pp.525~542 (2016)
- A. Romero, "Fitnets: Hints for thin deep nets", International Conference on Learning Representations (2015)
- Chen, Defang, "Cross-layer distillation with semantic calibration", Proceedings of the AAAI Conference on Artificial Intelligence, 35(8), pp.7028~7036 (2021)
- Hao-Ting Li, "Layer-Level Knowledge Distillation for Deep Neural Network Learning", Applied Sciences, 9(10), pp.1966~ (2019), MDPI AG[DOI]
- Han, Kai, "Ghostnet: More features from cheap operations", Proceedings of the IEEE, 2020, 1577~1586
- Liu, Li, "Exploring inter-channel correlation for diversity-preserved knowledge distillation", Proceedings of the IEEE, 2021, 8251~8260
- Vaswani, Ashish, "Attention is all you need", Advances in neural information processing systems 30, 6000~6010 (2017)
- Learning multiple layers of features from tiny images, "Alex Krizhevsky", 1~60 (2009)
- Yang, Zhendong, "Focal and global knowledge distillation for detectors", Proceedings of the IEEE, 2022, 4633~4642
- Zhao, Borui, "Decoupled knowledge distillation", Proceedings of the IEEE, 2022, 11943~11952
- Chen, Defang, "Knowledge distillation with the reused teacher classifier", Proceedings of the IEEE, 2022, 11923~11932
- Beyer, Lucas, "Knowledge distillation: A good teacher is patient and consistent", Proceedings of the IEEE, 2022, 10915~10924
- Park, Wonpyo, "Relational knowledge distillation", Proceedings of the IEEE, 2019, 3962~3971
- S. Zagoruyko, "Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer", International Conference on Learning Representations (2017)
- F. Tung, "Similarity-preserving knowledge distillation", International Conference on Computer Vision, pp.1365~1374 (2019)
- S. Ahn, "Variational information distillation for knowledge transfer", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.9163~9171 (2019)
- N. Passalis, "Heterogeneous knowledge distillation using information flow modeling", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.2336~2345 (2020)
- Lin, Sihao, "Knowledge distillation via the target-aware transformer", Proceedings of the IEEE, 2022, 10905~10914
- Paszke, Adam, "Pytorch: An imperative style, high-performance deep learning library", Advances in neural information processing systems 32, 8026~8037 (2019)
- Karen Simonyan, "Very deep convolutional networks for large-scale image recognition", International Conference on Learning Representations (2015)
- Sandler, Mark, "Mobilenetv2: Inverted residuals and linear bottlenecks", Proceedings of the IEEE conference on computer vision and pattern recognition, pp.4510~4520 (2018)
- Zagoruyko, Sergey, "Wide residual networks", Proceedings of the British Machine Vision Conference, pp.87~81 (2016)
- Ma, Ningning, "Shufflenet v2: Practical guidelines for efficient cnn architecture design", Proceedings of the European conference on computer vision (ECCV), pp.116~131 (2018)