블록 계층별 재학습을 이용한 다중 힌트정보 기반 지식전이 학습
Multiple Hint Information-based Knowledge Transfer with Block-wise Retrai
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
In this paper, we propose a stage-wise knowledge transfer method that usesblock-wise retraining to transfer the useful knowledge of a pre-trained residual network(ResNet) in a teacher-student framework (TSF). First, multiple hint information transfer andblock-wise supervised retraining of the information was alternatively performed between teacherand student ResNet models. Next, Softened output information-based knowledge transfer wasadditionally considered in the TSF. The results experimentally showed that the proposed methodusing multiple hint-based bottom-up knowledge transfer coupled with incremental block-wiseretraining provided the improved student ResNet with higher accuracy than existing KD andhint-based knowledge transfer methods considered in this study.
Abstract
In this paper, we propose a stage-wise knowledge transfer method that usesblock-wise retraining to transfer the useful knowledge of a pre-trained residual network(ResNet) in a teacher-student framework (TSF). First, multiple hint information transfer andblock-wise supervised retraining of the information was alternatively performed between teacherand student ResNet models. Next, Softened output information-based knowledge transfer wasadditionally considered in the TSF. The results experimentally showed that the proposed methodusing multiple hint-based bottom-up knowledge transfer coupled with incremental block-wiseretraining provided the improved student ResNet with higher accuracy than existing KD andhint-based knowledge transfer methods considered in this study.
참고문헌 (13)
- Y. LeCun, "Gradient-based Learning Applied to Document Recognition", Proceedings of IEEE, 86(11), pp.1~46 (1998)
- S. L. Phung, "A Pyramidal Neural Network for Visual Pattern Recognition", Journal of IEEE Transactions on Neural Networks, 18(2), pp.329~343 (2007)
- A. Krizhevsky, "ImageNet Classification with Deep Convolutional Neural Networks", Proceedings of Advanceds in Neural Information Proceesing Systems, pp.1106~1114 (2012)
- C. Szegedy, "Going Deeper with Convolutions", Proceedings of IEEE Conference on Computer Vision Pattern and Recognition, pp.1~9 (2015)
- K. Simonyan, "Very Deep Convolutional Networks for Large-scale Image Recognition", Proceedings of International Conference on Learning Represent, pp.1~14 (2015)
- K. He, "Deep Residual Learning for Image Recognition", Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp.1~12 (2016)
- G. Huang, "Densely Connected Convolutional Networks", Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp.2261~2269 (2017)
- 전동호, "Fully Printed Chipless RFID Tags Using Dipole Array Structures with Enhanced Reading Ranges", Journal of Electromagnetic Engineering and Science, 17(3), pp.159~164 (2017), 한국전자파학회
- 윤영석, "Exploring the dynamic knowledge structure of studies on the Internet of things: Keyword analysis", ETRI Journal, 40(6), pp.745~758 (2018), 한국전자통신연구원[DOI]
- G. Hinton, "Distilling the Knowledge in a Neural Network", Proceeding of Neural Information Processing Systems Workshop, pp.1~19
- A. Romero, "Fitnets : Hints for Thin Deep Nets", Proceedings of 5th International Conference on Learning Represent, pp.1~13 (2015)
- "CIFAR-10 and CIFAR-100 Datasets"[URL]
- Y. Netzer, "Reading Digits in Natural Images with Unsupervised Feature Learning", Proceedings of Neural Information Processing Systems Workshop on Deep Learning and Unsypervised Feature, pp.1~9 (2011)