초해상화 모델 경량화를 위한 지식 증류 방법의 비교 연구
A Comparative Study of Knowledge Distillation Methods in Lightening a Super-Resolution Model
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
지식 증류는 깊은 모델의 지식을 가벼운 모델로 전달하는 모델 경량화 기술이다. 대부분의 지식 증류 방법들은 분류 모델을 위해 개발되었으며, 초해상화를 위한 지식 증류 연구는 거의 없었다. 본 논문에서는 다양한 지식 증류 방법들을 초해상화 모델에 적용하고 성능을 비교한다. 구체적으로, 초해상화 모델에 각 지식 증류 방법을 적용하기 위해 손실 함수를 수정하고, 각 지식 증류 방법을 사용하여 교사 모델을 약 27배 경량화한 학생 모델을 학습하여 2배 초해상화하는 실험을 진행하였다. 실험을 통해, 일부 지식 증류 방법은 초해상화 모델에 적용할 경우 유효하지 않음을 알 수 있었으며, 관계 기반 지식 증류 방법과 전통적인 지식 증류 방법을 결합했을 때 성능이 가장 높은 것을 확인하였다.
Abstract
Knowledge distillation (KD) is a model lightening technology that transfers the knowledge of deep models to light models. Most KD methods have been developed for classification models, and there have been few KD studies in the field of super-resolution (SR). In this paper, various KD methods are applied to an SR model and their performance is compared. Specifically, we modified the loss function to apply each KD method to the SR model and conducted an experiment to learn a student model that was about 27 times lighter than the teacher model and to double the image resolution. Through the experiment, it was confirmed that some KD methods were not valid when applied to SR models, and that the performance was the highest when the relational KD and the traditional KD methods were combined.
참고문헌 (16)
- G. Hinton, "Distilling the knowledge in a neur al network", Proc. of NIPS (2014)
- C. Ledig, "Photo-realistic single image superresolution using a generative adversarial network", Proc. of CVPR, pp.105~114 (2017)
- B. Lim, "Enhanced deep residual networks for single image super-resolution", Proc. of CVPRW, pp.136~144 (2017)
- A. Romero, "FitNets: hints for thin deep nets", Proc. of ICLR (2015)
- N. Komodakis, "Paying more attenti on to attention: improving the performance of convolu tional neural networks via attention transfer", Proc. of ICLR (2017)
- Y. Zhang, "Deep mutual learning", Proc. of CVPR, pp.4320~4328 (2018)
- L. Zhang, "Be your own teacher: improve the performance o f convolutional neural networks via self distillation", Proc. of ICCV, pp.3713~3722 (2019)
- Z. He, "Fakd: feature-affinity based knowledge distillation for efficient image super-resolution", Proc. of ICIP, pp.518~522 (2020)
- H. Moon, "Compression of super-resolution mo del using contrastive learning", Proc. of KIBME Sum mer Conference, pp.557~559 (2022)
- W. Park, "Relational knowledge distillation", Proc. of CVPR, pp.3967~3976 (2019)
- P. Chen, "Distilling knowledge via knowledge review", Proc. of CVPR (2021)
- E. Agustsson, "Ntire 2017 challenge on single image super-resolution: dataset and study", Proc. of CVPRW, pp.126~135 (2017)
- M. Bevilacqua, "Low complexity single image s uper-resolution based on nonnegative neighbor embed ding", Proc. of BMVC (2012)
- R. Zeyde, "On single image scale-up using spa rse-representations", Proc. of Int. Conf. on Curves a nd Surfaces, pp.711~730 (2010)
- D. Martin, "A database of human segmented n atural images and its application to evaluating segmen tation algorithms and measuring ecological statistics", Proc. of ICCV, 2, pp.416~423 (2001)
- J.-B. Huang, "Single image super-resolution fr om transformed self-exemplars", Proc. of CVPR, pp.5197~5206 (2015)