데이터 없이 사전학습된 태스크 모델 기반 생성형 모델의 지속학습
Data-Free Continual Learning of Generative Model via Task-specific Pretrained Models
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
딥러닝의 치명적 망각 현상(Catastrophic forgetting)을 방지하기 위한 지속 학습 연구는 최근 분류모델에서 생성형 모델로 확장되고 있다. 하지만 생성형 모델의 지속 학습은 태스크별 입력 데이터에 의존하며, 이는 현실 세계에서 프라이버시 문제와 한정된 메모리 자원에 따른 제약이 존재한다. 즉, 신경망의 블랙박스 특성과 상대적으로 작은 메모리 크기를 고려하여, 접근 불가능한 데이터를 대체할 필요가 있다. 이에 본 논문에서는 태스크별 데이터에 직접 접근하는 대신, 해당 데이터를 사전 학습한 추론형 신경망만을 입력으로 받아 생성형 모델을 지속 학습하는 모델(Data-Free Continual Learning of Generative Network, DF-CLG)을 처음으로 제안한다. 구체적으로, 사전 학습된 추론형 신경망에 저장된 배치 정규화 계층의 통곗값을 제약하여 학습에 사용된 데이터를 역으로 합성하는 생성형 신경망을 학습한다. 이후, 대표적인 지속 학습 기법인 규제 기반 방법과 리허설 기반 방법을 적용하여, 제안된 생성형 신경망의 치명적 망각 현상이 완화될 수 있음을 보인다. 주목할 점은, 지속 학습에서 상대적으로 저조한 성능으로 알려진 규제 기반 방법이 제안된 생성형 모델에서 리허설 기반 방법에 준하거나 우수한 성능을 보인다는 것이다.
Abstract
Research on continual learning to prevent catastrophic forgetting problem of deep learning has recently expanded from classification models to generative models. However, the continual learning of generative models relies on task-specific input data, which poses constraints on pivacy issues and limited memory resources in the real world. In other words, considering the black-box property of neural networks and their relatively small memory size, there is a need to replace inaccessible data. In this paper, we propose the first model, called Data-Free Continual Learning of Generative Network (DF-CLG), which receives pre-trained inference model as input instead of directly accessing task-specific data. Specifically, the generative model is trained to synthesize the data used in the training by constraining the statistics of batch normalization layers stored in the pre-trained inference network. Subsequently, by applying representative continual learning methods, namely regularization-based and rehearsal-based approaches, it is demonstrated that the proposed generative model can alleviate catastrophic forgetting. It is worth noting that regularization-based methods, which are known for their relatively poor performance in continual learning, show comparable or superior performance to rehearsal-based methods in the proposed generative model.
참고문헌 (18)
- J. Kirkpatrick, "Overcoming catastrophic forgetting in neural networks", Proceedings of the national academy of sciences, 114(13), pp.3521~3526 (2017)
- C. Wu, "Memory replay gans: learning to generate images from new categories without forgetting", Advances in Neural Information Processing Systems, 31 (2018)
- Yoo, "Knowledge extraction with no observable data", Advances in Neural Information Processing Systems, 32 (2019)
- "Inceptionism: Going deeper into neural networks"[URL]
- Haroush, "The knowledge within: Methods for data-free model compression", Proceedings of the IEEE, 2020, ~
- H. Yin, "Dreaming to distill: Data-free knowledge transfer via deepinversion", Proceedings of the IEEE, 2020, ~
- L. Luo, "Large-scale generative data-free distillation" (2020)
- H. Shin, "Continual learning with deep generative replay", Advances in neural information processing systems, 30 (2017)
- T. Lesort, "Generative models from the perspective of continual learning", International Joint Conference on Neural Networks (IJCNN) (2019), IEEE
- A. Radford, "Unsupervised representation learning with deep convolutional generative adversarialnetworks", International Conference on Learning Representations, ICLR 2016
- H. Chen, "Data-free learning of student networks", IEEE, ~
- G. K. Nayak, "Zeroshot knowledge distillation in deep networks", Proceedings of the 36th International Conference on Machine Learning, ICML 2019
- I. J. Goodfellow, "Generative adversarial nets", Advances in neural information processing systems, 27 (2014)
- LeCun, Y., "Gradient-based learning applied to document recognition", Proceedings of the IEEE, 86(11), pp.2278~2324 (1998)
- He, K., "Deep residual learning for image recognition", Proceedings of the IEEE conference on computer vision and pattern recognition (2016)
- HINTON, Geoffrey, "Distilling the knowledge in a neural network" (2015)
- ZHAI, Mengyao, "Lifelong gan: Continual learning for conditional image generation", Proceedings of the IEEE, 2019, ~
- ZHU, Jun-Yan, "Toward multimodal imageto-image translation", Advances in neural information processing systems (2017)