어텐션 기반의 지속학습에서 정규화값 제어 방법
Regularization Strength Control for Continuous Learning based on Attention Transfer
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 논문에서는 LwF에서 도메인 차이에 따른 성능 하락 현상을 해결하기 위해, 각 손실값에 각각 다른 가변람다를 적용하는 알고리즘을 제안하여, 향상된 과거 지식유지가 이루어 지게 한다. 이 지식 전달 기반의 방법을 LwF와 접목하여, 과거 학습 태스크의 지식 유지 성능을 강화하였다. 가변 람다 방법을 추가적으로 적용하여, 현재 학습할 태스크를 잘 학습할 수 있도록 람다 값을 가변적으로 조절할 수 있었다. 본 논문의 제안 방법을 적용하여 학습한 결과 시나리오에 상관없이 평균 5% 정도 데이터의 정확도가 향상하였고, 특히 본 논문의 목표인 과거 지식을 유지하는 성능이 최대 70% 가까이 개선되었고, 과거 학습 데이터의 정확도가 기존LwF 대비 평균 22% 상승하였다.
Abstract
In this paper, we propose an algorithm that applies a different variable lambda to each loss value to solve theperformance degradation caused by domain differences in LwF, and show that the retention of past knowledge isimproved. The lambda value could be variably adjusted so that the current task to be learned could be welllearned, by the variable lambda method of this paper. As a result of learning by this paper, the data accuracyimproved by an average of 5% regardless of the scenario. And in particular, the performance of maintaining pastknowledge, the goal of this paper, was improved by up to 70%, and the accuracy of past learning data increasedby an average of 22% compared to the existing LwF.
참고문헌 (13)
- R. M. French, "Catastrophic forgetting in connectionist networks", Trends in cognitive sciences, 3(4), pp.128~135 (1999)[DOI]
- German I. Parisi, "Continual lifelong learning with neural networks: A review", Neural Networks, 113, pp.54~71 (2019), Elsevier BV[DOI]
- F. Zenke, "Continual learning through synaptic intelligence", Proceedings of the 34th International Conference on Machine Learning, 70, pp.3987~3995 (2017)
- Y. Hsu, "Re-evaluating continual learning scenarios: A categorization and case for strong baselines", arXiv:1810.12488, ~ (2019)
- J. Yoon, "Lifelong learning with dynamically expandable networks,”", arXiv : 1708. 01547, ~ (2017)
- H. Shin, "Continual learning with deep generative replay", arXiv : 1705. 08690, ~ (2017)
- G. Hinton, "Distilling the knowledge in a neural network", NIPS Workshop (2014)
- K. McRae, "Catastrophic interference is eliminated in pretrained networks", Proceedings of the 15h Annual Conference of the Cognitive Science Society, pp.723~728 (1993)
- R. M. French, "Catastrophic forgetting in connectionist networks", Trends in Cognitive Sciences, 3(4), pp.128~135 (1999), Elsevier BV[DOI]
- J. Kirkpatrick, "Overcoming catastrophic forgetting in neural networks", Proceedings of the national academy of sciences, 114(13), pp.3521~3526 (2017)
- Z. Li, "Learning without forgetting", IEEE transactions on pattern analysis and machine intelligence, 40(12), pp.2935~2947 (2017)[DOI]
- S. Zagoruyko, "Paying more attention to attention : Improving the performance of convolutional neural networks via attention transfer", arXiv : 1612. 03928, ~ (2016)
- B. Heo, "Knowledge transfer via distillation of activation boundaries formed by hidden neurons", Proceedings of the AAAI Conference on Artificial Intelligence, 33(1), pp.3779~3787 (2019)[DOI]