시계열 예측을 위한 3D 컨볼루션 트랜스포머
3D Convolutional Transformer for Time-Series Forecasting
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
다양한 분야에서 시계열 데이터의 중요성이 증가하면서 미래의 시계열 데이터를 정확하게 예측하는 것이 중요한 문제로 대두되고 있다. 시계열 데이터 예측 분야에서는 오랫동안 LSTM 기반 모델을 사용했지만, 최근에는 트랜스포머 모델이 주목받기 시작했다. 하지만 트랜스포머 모델 구조는 시계열의 순차적인 특성을 보존하는 능력이 약하다는 한계를 가진다. 본 논문에서는 이것을 해결하기 위해 3D 컨볼루션 네트워크와 Transformer를 결합한 구조의 3DCformer를 제안한다. 3DCformer의 인코더는 dilated 컨볼루션을 사용하여 입력 데이터의 순차적인 특성을 강화한 뒤 2D 입력 데이터를 3D로 확장한다. 그 후 3D 컨볼루션 네트워크를 통해 확장된 특성에서 표현을 추출한다. 3DCformer의 디코더는 인코더에서 추출한 짧은 기간과 긴 기간의 시계열 데이터의 표현을 효율적으로 결합 및 학습하는 구조로 설계한다. 광범위한 실험을 통해 3DCformer가 기존 State-Of-The-Art 모델에 비해 16.18% 낮은 MAE와 24.53% 낮은 MSE를 달성함을 보인다.
Abstract
As the importance of time-series data increases in many industrial fields, the problem of time-series forecasting is becoming crucial. For time-series forecasting, Transformer-based models have recently received much attention in place of traditional LSTM-based models. However, Transformer-based models have a limitation in their weak ability to preserve sequential features of time-series. To address this problem, we propose a novel Transformer-based forecasting model, called 3DCformer, which combines a 3D convolution network and Transformer. The 3DCformer encoder uses dilated convolution to capture sequential features from time-series and then expands 2D time-series to 3D. The 3D convolution network then extracts representations from the extended 3D data. The 3DCformer decoder is designed to efficiently combines and learns representations of both short-term and long-term features extracted from the encoder. Through extensive experiments, we show that our proposed 3DCformer achieves 16.18% lower MAE and 24.53% lower MSE compared to the existing State-Of-The-Art models.
참고문헌 (41)
- S. M. Nosratabadi, "A comprehensive review on microgrid and virtual power plant concepts employed for distributed energy resources scheduling in power systems", Renewable and Sustainable Energy Reviews, 67, pp.341~363 (2017)
- Y. Matsubara, "Funnel : automatic mining of spatially coevolving epidemics", Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.105~114 (2014)
- S. Papadimitriou, "Optimal multiscale patterns in time series streams", Proceedings of the 2006 ACM SIGMOD international conference on Management of data, pp.647~658 (2006)
- Y. Zhu, "Statstream : Statistical monitoring of thousands of data streams in real time", VLDB’02: Proceedings of the 28th International Conference on Very Large Databases, pp.358~369 (2002), Elsevier
- S. Hochreiter, "Long shortterm memory", Neural computation, 9(8), pp.1735~1780 (1997)
- A. Vaswani, "Attention is all you need", Advances in neural information processing systems, 30 (2017)
- H. Zhou, "Informer : Beyond efficient transformer for long sequence timeseries forecasting", Proceedings of the AAAI Conference on Artificial Intelligence, 35(12), pp.106~115 (2021)
- D. E. Rumelhart, "Learning representations by backpropagating errors", nature, 323(6088), pp.533~536 (1986)
- L. Wu, "Random warping series : A random features method for time-series embedding", International Conference on Artificial Intelligence and Statistics, pp.793~802 (2018), PMLR
- J. -Y. Franceschi, "Unsupervised scalable representation learning for multivariate time series", Advances in neural information processing systems, 32 (2019)
- C. Lea, "Temporal convolutional networks for action segmentation and detection", proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.156~165 (2017)
- Q. Le, "Distributed representations of sentences and documents", International conference on machine learning, pp.1188~1196 (2014), PMLR
- M. Mishra, "Deep learning and wavelet transform integrated approach for short-term solar pv power prediction", Measurement, 166, pp.108250~ (2020)
- Y. -Y. Hong, "Day-ahead solar irradiation forecasting utilizing gramian angular field and convolutional long short-term memory", IEEE Access, 8, pp.18741~18753 (2020)
- I. Sutskever, "Sequence to sequence learning with neural networks", Advances in neural information processing systems, 27 (2014)
- G. Zerveas, "A transformerbased framework for multivariate time series representation learning", Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.2114~2124 (2021)
- C. Feichtenhofer, "X3d : Expanding architectures for efficient video recognition", Proceedings of the IEEE, 2020, 203~213
- G. Li, "Recurrent neural networks based photovoltaic power forecasting approach", Energies, 12(13), pp.2538~ (2019)
- M. Chai, "Pv power prediction based on lstm with adaptive hyperparameter adjustment", Ieee Access, 7, pp.115473~115486 (2019)
- B. Ray, "A new data driven long-term solar yield analysis model of photovoltaic power plants", IEEE Access, 8, pp.136223~136233 (2020)
- D. Yu, "Forecasting photovoltaic power generation using satellite images", Energies, 13(24), pp.6603~ (2020)
- H. Zhou, "Short-term photovoltaic power forecasting based on long short term memory neural network and attention mechanism", Ieee Access, 7, pp.78063~78074 (2019)
- H. Song, "Attend and diagnose: Clinical time series analysis using attention models", Proceedings of the AAAI conference on artificial intelligence, 32(1) (2018)
- S. Li, "Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting", Advances in neural information processing systems, 32 (2019)
- K. He, "Deep residual learning for image recognition", Proceedings of the IEEE conference on computer vision and pattern recognition, 770(778), pp.770~778 (2016)
- M. Rhudy, "Microphone array analysis methods using cross-correlations", ASME International Mechanical Engineering Congress and Exposition, 43888, pp.281~288 (2009)
- A. A. Ariyo, "Stock price prediction using the arima model", 2014 UKSim-AMSS 16th international conference on computer modelling and simulation, 106(112), pp.106~112 (2014), IEEE
- S. J. Taylor, "Forecasting at scale", The American Statistician, 72(1), pp.37~45 (2018)