특성맵 차분을 활용한 커널 기반 비디오 프레임 보간 기법
Kernel-Based Video Frame Interpolation Techniques Using Feature Map Differencing
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
비디오 프레임 보간(Video Frame Interpolation)은 움직임의 연속성을 증가시켜 영상을 부드럽게 재생할 수 있어 영상, 미디어 분야에서 사용되는중요한 기술이다. 딥러닝 기반 비디오 프레임 보간 연구에서 널리 사용되는 방법 중 하나인 커널 기반 방법(Kernel Based Method)의 경우, 지역적인변화를 잘 포착하지만 전체적인 변화를 처리하는 데 한계가 있었다. 이에 본 논문에서는 주요 변화 포착에 집중하기 위한 특성맵 차분, Two Direction을 적용한 새로운 U-Net 구조를 통해 파라미터 수를 줄이면서 중간 프레임을 보다 정확하게 생성하고자 한다. 실험 결과 제안한 구조가 기존보다Vimeo, Middle-burry 등의 일반적인 데이터셋과 새로운 YouTube 데이터셋에서 기존 모델보다 약 61% 더 적은 파라미터로 PSNR 수치가 최대0.3 우수한 성능을 달성하였다. 본 논문에서 사용한 코드는 https://github.com/Go-MinSeong/SF-AdaCoF에서 확인 가능하다.
Abstract
Video frame interpolation is an important technique used in the field of video and media, as it increases the continuity of motionand enables smooth playback of videos. In the study of video frame interpolation using deep learning, Kernel Based Method captureslocal changes well, but has limitations in handling global changes. In this paper, we propose a new U-Net structure that applies featuremap differentiation and two directions to focus on capturing major changes to generate intermediate frames more accurately while reducingthe number of parameters. Experimental results show that the proposed structure outperforms the existing model by up to 0.3 in PSNRwith about 61% fewer parameters on common datasets such as Vimeo, Middle-burry, and a new YouTube dataset. Code is available athttps://github.com/Go-MinSeong/SF-AdaCoF.
참고문헌 (20)
- J. Huh, "Research trends in deep learning-based video frame interpolation techniques", The Korean Institute of Broadcast and Media Engineers : Broadcast and Media Maganize, 18(2), pp.51~61 (2022)
- J. Dong, "Video frame interpolation : A comprehensive survey", ACM Transactions on Multimedia Computing, Communications, and Applications, 19(78), pp.1~31 (2023)
- H. Lee, "AdaCoF: Adaptive collaboration of flows for video frame interpolation", Proceedings of the IEEE, 2020, ~
- J. Dai, "Deformable convolutional networks", Proceedings of the IEEE, 2020, ~
- O. Ronneberger, "U-Net : Convolutional networks for biomedical image segmentation", Proceedings of the IEEE, 2015, ~
- S. Niklaus, "Video frame interpolation via adaptive convolution", Proceedings of the IEEE, 2017, ~
- K. M. Briedis, "Kernelbased frame interpolation for spatio-temporallyadaptive rendering", SIGGRAPH '23 : ACM SIGGRAPH 2023 Conference Proceedings (2023)
- Z. Teed, "RAFT : Recurrent all-pairs field transforms for optical flow", ECCV(European Conference on Computer Vision) 2020, pp.402~419 (2020)
- Z. Huang, "FlowFormer : A transformer architecture for optical flow", ECCV(European Conference on Computer Vision) 2022, pp.668~685 (2022)
- G. Zhang, "Extracting motion and appearance via inter-frame attention for efficient video frame interpolation", Proceedings of the IEEE, 2023, 5682~5692
- N. Singla, "Motion detection based on frame difference method", International Journal of Information & Computation Technology, 4(15), pp.1559~1565 (2014)
- S. Niklaus, "Video frame interpolation via adaptive separable convolution", Proceedings of the IEEE, 2017, ~
- O. Köpüklü, "Convolutional neural networks with layer reuse", Proceedings of the IEEE, 2019, ~
- "Video Enhancement with Task-Oriented Flow"[URL]
- D. Scharstein, "High-resolution stereo datasets with subpixel-accurate ground truth", German Conference on Pattern Recognition (GCPR 2014) (2014)
- "A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation"[URL]
- "UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild"[URL]
- D. P. Kingma, "Adam : A Method for Stochastic Optimization", Conference Paper at the 3rd International Conference for Learning Representations (2015)
- A. Horé, "Image quality metrics: PSNR vs. S", 2010 20th International Conference on Pattern Recognition (2010)
- J. Johnson, "Perceptual losses for real-time style transfer and super-resolution", Proceedings of the IEEE, 2016, ~