적대적 생성 신경망을 통한 얼굴 비디오 스타일 합성 연구
Style Synthesis of Speech Videos Through Generative Adversarial Neural Networks
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 연구에서는 기존의 동영상 합성 네트워크에 스타일 합성 네트워크를 접목시켜 동영상에 대한 스타일 합성의 한계점을 극복하고자 한다. 본 논문의 네트워크에서는 동영상 합성을 위해 스타일갠 학습을 통한 스타일 합성과 동영상 합성 네트워크를 통해 스타일 합성된 비디오를 생성하기위해 네트워크를 학습시킨다. 인물의 시선이나 표정 등이 안정적으로 전이되기 어려운 점을 개선하기 위해 3차원 얼굴 복원기술을 적용하여 3차원얼굴 정보를 이용하여 머리의 포즈와 시선, 표정 등의 중요한 특징을 제어한다. 더불어, 헤드투헤드++ 네트워크의 역동성, 입 모양, 이미지, 시선처리에 대한 판별기를 각각 학습시켜 개연성과 일관성이 더욱 유지되는 안정적인 스타일 합성 비디오를 생성할 수 있다. 페이스 포렌식 데이터셋과메트로폴리탄 얼굴 데이터셋을 이용하여 대상 얼굴의 일관된 움직임을 유지하면서 대상 비디오로 변환하여, 자기 얼굴에 대한 3차원 얼굴 정보를이용한 비디오 합성을 통해 자연스러운 데이터를 생성하여 성능을 증가시킴을 확인했다.
Abstract
In this paper, the style synthesis network is trained to generate style-synthesized video through the style synthesis through trainingStylegan and the video synthesis network for video synthesis. In order to improve the point that the gaze or expression does not transferstably, 3D face restoration technology is applied to control important features such as the pose, gaze, and expression of the head using3D face information. In addition, by training the discriminators for the dynamics, mouth shape, image, and gaze of the Head2head network,it is possible to create a stable style synthesis video that maintains more probabilities and consistency. Using the FaceForensic datasetand the MetFace dataset, it was confirmed that the performance was increased by converting one video into another video while maintainingthe consistent movement of the target face, and generating natural data through video synthesis using 3D face information from thesource video’s face.
참고문헌 (36)
- T. Karras, "A style-based generator architecture for generative adversarial networks", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2019)
- T. Karras, "Analyzing and improving the image quality of stylegan", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2020)
- M. J. Chong, "GANs N’ roses: Stable, controllable, diverse image to image translation", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2021)
- Luan Tran, "On Learning 3D Face Morphable Model from In-the-wild Images", IEEE Transactions on Pattern Analysis and Machine Intelligence, pp.1~1 (2019), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- T. C. Wang, "Video-to-video synthesis", Advances in Neural Information Processing Systems (2018)
- A. Rossler, "FaceForensics++ : Learning to detect manipulated facial images", Proceedings of the IEEE International Conference on Computer Vision (2019)
- M. R. Koujan, "Head2Head : Video-based neural head synthesis", Proceedings - 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (2020)
- M. C. Doukas, "Head2Head++ : Deep facial attributes re-targeting", arXiv e-prints arXiv : 2006. 10199, ~ (2020)
- "MetFace dataset"[URL]
- R. Zhang, "The unreasonable effectiveness of deep features as a perceptual metric", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2018)
- D. Bank, "Autoencoder", arXiv preprint arXiv : 2003. 05991, ~ (2020)
- D. P. Kingma, "Auto-encoding variational bayes", 2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings (2014)
- I. J. Goodfellow, "Generative adversarial nets", Advances in Neural Information Processing Systems (2014)
- A. Radford, "Unsupervised representation learning with deep convolutional GANs", International Conference on Learning Representations (2016)
- X. Huang, "Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization", Proceedings of the IEEE International Conference on Computer Vision (2017)
- X. Han, "ProGAN : Protein solubility generative adversarial nets for data augmentation in DNN framework", Computers and Chemical Engineering, 131 (2019)
- M. D. Hoffman, "Stochastic variational inference", Journal of Machine Learning Research, 14 (2013)[DOI]
- N. -A. Lahonce, "Flickr-Faces-HQ Dataset(FFHQ)" (2020), Nvidia
- Z. Zhang, "Generalized cross entropy loss for training deep neural networks with noisy labels", Advances in Neural Information Processing Systems, 31 (2018)
- Derek A. Pisner, "Support vector machine", Machine Learning, pp.101~121 (2020), Elsevier[DOI]
- A. Mathiasen, "Fast fréchet inception distance", arXiv preprint arXiv : 2009. 14075, ~ (2020)
- O. Nizan, "Breaking the cycle-colleagues are all you need", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2020)
- P. Isola, "Pix2Pix", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
- J. Y. Zhu, "Unpaired image-to-image translation using cycle-consistent adversarial networks", arXiv preprint arXiv : 1703. 10593, ~ (2017)
- Jun Han, "FlowNet: A Deep Learning Framework for Clustering and Selection of Streamlines and Stream Surfaces", IEEE Transactions on Visualization and Computer Graphics, 26(4), pp.1732~1744 (2020), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- E. Ilg, "FlowNet 2.0: Evolution of optical flow estimation with deep networks", Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (2017)
- Liuchun Yuan, "Image Inpainting Based on Patch-GANs", IEEE Access, 7, pp.46411~46421 (2019), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- X. Mao, "LSGAN", Proceedings of the IEEE International Conference on Computer Vision (2017)
- K. Zhang, "MTCNN", IEEE Signal Processing Letters, 23(10) (2016)
- H. Wang, "Normalized object coordinate space for category-level 6D object pose and size estimation", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2019)
- B. J. B. Rani, "Survey on applying GAN for anomaly detection", 2020 International Conference on Computer Communication and Informatics (ICCCI) (2020)
- Justin Johnson, "Perceptual Losses for Real-Time Style Transfer and Super-Resolution", Computer Vision – ECCV 2016, pp.694~711 (2016), Springer International Publishing[DOI]
- H. Y. Lee, "DRIT++ : Diverse Image-to-Image Translation via Disentangled Representations", arXiv preprint arXiv : 1905. 01270, ~ (2019)
- E. Härkönen, "GANSpace : Discovering interpretable GAN controls", Advances in Neural Information Processing Systems, 33, pp.9841~9850 (2020)
- X. Zhu, "Face alignment in full pose range : A 3D total solution", arXiv preprint arXiv : 1804. 01005, ~ (2018)
- J. H. Lee, "Learning dense representa tions of phrases at scale", arXiv preprint arXiv : 2012. 12624, ~ (2020)