PIGan: Pose Invariant face reenactment for game character using a GAN-based face rotation technique
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The recent emergence of generative adversarial networks (GANs) accelerate the research on face reenactment. Face reenactment is a technique for synthesizing a face image of desired expressions from a source face using a target image of desired expressions or a vector with facial expression information. In this paper, we propose a GAN architecture-based method for reenacting the expressions of game characters with various angles through a rotation module and action unit (AU) vectors. In the first step, we devise a rotate module that synthesizes the frontalized face images from the face of a game character with arbitrary poses. This prevents the side effects of the face poses in reenacting the facial expressions. We feed the frontalized image as well as a vector with facial expression information to the generator in order to synthesize a face with the desired expression. The generator generates an attention mask indicating a region of interest of facial expression and a color mask indicating color information for the expression. This allows us to synthesize a reenacted image that generates facial expressions while preserving the identity and other attachments of a game character. The attention mask and color mask incorporate to synthesize a reenacted image with target expression. Finally, we recover the original pose of a character from the frontalized reenacted image.
Abstract
The recent emergence of generative adversarial networks (GANs) accelerate the research on face reenactment. Face reenactment is a technique for synthesizing a face image of desired expressions from a source face using a target image of desired expressions or a vector with facial expression information. In this paper, we propose a GAN architecture-based method for reenacting the expressions of game characters with various angles through a rotation module and action unit (AU) vectors. In the first step, we devise a rotate module that synthesizes the frontalized face images from the face of a game character with arbitrary poses. This prevents the side effects of the face poses in reenacting the facial expressions. We feed the frontalized image as well as a vector with facial expression information to the generator in order to synthesize a face with the desired expression. The generator generates an attention mask indicating a region of interest of facial expression and a color mask indicating color information for the expression. This allows us to synthesize a reenacted image that generates facial expressions while preserving the identity and other attachments of a game character. The attention mask and color mask incorporate to synthesize a reenacted image with target expression. Finally, we recover the original pose of a character from the frontalized reenacted image.
참고문헌 (33)
- Goodfellow, I., "Generative adversarial nets", Proc. of NIPS 2014, pp.2672~2680 (2014)
- Sagonas, C., "300 faces in-the-wild challenge: The first facial landmark localization challenge", Proc. of ICCV 2013, pp.397~403 (2013)
- Liu, Y., "Grand challenge of 106-point facial landmark localization", Proc. of ICMEW 2019, pp.613~616 (2019)
- Gu, K., "FLNet:Landmark driven fetching and learning network for faithful talking facial animation synthesis", Proc. of AAAI 2020, pp.10861~10868 (2020)
- Zhang, J., "Freenet:Multi-identity face reenactment", Proc. of CPVR 2020, pp.5326~5335 (2020)
- Wu, W., "Reenactgan: Learning to reenact faces via boundary transfer", Proc. of ECCV 2018, pp.603~619 (2018)
- Friesen, E, "Facial action coding system: a technique for the measurement of facial movement" (1978), Consulting Psychologists Press
- Du, S., "Compound facial expressions of emotion", Proc. of NAS, 111(15), pp.1454~1462 (2014)
- Pumarola, A., "Ganimation: One-shot anatomically consistent facial animation", Proc. of ECCV 2018, pp.818~833 (2018)
- Ling, J., "Toward fine-grained facial expression manipulation", Proc. of ECCV 2020, pp.37~53 (2020)
- Yin, X., "Towards large-pose face frontalization in the wild", Proc. of ICCV 2017, pp.3990~3999 (2017)
- Deng, J., "Uv-gan: Adversarial facial uv map completion for pose-invariant face recognition", Proc. of CVPR 2018, pp.7093~7102 (2018)
- Cao, J., "Learning a high fidelity pose invariant model for high-resolution face frontalization", arXiv preprint arXiv : 1806. 08472, ~ (2018)
- Zhou, H., "Rotate-and-render: Unsupervised photo-realistic face rotation from single-view images", Proc. of CVPR 2020, pp.5911~5920 (2020)
- Arjovsky, M., "Wasserstein generative adversarial networks", Proc. of ICML 2017, pp.214~223 (2017)
- Gulrajani, I., "Improved training of wasserstein gans", Proc. of NIPS 2017, pp.5767~5777 (2017)
- Zhang, H., "Stackgan:Text to photo-realistic image synthesis with stacked generative adversarial networks", Proc. of ICCV 2017, pp.5907~5915 (2017)
- Karras, T., "A style-based generator architecture for generative adversarial networks", Proc. of CVPR 2019, pp.4401~4410 (2019)
- Wang, Z., "Deep networks for image superresolution with sparse prior", Proc. of ICCV 2015, pp.370~378 (2015)
- Ledig, C., "Photo-realistic single image superresolution using a generative adversarial network", Proc. of CVPR 2017, pp.4681~4690 (2017)
- Karras, T., "Progressive growing of GANs for improved quality, stability, and variation", Proc. of ICLR 2018 (2018)
- Zhu, J. Y., "Unpaired image-to-image translation using cycle-consistent adversarial networks", Proc. of ICCV 2017, pp.2223~2232 (2017)
- Kim, T., "Learning to discover cross-domain relations with generative adversarial networks", Proc. of ICML 2017, pp.1857~1865 (2017)
- Choi, Y., "Stargan: Unified generative adversarial networks for multi-domain image-to-image translation", Proc. of CVPR 2018, pp.8789~8797 (2018)
- Ronneberger, O., "Convolutional networks for biomedical image segmentation", Proc. of MICCAI 2015, pp.234~241 (2015)
- Mirza, M, "Conditional generative adversarial nets", arXiv preprint arXiv:1411.1784, ~ (2014)
- Baltrušaitis, T., "Openface: an open source facial behavior analysis toolkit", Proc. of WACV 2016, pp.1~10 (2016)
- Zhu, X., "Face alignment across large poses: A 3d solution", Proc. of CVPR 2016, pp.146~155 (2016)
- He, K., "Deep residual learning for image recognition", Proc. of CVPR 2016, pp.770~778 (2016)
- Isola, P., "Image-to-image translation with conditional adversarial networks", Proc. of CVPR 2017 (2017)
- Bang, D., "Improved training of generative adversarial networks using representative features", Proc. of ICML 2018, pp.433~442 (2018)
- "PyTorch" (2016)[URL]
- Tripathy, S., "Icface:Interpretable and controllable face reenactment using gans", Proc. of WACV 2020, pp.3385~3394 (2020)