Urban Landscape Game Scene Sketch Generation Framework with Stable Diffusion
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Games are a vital part of our culture, leading the generation of tools such as Adobe Photoshop and Unity for game developers. Although sketches are a fundamental form that can be stylized in various ways, there is a scarcity of tools capable of generating images into sketch. To bridge this gap in the artistic sketch domain of gaming and the field of deep generative models, we propose a multimodal sketch generation framework with Stable Diffusion, focusing especially on urban scenes. Stable Diffusion, a model within the diffusion-based category, has gained notable attention in the open-source community and is user-friendly. Thus, we have chosen to utilize Stable Diffusion in our approach. This model processes input prompts and images through the CLIP encoder and effectively generates images. However, Stable Diffusion faces challenges such as a losing locality of input image and difficulties in fine-tuning. To overcome these issues, we incorporate ControlNet and DreamBooth into our framework. We conclude with a demonstration of promising results, urban landscape sketch, in both text-to-sketch and image-to-sketch generation.
Abstract
Games are a vital part of our culture, leading the generation of tools such as Adobe Photoshop and Unity for game developers. Although sketches are a fundamental form that can be stylized in various ways, there is a scarcity of tools capable of generating images into sketch. To bridge this gap in the artistic sketch domain of gaming and the field of deep generative models, we propose a multimodal sketch generation framework with Stable Diffusion, focusing especially on urban scenes. Stable Diffusion, a model within the diffusion-based category, has gained notable attention in the open-source community and is user-friendly. Thus, we have chosen to utilize Stable Diffusion in our approach. This model processes input prompts and images through the CLIP encoder and effectively generates images. However, Stable Diffusion faces challenges such as a losing locality of input image and difficulties in fine-tuning. To overcome these issues, we incorporate ControlNet and DreamBooth into our framework. We conclude with a demonstration of promising results, urban landscape sketch, in both text-to-sketch and image-to-sketch generation.
참고문헌 (10)
- He, K., "Deep residual learning for image recognition", CVPR 2016, pp.770~778 (2016)
- Ronneberger, O., "U-net: Convolutional networks for biomedical image segmentation", MICCAI 2015, pp.234~241 (2015)
- Goodfellow, I., "Generative adversarial nets" (2014), NeuIPS
- Isola, P., "Image-to image translation with conditional adversarial networks", CVPR 2017, pp.1125~1134 (2017)
- Zhu, J., "Unpaired image-to-image translation using cycle-consistent adversarial networks", CVPR 2021, pp.2223~2232 (2021)
- Song, J., "Denoising diffusion implicit models", ICLR 2020 (2020)
- Rombach, R., "High-resolution image synthesis with latent diffusion models", CVPR 2022, pp.10684~10695 (2022)
- Radford, A., "Learning transferable visual models from natural language supervision", ICML 2021, pp.8748~8763 (2021)
- Zhang, L., "Adding conditional control to text-to-image diffusion models" (2023)
- Ruiz, N., "Dreambooth:Fine tuning text-to-image diffusion models for subject-driven generation", CVPR 2023, pp.22500~22510 (2023)