Textual Inversion을 활용한 Adversarial Prompt 생성 기반 Text-to-Image 모델에 대한 멤버십 추론 공격
Membership Inference Attack against Text-to-Image Model Based on Generating Adversarial Prompt Using Textual Inversion
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
최근 생성 모델이 발전함에 따라 생성 모델을 위협하는 연구도 활발히 진행되고 있다. 본 논문은 Text-to-Image 모델에 대한 멤버십 추론 공격을 위한 새로운 제안 방법을 소개한다. 기존의 Text-to-Image 모델에 대한 멤버십 추론 공격은 쿼리 이미지의 caption으로 단일 이미지를 생성하여 멤버십을 추론하였다. 반면, 본 논문은 Textual Inversion을 통해 쿼리 이미지에 personalization된 임베딩을 사용하고, Adversarial Prompt 생성 방법으로 여러 장의 이미지를 효과적으로 생성하는 멤버십 추론 공격을 제안한다. 또한, Text-to-Image 모델 중 주목받고 있는 Stable Diffusion 모델에 대한 멤버십 추론 공격을 최초로 진행하였으며, 최대 1.00의 Accuracy를 달성한다.
Abstract
In recent years, as generative models have developed, research that threatens them has also been actively conducted. We propose a new membership inference attack against text-to-image model. Existing membership inference attacks on Text-to-Image models produced a single image as captions of query images. On the other hand, this paper uses personalized embedding in query images through Textual Inversion. And we propose a membership inference attack that effectively generates multiple images as a method of generating Adversarial Prompt. In addition, the membership inference attack is tested for the first time on the Stable Diffusion model, which is attracting attention among the Text-to-Image models, and achieve an accuracy of up to 1.00.
참고문헌 (21)
- Y. Wu, "Membership inference attacks against text-to-image generation models" (2022)
- R. Rombach, "High-resolution image synthesis with latent diffusion models", Proceedings of the IEEE, 2022, 10684~10695
- A. Ramesh, "Hierarchical text-conditional image generation with clip latents" (2022)
- J. Hayes, "LOGAN : Membership inference attacks against generative models" (2017)
- K. S. Liu, "Performing co-membership attacks against deep generative models", Proceedings of the 19th IEEE International Conference on Data Mining, pp.459~467 (2019)
- B. Hilprecht, "Monte carlo and reconstruction membership inference attacks against generative models", Proceedings on Privacy Enhancing Technologies, pp.232~249 (2019)
- D. Chen, "GAN-Leaks : A taxonomy of membership inference attacks against generative models", Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pp.343~362 (2020)
- N. Maus, "Adversarial prompting for black box foundation models" (2023)
- H. Zhuang, "A pilot study of query-free adversarial attack against stable diffusion", Proceedings of the IEEE, 2023, 2384~2391
- Q. Liu, "Intriguing properties of text-guided diffusion models" (2023)
- H. Liu, "RIATIG : Reliable and imperceptible adversarial text-to-image generation with natural prompts", Proceedings of the IEEE, 2023, 20585~20594
- dR. Gal, "An image is worth one word : personalizing text-to-image generation using textual inversion" (2022)
- Z. Fei, "Gradient-free textual inversion" (2023)
- R. Gal, "Encoder-based domain tuning for fast personalization of text-to-image models", ACM Transactions on Graphics (TOG), 42(4), pp.1~13 (2023)
- L. Melas-Kyriazi, "Realfusion : 360° reconstruction of any object from a single image", Proceedings of the IEEE, 2023, pp.6, 8446~8455
- N. Carlini, "Extracting training data from diffusion models", 32nd USENIX Security Symposium(USENIX Security 23), pp.5253~5270 (2023)
- C. Schuhmann, "Laion-5b: An open large-scale dataset for training next generation image-text models", Advances in Neural Information Processing Systems (2022)
- S. Changpinyo, "Conceptual 12m : Pushing web-scale image-text pre-training to recognize long-tail visual concepts", Proceedings of the IEEE, 2021, 25278~25294
- A. Botchkarev, "Performance metrics (error measures) in machine learning regression, forecasting and prognostics: Properties and typology" (2018)
- A. Horé, "Image quality metrics : PSNR vs. SSIM", 20th international conference on pattern recognition, pp.2366~2369 (2010), IEEE
- Z. Wang, "Image quality assessment : from error visibility to structural similarity", IEEE transactions on image processing, 13(4), pp.600~612 (2004)