본문 바로가기
KERD
논문KCI 등재

스케치 기반 이미지 검색의 최신 연구 동향

Recent advances in sketch based image retrieval: a survey

오세홍|석호식
한국전기전자학회 (Institute of Korean Electrical and Electronics Engineers)|2024.06|전기전자학회논문지 (Journal of IKEEE)|제28권 제2호|pp. 209-220|DOI 10.7471/ikeee.2024.28.2.209

1

조회

0

다운로드

0

인용

0

참고문헌

키워드

image recognitionsketch recognitionsketch based image retrievalzero-shot learningimage generation
주제분류전자/정보통신공학

초록

스케치는 직관적으로 정보를 표현할 수 있는 수단이나 실제 이미지와 비교하였을 때 추상화가 심하고 동일한 대상에 대한 스케치임에도 변화가 매우 크다는 문제점이 있다. 따라서 스케치에 기반한 이미지 인식에 도움이 되는 특성을 표현하기 어려웠는데, 딥뉴럴모델의 발전은 이미지와 스케치라는 두 도메인에 공통으로 존재하는 특성을 발견할 수 있게 해 주었다. 본 논문에서는 스케치에 기반한 이미지 검색 기술의 최신 연구 동향을 소개한다. 이미지 검색 외에 스케치에 기반한 이미지 인식과 이미지 생성의 연구 동향도함께 요약하였으며, 스케치 기반 연구에서 주로 활용되는 데이터셋 및 해당 데이터셋에 기반하여 측정한 검색 성능을 포함하였다

Abstract

A sketch is an intuitive means to express information, but compared to actual images, it has the problem ofbeing highly abstract, diverse, and sparse. Recent advances in deep learning models have made it possible todiscover features that are common to images and sketches. In this paper, we summarize recent trends in sketchbasedimage retrieval (SBIR) but it is not limited to SBIR. Besides SBIR, we also introduce sketch-based imagerecognition and generation studies. Zero-shot learning enables models to recognize categories not encounteredduring training. Zero-shot SBIR methods are also discussed. Commonly used free-hand sketch datasets aresummarized and retrieval performance based on these datasets is reported.

참고문헌 (68)

  1. A. Sain, "StyleMeUp : towards style-agnostic sketch-based image retrieval", Proc. of 2021IEEE, 2021, ~
  2. S. Koley, "Picture that sketch : photorealistic image generation from abstract sketches", Proc. of 2023 IEEE, 2023, ~
  3. P. Xu, "Deep learning for freehand sketch : a survey", IEEE Trans. Pattern Anal. Mach. Intell, 45(1), pp.285~312 (2022)[DOI]
  4. R. Hu, "A performance evaluation of gradient field HOG descriptor for sketch based image retrieval", CVIU, 117(7), pp.790~806 (2013)[DOI]
  5. S. K. Yelamarthi, "A zero-shot framework for sketch based image retrieval", Proc. the 15th European Conference on Computer Vision (ECCV2018) (2018)
  6. F. Lin, "Zero-shot everything sketch-based image retrieval, and in explainable style", Proc. of 2023 IEEE, 2023, ~
  7. H. Zhang, "SketchNet: sketch classification with web images", Proc. of 2016 IEEE, 2016, ~
  8. X. Zhang, "A Hybrid convolutional neural network for sketch recognition", Pattern Recognit. Lett, 130, pp.73~82 (2020)[DOI]
  9. K. Zhang, "Cousin network guided sketch recognition via latent attribute warehouse", Proc. of the AAAI Conference on Artificial Intelligence (AAAI2019), pp.9203~9210 (2019)
  10. G. Jain, "TransSketchNet : attention-based sketch recognition using transformers", Proc. of the 24th European Conference on Artificial Intelligence (ECAI 2020) (2020)
  11. L. Li, "Sketch-R2CNN : An Attentive Network for Vector Sketch Recognition", IEEE Trans. Vis. Comput. Graph, 27(9), pp.3745~3754 (2021)
  12. A. Tripathi, "Sketch-Guided Object Localization in Natural Images", Proc. of the 16th European Conference on Computer Vision (ECCV2020) (2020)
  13. P. N. Chowdhury, "What can human sketches do for object detection?", Proc. of 2023 IEEE, 2023, ~
  14. S. Woo, "Sketch-based video object localization", Proc. of IEEE, 2024, ~
  15. J. Song, "Learning to Sketch with Shortcut Cycle Consistency", Proc. of 2018 IEEE, 2018, ~
  16. J. -Y. Zhu, "Unpaired image-to-image translation using cycleconsistent adversarial networks", Proc. of 2017IEEE, 2017, ~
  17. D. Ha, "A neural representation of sketch drawings", Proc. of the 6th International Conference on Learning Representations (ICLR2018) (2018)
  18. F. Huang, "Sketchforme : Composing Sketched Scenes from Text Descriptions for Interactive Applications", Proc. of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST19) (2019)
  19. X. Xing, "DiffSketcher : text guided vector sketch synthesis through latent diffusion models", Proc. of the 28th Annual Conference on Neural Information Processing Systems (NeurIPS 2024) (2024)
  20. K. Sasaki, "Adaptive drawing behavior by visuomotor learning using recurrent neural networks", IEEE Trans. Cogn. Dev. Syst, 11(1), pp.119~128 (2019)[DOI]
  21. S. Ge, "Creative sketch generation", Proc. of International Conference on Learning Representations (ICLR2021) (2021)
  22. A. Das, "Cloud2Curve : generation and vectorization of parametric sketches", Proc. of 2021 IEEE, 2021, ~
  23. Q. Wang, "SketchKnitter : vectorized sketch generation with diffusion models", Proc. of International Conference on Learning Representations (ICLR2023) (2023)
  24. T. Karras, "A style-based generator architecture for generative adversarial networks", Proc. of 2019 IEEE, 2019, ~
  25. K. He, "Deep residual learning for image recognition", Proc. of 2016 IEEE, 2016, ~
  26. L. Gao, "SketchFaceNeRF : sketch-based facial generation and editing in neural radiance fields", ACM Trans. Graph, 42(4), pp.1~17 (2023)[DOI]
  27. T. Karras, "Analyzing and improving the image quality of stylegan", Proc. of 2020IEEE, 2020, ~
  28. E. R. Chan, "Efficient geometry-aware 3D generative adversarial networks", Proc. of 2022IEEE, 2022, ~
  29. A. Voynov, "Sketch-guided text-to-image diffusion models", Proc. of Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH '23) (2023)
  30. A. Mikaeili, "SKED : sketch-guided textbased 3D editing", Proc. 2023 IEEE, 2023, ~
  31. Y. Li, "A survey of sketch-based image retrieval", Mach. Vis. Appl, 29, pp.1083~1100 (2018)
  32. F. Yang, "A systematic literature review of deep learning approaches for sketch-based image retrieval : datasets, metrics, and future directions", IEEE Access, 12, pp.14847~14869 (2024)[DOI]
  33. A. Creswell, "Adversarial training for sketch retrieval", Proc. of the 14th European Conference on Computer Vision (ECCV 2016) (2016)
  34. J. Song, "Deep spatial-semantic attention for fine-grained sketch-based image retrieval", Proc. of 2017 IEEE International Conference on Computer Vision (ICCV2017) (2017)
  35. L. Liu, "Deep Sketch Hashing : Fast Free-Hand Sketch-Based Image Retrieval", Proc. of 2017 IEEE, 2017, ~
  36. L. Guo, "Sketch-based image retrieval using generative adversarial networks", Proc. of the 25th ACM International Conference on Multimedia (MM17) (2017)
  37. T. Portenier, "SmartSketcher : sketch-based image retrieval with dynamic semantic re-ranking", Proc. of the Symposium on Sketch-Based Interfaces and Modeling(SBIM’17) (2017)
  38. D. Yu, "A multi-layer deep fusion convolutional neural network for sketch based image retrieval", Neurocomputing, 296, pp.23~32 (2018)[DOI]
  39. T. Bui, "Sketching out the details : Sketch-based image retrieval using convolutional neural networks with multi-stage regression", Comput. Graph, 71, pp.77~87 (2018)[DOI]
  40. H. Fei, "Sketch-based image retrieval with deep visual semantic descriptor", Pattern Recognit, 76, pp.537~548 (2018)[DOI]
  41. H. Zhang, "Learning structural representations via dynamic object landmarks discovery for sketch recognition and retrieval", IEEE Trans. Image Process, 28(9), pp.4486~4499 (2019)[DOI]
  42. F. Huang, "Swire : sketch-based user interface retrieval", Proc. of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19) (2019)
  43. M. Bhattarai, "Diagram image retrieval using sketch-based deep learning and transfer learning", Proc. of 2020 IEEE, 2020, ~
  44. C. Bai, "Cross-domain representation learning by domainmigration generative adversarial network for sketch based image retrieval", J. Vis. Commun. Image R, 71, pp.102835~ (2020)[DOI]
  45. Z. Zhang, "Zero-shot sketch-based image retrieval via graph convolution network", Proc. of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) (2020)
  46. A. Fuentes, "Sketch-QNet : a quadruplet convnet for color sketch-based image retrieval", Proc. of 2021 IEEE, 2021, ~
  47. P. Lu, "Domain-aware SE network for sketchbased image retrieval with multiplicative euclidean margin softmax", Proc. of the 29th ACM International Conference on Multimedia (MM ’21) (2021)
  48. J. Xu, "DLA-Net for FG-SBIR : dynamic local aligned network for fine-grained sketch-based image retrieval", Proc. of the 29th ACM International Conference on Multimedia (MM ’21) (2021)
  49. A. K. Bhunia, "Sketching without worrying : noise-tolerant sketchbased image retrieval", Proc. of 2022 IEEE, 2022, ~
  50. Dai, "Multi-granularity association learning for on-thefly fine-grained sketch-based image retrieval", Knowl. -Based Syst, 253(109447) (2022)[DOI]
  51. Z. Ling, "Multi-level region matching for fine-grained sketch-based image retrieval", Proc. of the 30th ACM International Conference on Multimedia (MM ’22) (2022)
  52. X. Liu, "Differentiable auxiliary learning for sketch re-identifcation", Proc. of the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24) (2024)
  53. S. Koley, "How to handle sketch-abstraction in sketch-based image retrieval?", 2024 IEEE, 2024, ~
  54. Q. Liu, "Semantic-aware knowledge preservation for zero-shot sketch-based image retrieval", Proc. of 2019 IEEE, 2019, ~
  55. O. Tursun, "An efficient framework for zero-shot sketch-based image retrieval", Pattern Recognit, 126, pp.108528~ (2022)[DOI]
  56. U. Chaudhuri, "BDA-SketRet : Bi-level domain adaptation for zero-shot SBIR", Neurocomputing, 514, pp.245~255 (2022)[DOI]
  58. A. Chaudhuri, "Data-free sketch-based image retrieval", Proc. of 2023 IEEE, 2023, ~
  59. H. Thakur, "Active learning for fine-grained sketch-based image retrieval", Proc. of the 34th British Machine Vision Conference (BMVC2023) (2023)
  60. M. Eitz, "Sketch-based image retrieval : benchmark and bag-of-features descriptors", IEEE Trans. Vis. Comput. Graphics, 17(11), pp.1624~1636 (2011)[DOI]
  61. M. Eitz, "How do humans sketch object?", ACM Trans. Graph, 31(4), pp.1~10 (2012)
  62. H. Zhang, "SketchNet: sketch classification with web images", Proc. of 2016 IEEE, 2016, ~
  63. P. Sangkloy, "The sketchy database : learning to retrieve badly drawn bunnies", ACM Trans. Graph, 35(4), pp.1~12 (2016)[DOI]
  64. D. Ha, "A neural representation of sketch drawings", Proc. of 2018 International Conference on Learning Representations (ICLR 2018) (2018)
  65. Y. Wang, "Deep cascaded cross-modal correlation leaning for fine-grained sketch-based image retrieval", Pattern Recognit, 100(107148) (2020)[DOI]
  66. Q. Yu, "Fine-grained instance-level sketchbased image retrieval", Int. J. Comput. Vis, 129(2), pp.484~500 (2021)[DOI]
  67. L. S. Ferraz Ribeiro, "Scene designer : a unified model for scene search and synthesis from sketch", Proc. of 2021 IEEE, 2021, ~
  68. X. Zhang, "A deformable CNN-based triplet model for finegrained sketch-based image retrieval", Pattern Recognit, 125, pp.108508~ (2022)[DOI]

관련 논문AI추천

이미지 분석을 위한 퓨샷 학습의 최신 연구동향석호식한국전기전자학회 · 전기전자학회논문지2023 · 제27권 제4호효율적인 게임 제작을 위한 스케치 기반 영상 검색 기법민경하, 양희경, 이정준(사)한국컴퓨터게임학회 · 컴퓨터게임및콘텐츠논문지2013 · 제26권 제4호체계적 분석 기법을 이용한 의미기반 이미지검색 분야 고찰에 관한 연구정은경한국비블리아학회 · 한국비블리아학회지2014 · 제25권 제4호고품질 이미지 생성을 위한 StyleGAN 기반 생성 모델 연구 동향고훈준한국인공지능융합기술학회 · 인공지능융합기술연구2025 · 제5권 제4호뉴럴 네트워크를 활용한 지식 정보 검색 연구 현황박혜진, 한요섭문화융복합아카이빙연구소 · 디지털문화아카이브지2022 · 제5권 제1호단일 이미지에 기반을 둔 사람의 포즈 추정에 대한 연구 동향조정찬한국차세대컴퓨팅학회 · 한국차세대컴퓨팅학회 논문지2019 · 제15권 제5호

이 간행물의 다른 논문 (16건)

마이크로서비스아키텍처 기반 경량형 모의실험환경김진우, 김은경, 김현기, 장수영, 최창범, 함규식pp. 123-130부분방전 측정용 비접촉식 HFCT 센서개발 및특성평가에 관한 연구한상보pp. 131-135실험 데이터 기반의 PID제어기를 이용한 쿼드로터의 고도제어이호성, 박지선, 최호림pp. 136-144완전지도 시간적 행동 검출에서역재생 비디오를 이용한 양방향 정보 학습 방법권희원, 조혜정, 조선희, 정찬호pp. 145-149YOLOv5 및 다항 회귀 모델을 활용한사과나무의 착과량 예측 방법정윤주, 곽희진, 전익조, 이철희pp. 150-157
원문 보기