다중 모달리티 의미론적 분할에서의 무작위 마스킹과 가중치 공유 기법
Random masking and weight sharing techniques in multimodal semantic segmentation
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
RGB 영상과 Thermal 영상의 다중 모달리티의 활용은 악천후나 안개, 조명 환경 등으로 가시성이 저하되는 다양한 상황에서 그 가치를 발휘한다. 이전 의미론적 분할 작업에서 RGB 영상에서 추출된 특징을 Thermal 영상의 특징과 결합하는 방식으로 높은 인지 성능을 달성했지만, 모달리티의 특성에 따른 모델 가중치는 고려하지 않고 출력된 특징을 결합했기 때문에 제한적인 성능을 보인다. 본 논문에서는 RGB 영상과 Thermal 영상을 학습하는 과정에서 네트워크가 공통적인 가중치를 갖도록 구성하고, 손실 계산 단계에서 더 많은 정보를 가진 모달리티 특징이 다른 모달리티를 이끌도록 구성하였다. 또한 상호보완적인 마스킹 모듈과 잔차 네트워크를 구성하여 FMB[1] 데이터 세트에서 12.25의 성능 향상을 이뤘다.
Abstract
Utilizing the multi-modality of RGB and thermal images is valuable in various situations where visibility is reduced due to bad weather, fog, or lighting conditions. In previous semantic segmentation work, high recognition performance has been achieved by combining features extracted from RGB images with features from thermal images, but the performance is limited because the output features are combined without considering the model weights according to the characteristics of the modalities. In this paper, we configure the networks to have common weights during the learning process of RGB and Thermal images, and configure the modality features with more information to lead the other modalities in the loss calculation step. We also configure complementary masking modules and residual networks to achieve a performance improvement of 12.25 on the FMB dataset
참고문헌 (22)
- LIU, Jinyuan, "Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation", Proceedings of the IEEE, 2023, 8115~8124
- HA, Qishen, "MFNet : Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes", 2017 IEEE, 2017, 5108~5115[DOI]
- ZHANG, Jiaming, "CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers", IEEE Transactions on Intelligent Transportation Systems (2023)
- DENG, Fuqin, "FEANet : Feature-enhanced attention network for RGB-thermal real-time semantic segmentation", 2021 IEEE, 2021, 4467~4473
- CHEN, Liang-Chieh, "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs", IEEE transactions on pattern analysis and machine intelligence, 40(4), pp.834~848 (2017)
- XIAO, Tete, "Unified perceptual parsing for scene understanding", Proceedings of the European conference on computer vision (ECCV), pp.418~434 (2018)
- XIE, Enze, "SegFormer : Simple and efficient design for semantic segmentation with transformers", Advances in Neural Information Processing Systems, 34, pp.12077~12090 (2021)
- CORDTS, Marius, "The cityscapes dataset for semantic urban scene understanding", Proceedings of the IEEE conference on computer vision and pattern recognition, pp.3213~3223 (2016)
- CAESAR, Holger, "Coco-stuff : Thing and stuff classes in context", Proceedings of the IEEE conference on computer vision and pattern recognition, pp.1209~1218 (2018)
- HE, Kaiming, "Masked autoencoders are scalable vision learners", Proceedings of the IEEE, 2022, 16000~16009
- XIE, Zhenda, "Simmim : A simple framework for masked image modeling", Proceedings of the IEEE, 2022, 9653~9663
- KRIZHEVSKY, Alex, "Imagenet classification with deep convolutional neural networks", Advances in neural information processing systems, 25 (2012)
- HE, Kaiming, "Deep residual learning for image recognition", Proceedings of the IEEE conference on computer vision and pattern recognition, pp.770~778 (2016)
- IOFFE, Sergey, "Batch normalization: Accelerating deep network training by reducing internal covariate shift", International conference on machine learning, pp.448~456 (2015), pmlr