위치 정보 인코딩 기반 ISP 신경망 성능 개선
Enhancing A Neural-Network-based I SP Model through Positional Encoding
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
영상 신호 프로세서(Image Signal Processor, ISP)는 카메라 센서로부터 획득된 R AW 영상을 사람의 눈에 보기 좋은 sRGB 영상으로 변환한다. R AW 영상은 sRGB 영상에 비해 영상 처리에 도움이 되는 정보를 가지고 있지만 상대적으로 큰 용량으로 인해 주로 sRGB 영상만 저장되고 사용된다. 또한, 실제 카메라의 ISP 과정이 공개되어 있지 않아 그 역과정을 모사하는 것은 매우 어렵다. 이에 sRGB와 R AW 영상의 상호 변환을 위한 카메라 ISP 모델링 연구가 활발히 진행되고 있으며, 최근 기존의 단순한 ISP 신경망 구조를 고도화하고 실제 카메라 ISP의 동작과 유사하게 카메라 파라미터(노출 시간, 감도, 조리개 크기, 초점 거리)를 직접 반영하는 ParamISP[1] 모델이 제안되었다. 하지만 ParamISP[1]를 포함한 기존의 연구는 카메라 ISP를 모델링함에 있어 렌즈로 인해 발생하는 렌즈 쉐이딩(Lens Shading), 광학 수차(Optical Aberration), 렌즈 왜곡(Lens Distortion) 등을 고려하지 않아 복원 성능에 한계가 있다. 본 연구는 ISP 신경망이 렌즈로 인해 발생하는 열화를 보다 잘 다룰 수 있도록 위치 정보 인코딩(Positional Encoding)을 도입한다. 제안하는 위치 정보 인코딩 기법은 영상을 분할하여 패치(Patch) 단위로 학습하는 카메라 ISP 신경망에 적합하며 기존 모델에 비해 영상의 공간적 맥락을 반영할 수 있어 더욱 정교한 영상 복원을 가능하게 한다.
Abstract
The Image Signal Processor (ISP) converts RAW images captured by the camera sensor into user-preferred sRGB images. While RAW images contain more meaningful information for image processing than sRGB images, RAW images are rarely shared due to their large sizes. Moreover, the actual ISP process of a camera is not disclosed, making it difficult to model the inverse process. Consequently, research on learning the conversion between sRGB and RAW has been conducted. Recently, the ParamISP[1] model, which directly incorporates camera parameters (exposure time, sensitivity, aperture size, and focal length) to mimic the operations of a real camera ISP, has been proposed by advancing the simple network structures. However, existing studies, including ParamISP[1], have limitations in modeling the camera ISP as they do not consider the degradation caused by lens shading, optical aberration, and lens distortion, which limits the restoration performance. This study introduces Positional Encoding to enable the camera ISP neural network to better handle degradations caused by lens. The proposed positional encoding method is suitable for camera ISP neural networks that learn by dividing the image into patches. By reflecting the spatial context of the image, it allows for more precise image restoration compared to existing models.
참고문헌 (15)
- Woohyeok Kim, "ParamISP: Learned Forward and Inverse ISPs using Camera Parameters", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
- Mildenhall, B., "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" (2020), Proceedings of the European Conference on Computer Vision (ECCV)
- Duc-Tien Dang-Nguyen, "RAISE: A raw images dataset for dig- ital image forensics", Proceedings of the 6th ACM multi- media systems conference (MMSys), pp.6~ (2015)
- Abdelrahman Abdelhamed, "A high-quality denoising dataset for smartphone cameras", Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR) (2018)
- Ben Mildenhall, "Burst denoising with kernel prediction networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
- Eli Schwartz, "DeepISP:Toward learning an end-to-end image processing pipeline", IEEE Transactions on Image Processing (TIP), 28(2), pp.912~923 (2018)
- Chen Chen, "Learning to see in the dark", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
- Yu-Lun Liu, "Single-image hdr reconstruction by learning to reverse the camera pipeline", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
- Mahmoud Afifi, "CIE XYZ Net: Unprocessing images for low-level computer vision tasks", IEEE Transactions on Pattern Analysis and Ma- chine Intelligence (TPAMI), 44(9), pp.4688~4700 (2021)
- Tim Brooks, "Unprocessing im- ages for learned raw denoising", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
- Marcos V. Conde, "Model-based image signal processors via learnable dictionaries", Pro- ceedings of the AAAI Conference on Artificial Intelligence (AAAI), 36(1), pp.481~489 (2022)
- Yazhou Xing, "Invertible image signal processing", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
- Syed Waqas Zamir, "Cycleisp: Real image restoration via improved data synthesis", Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR) (2020)
- Nimrod Shabtay, "PIP: Positional Encoding Image Prior", ~
- Qiaole Dong, "Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding", Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR) (2022)