Research on visible and NIR images fusion

Horacio Santos Chi

Abstract：Visible and NIR images has different responding on different objects， by fusing visible and NIR images， it can effectively enhance the quality of the images. In this paper the image will be fused by Laplacian–Gaussian pyramid.

1 Introduction

The visibility and color of images can be effect by bad weathers like haze， smoke， fog or rain. Because on those condition will cause image scattering， loss of contrast and detail. Some of the research shows， by using different sensor to capture image of same scenes can enhance the visibility and quality of the image. In recent years， image fusion enhancement technology based on the fusing the visible  images and near infrared images （NIR） images has been widely used. Digital camera can capture the visible（400-700 nm） images and the near-infrared（700-1100 nm） images. NIR can penetrate smog and has a good brightness response to vegetation and clouds [1]. These features of NIR images are very helpful to enhance image visibility.

2Visible and NIR images fusion base on Laplacian–Gaussian pyramid

The Laplacian-Gaussian pyramid fusion algorithm was proposed by Tom Mertens [2]. For the Laplacian pyramid fusion algorithm， the process can be roughly divided into three steps [3]： 1） For the Laplacian pyramid fusion algorithm， the process can be roughly divided into three steps [3]：1） Decomposition of the image pyramid.2） Generate a weight map.3） Image reconstruction. The most important step will be generating the weight map. Fusing the images by different weight will lead a different result. In this case， the weight map will be generated by three weight， the local contrast of image， the saturation of the image and the local entropy of the image. And because visible images have three channel， which is R， G and B， and NIR images have only one channel， its impossible to fuse them directly， and apparently replacing one of the RGB channel with NIR channel is not a good choice. So for the visible and NIR image fusion， converting the visible image from RGB color space to HSV color Space， and then fuse the V（lightness） channel of visible image with NIR channel will be a good way to fuse the images.

After getting the HSV color space images by converting the visible image from RGB color space， it is the time to fuse the visible and NIR image by Laplacian–Gaussian pyramid. An image pyramid model means images will be sort from low resolution to high resolution in a pyramid shape， the resolution of image in this pyramid will decrease layer by layer. By decomposition the image into pyramid model， we can perform a multi-resolution process on the images.

The Gaussian pyramid is constructed from a collection of pyramidal images. The core idea of the algorithm is to process the image that needs to be processed by the Gaussian pyramid algorithm， first downsampling the original image to get the result image by deleting even rows and columns of image， this result will be placed in next layer of pyramid， and repeat this step until the pyramid model is constructed. And in these procedure， it will cause the loss of some high frequency information of image， causing the visibility of image become bad.

In order to make up for this situation， the image of each layer of the Gaussian pyramid can be upsampled by the Laplacian pyramid， and at the same time， the Gaussian convolution is performed to obtain an intermediate result map， and finally obtain a series of images after the subtraction. After fusing all the images in each layer，perform an upsample process to the fuse Pyramid image， and add all the images after the upsampling， we can get a result image with same resolution of original image. And  after replacing the result image with the V channel of visible image and covert it into RGB color space， we can get the Visible-NIR fused image.

3  Conclution

In order to improve the visibility of the visible image and enrich the amount of information contained in the image， by fusing the NIR image and the visible image， the quality of the image can be significantly improved. When the NIR image is fused with the visible image by the Laplace–Gaussian  Pyramid fusion algrithm， it is necessary to select weights to form a weight map to determine the information retained by each pixel. Therefore， after fusion， the respective advantages of both sides of the source image can be retained as much as possible.

4  References

[1] Fredembach C， Susstrunk S. Colouring the near infrared[R]. Portland： Proceedings of the IS&T/SID 16th Color Imaging Conference， 2008

[2] Mertens T， Kautz J， Reeth F V. Exposure fusion[R]. USA： Proceedings of the 15th Pacific Conference on Computer Graphics and Applications， 2007

[3]Ashish V Vanmali，Vikram M Gadre. Visible and NIR image fusion using weight-map-guided Laplacian–Gaussian pyramid for improving scene visibility[J]. Springer India，2017，42（7）.