Image Fusion for Misaligned Visual and Thermal Images

Abstract

The main challenge of a Wilderness Search and Rescue (WiSAR) mission is tocover a large area within a reasonable and short amount of time effectively. The recent development in the area of sensor-equipped Unmanned Aerial Vehicles (UAV) and object detection algorithms have pushed the idea of using UAVs equipped with appropriate sensors for the surveillance of the area and finding humans in an efficient and time-saving way. However, visual and thermal cameras used for surveillance can produce images with misaligned features due to varying viewpoints and sensor characteristics, leading to difficulties in accurate detection. The present work discusses a novel deep learning approach using a Generative Adversarial Network (GAN) with two Discriminators and an attention mechanism used in the Generator, which enhances the ability of the network to capture more complex features in the input images. The proposed solution addresses the challenges of diverse terrain and weather conditions by fusing visual and thermal images to produce a single composite image with complementary information. To address the misalignment issue, we propose a novel attention-based Generator that selectively extracts features from the visual and thermal images based on their similarities, while maintaining their unique characteristics. The results obtained further motivate the idea of developing an end-to-end pipeline for human detection in a wilderness environment using both, visual and thermal images as inputs.