End-to-End Learning for Simultaneously Generating Decision Map and Multi-Focus Image Fusion Result

The general aim of multi-focus image fusion is to gather focused regions of different images to generate a unique all-in-focus fused image. Deep learning based methods become the mainstream of image fusion by virtue of its powerful feature representation ability. However, most of the existing deep learning structures failed to balance fusion quality and end-to-end implementation convenience. End-to-end decoder design often leads to unrealistic result because of its non-linear mapping mechanism. On the other hand, generating an intermediate decision map achieves better quality for the fused image, but relies on the rectification with empirical post-processing parameter choices. In this work, to handle the requirements of both output image quality and comprehensive simplicity of structure implementation, we propose a cascade network to simultaneously generate decision map and fused result with an end-to-end training procedure. It avoids the dependence on empirical post-processing methods in the inference stage. To improve the fusion quality, we introduce a gradient aware loss function to preserve gradient information in output fused image. In addition, we design a decision calibration strategy to decrease the time consumption in the application of multiple images fusion. Extensive experiments are conducted to compare with 19 different state-of-the-art multi-focus image fusion structures with 6 assessment metrics. The results prove that our designed structure can generally ameliorate the output fused image quality, while implementation efficiency increases over 30% for multiple images fusion.

We provide the training and testing method of GACN in this branch.

• Pytorch = 1.2.0
• Python = 3.6
• torchvision = 0.5.0
• numpy = 1.17.0
• opencv-python
• scikit-image
• pillow
• matplotlib
• jupyter notebook

# Clone our code
git clone https://github.com/Keep-Passion/GACN.git
cd GACN

# Replicate our image method on fusing multi-focus images
python main.py

# Train GACN 
python train_net.py

If you use it successfully for your research please be so kind to cite the paper.

Ma, B., Yin, X., Di Wu., et al. End-to-end learning for simultaneously generating decision map and multi-focus image fusion result. Neurocomputing, 2022, 470, 204-216.

The authors acknowledge the financial support from the National Key Research and Development Program of China (No. 2016YFB0700500). Besides, the dataset of multiple image fusion is driven from the website provided by Zhuhai Boming Vision Technology Co., Ltd.

Our previous work SESF-Fuse can be found at the paper and the code.