Joan Duran

Recent Publications

1. SR4RS

   A comprehensive overview of satellite image fusion: From classical model-based to cutting-edge deep learning approaches

   Pereira-Sánchez, I., Sans, E., Navarro, J.,  and Duran, J.

   In Super Resolution for Remote Sensing, Springer Nature, 2024

   Abs Bib

   Earth observation satellites usually acquire a high-resolution image with a very limited number of spectral bands along with a lower-resolution image that accurately encodes the spectral responses of objects in the scene. Satellite image fusion, also known as pansharpening or hypersharpening depending on the characteristics of the data, aims to combine the spatial and spectral information into a single high-resolution multispectral or hyperspectral image. The resulting image is then used in a wide variety of remote sensing applications, for which low ground sampling distance and detailed description of the chemical-physical composition of the objects may be required. In this chapter, we review the state of the art of satellite image fusion, emphasizing the crucial role that the modeling of the problem plays in performance and analyzing how deep learning has changed the paradigm. This study enables comprehending the evolution of various approaches and their respective outcomes. We establish a fair comparison process, standardizing a general strategy to train and evaluate fusion methods. Quantitative and qualitative comparisons are conducted on several datasets with distinct resolutions and sensor characteristics. The source code used for the comparison are published freely for non-commercial use.

   @incollection{PereiraSansSR4RS,
     author = {Pereira-S\'anchez, I. and Sans, E. and Navarro, J. and Duran, J.},
     title = {A comprehensive overview of satellite image fusion: From classical model-based to cutting-edge deep learning approaches},
     booktitle = {Super Resolution for Remote Sensing},
     publisher = {Springer Nature},
     year = {2024},
     editor = {Kawulok, M. and Kawulok, J. and Smolka, B. and Emre Celebi, M.},
   }

2. MIGARS

   A simple nonlocal back-projection unfolded network for pansharpening

   Pereira-Sánchez, I., Sans, E., Navarro, J.,  and Duran, J.

   In Proceedings of the International Conference on Machine Intelligence for Geoanalytics and Remote Sensing (MIGARS), Wellington, New Zealand, 2024

   Abs Bib

   Pansharpening is the fusion process that combines the geometry of a high-resolution panchromatic image with the spectral information encoded in a low-resolution multispectral image. We introduce a back-projection method to minimize the reconstruction error between the target image and the output produced by the Brovey pansharpening model. We replace the back-projection kernel with a residual network that incorporates a nonlocal module, exploiting self-similarity and built upon the multi-head attention mechanism. Experimental validation showcases that our method achieves state-of-the-art results.

   @inproceedings{PereiraSansMIGARS2024,
     title = {A simple nonlocal back-projection unfolded network for pansharpening},
     author = {Pereira-S\'anchez, I. and Sans, E. and Navarro, J. and Duran, J.},
     booktitle = {Proceedings of the International Conference on Machine Intelligence for Geoanalytics and Remote Sensing (MIGARS)},
     address = {Wellington, New Zealand},
     year = {2024},
   }

3. VISAPP

   Beyond variational models and self-similarity in super-resolution: Unfolding models and multi-head attention

   Pereira-Sánchez, I., Sans, E., Navarro, J.,  and Duran, J.

   In Proceedings of the International Conference on Computer Vision, Theory and Applications (VISAPP), Rome, Italy, 2024

   Abs Bib

   Classical variational methods for solving image processing problems are more interpretable and flexible than pure deep learning approaches, but their performance is limited by the use of rigid priors. Deep unfolding networks combine the strengths of both by unfolding the steps of the optimization algorithm used to estimate the minimizer of an energy functional into a deep learning framework. In this paper, we propose an unfolding approach to extend a variational model exploiting self-similarity of natural images in the data fidelity term for single-image super-resolution. The proximal, downsampling and upsampling operators are written in terms of a neural network specifically designed for each purpose. Moreover, we include a new multi-head attention module to replace the nonlocal term in the original formulation. A comprehensive evaluation covering a wide range of sampling factors and noise realizations proves the benefits of the proposed unfolding techniques. The model shows to better preserve image geometry while being robust to noise.

   @inproceedings{PereiraSansVISAPP2024,
     title = {Beyond variational models and self-similarity in super-resolution: Unfolding models and multi-head attention},
     author = {Pereira-S\'anchez, I. and Sans, E. and Navarro, J. and Duran, J.},
     booktitle = {Proceedings of the International Conference on Computer Vision, Theory and Applications (VISAPP)},
     address = {Rome, Italy},
     year = {2024},
   }

4. VISAPP

   Combining total variation and nonlocal variational models for low-light image enhancement

   Torres, D., Sbert, C.,  and Duran, J.

   In Proceedings of the International Conference on Computer Vision, Theory and Applications (VISAPP), Rome, Italy, 2024

   Abs Bib

   The rapid expansion of technology has popularized the use of image processing techniques in several fields. However, images captured under low-light conditions impose significant limitations on the performance of these methods. Therefore, improving the quality of these images by discounting the effect of the illumination is crucial. In this paper, we present a low-light image enhancement method based on the Retinex theory. Our approach estimates illumination and reflectance in two steps. First, the illumination is obtained as the minimizer of an energy functional involving total variation regularization, which favours piecewise smooth solutions. Afterwards, the reflectance component is computed as the minimizer of an energy involving contrast-invariant nonlocal regularization and a fidelity term preserving the largest gradients of the input image.

   @inproceedings{TorresSbertVISAPP2024,
     title = {Combining total variation and nonlocal variational models for low-light image enhancement},
     author = {Torres, D. and Sbert, C. and Duran, J.},
     booktitle = {Proceedings of the International Conference on Computer Vision, Theory and Applications (VISAPP)},
     address = {Rome, Italy},
     year = {2024},
   }

5. CVPR

   Deep unfolding for hypersharpening using a high-frequency injection module

   Mifdal, J., Tomás-Cruz, M., Sebastianelli, A., Coll, B.,  and Duran, J.

   In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Earthvision Workshop, Vancouver, Canada, 2023

   Abs Bib HTML Poster

   The fusion of multi-source data with different spatial and spectral resolutions is a crucial task in many remote sensing and computer vision applications. Model-based fusion methods are more interpretable and flexible than pure data-driven learning networks, however, their performance depends greatly on the established fusion model and the hand-crafted prior. In this work, we propose an end-to-end trainable model-based network for hyperspectral and panchromatic image fusion. We introduce an energy functional that takes into account classical observation models and incorporates a high-frequency details injection constraint. The resulting optimization function is solved by a forward-backward splitting algorithm and unfolded into a deep-learning framework that uses two modules trained in parallel to ensure both data observation fitting and constraint compliance. Extensive experiments are conducted on the remote-sensing hyperspectral PRISMA dataset and on the CAVE dataset, proving the superiority of the proposed deep unfolding network both qualitatively and quantitatively.

   @inproceedings{MifdalTomasCVPR2023,
     title = {Deep unfolding for hypersharpening using a high-frequency injection module},
     author = {Mifdal, J. and Tom{\'a}s-Cruz, M. and Sebastianelli, A. and Coll, B. and Duran, J.},
     booktitle = {Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Earthvision Workshop},
     address = {Vancouver, Canada},
     pages = {2105--2114},
     year = {2023},
     doi = {10.1109/CVPRW59228.2023.00204},
   }