Multi-Modal Image Synthesis with Attention Conditional GANs: SAR, Optical, and DEM

Abstract

This research addresses challenges in satellite-derived Earth observation data, such as cloud cover, atmospheric condition, seasonality and calibration inconsistencies, by leveraging Synthetic Aperture Radar for cloud penetration and Generative Adversarial Networks (GANs) for cloud removal and image synthesis. The main objective of this research is to develop conditional GAN (cGAN) models capable of synthesizing multispectral images from SAR (Sentinel-1), Optical (Sentinel-2) images and DEM (SRTM 30). The research dataset consists of above three products downloaded for 10 locations distributed across different geographical regions of Nepal from 2022-2023. Using these dataset the cGAN models are trained with different configuration. An initial cGAN model by Bermudez saw improved performance with DEM inclusion and further enhanced by integrating an attention block in the residual block. Thus, attention cGAN (A_ cGAN) model was selected for further analysis. Among, the A_cGAN models the A_ cGAN_SOD model performed the best which uses the dataset (SAR, DEM and optical images) while comparing the MAE, RMSE, SSIM and PSNR values. The image generated from A_cGAN_SOD model for Hetauda was used for LULC classification using random forest algorithm obtaining the overall accuracy of 89.96% which shows that the model output is applicable.