ArcGIS API for Python

SAR to RGB Translation using CycleGAN

This blog will provide an overview of Synthetic-Aperture Radar (SAR) to RGB image translation using the recently implemented CycleGAN model in the ArcGIS API for Python.

Motivation

Consider a scenario in which a cloudy day is preventing the use of optical imagery for earth observation. Synthetic-aperture Radar (SAR) which is an active data collection type provides an alternative method of image capture that can penetrate clouds and produce the desired ground imagery.

SAR vs Optical image on a cloudy day

While this application of SAR is obviously useful, it is a complex technology with its own hurdles for those unfamiliar with it. Fortunately, deep learning image translation models allow users to convert SAR images to a more easily understandable optical RGB image.

One such model is CycleGAN, which has recently been added to the arcgis.learn module of ArcGIS API for Python. The rest of this blog will go through the steps showing how the model can be used.

Data and preprocessing

The sample data we will be using is a single band (HH) Capella Space’s simulated SAR imagery which we received in tiff format and optical RGB imagery for Rotterdam in the Netherlands. We have converted the single band SAR imagery to 8 bit unsigned, 3 bands raster using the Extract Bands raster function in ArcGIS Pro, which will allow us to export 3 band JPEG images for data preparation.

Exporting training samples

Deep learning models need training data to learn from, so we will use the Export Training Data for Deep Learning tool in ArcGIS Pro to export appropriate training samples from our data. ArcGIS Pro has recently added support for exporting data in the Export Tiles format, which we will be using for this task. This newly added format allows you to export image chips of a defined size without requiring any labels. With this process, we exported 3087 chips, of which we used approximately 90% (2773) images for training and the remaining 308 images for validating our model.

Exported chips using Export Tiles format

Training the SAR to RGB image translation model

After exporting the training data, we used ArcGIS Notebooks and the arcgis.learn module in the Python API to train the model.

Data preparation

Before we can begin to train our model, we first need to prepare our data. To do this, we used the prepare_data() function available in the API and passed the dataset_type parameter as “CycleGAN”. The prepare_data() function prepares a data object from the training data that we exported in the previous step. This data object consists of training and validation data sets with the specified transformations, chip size, batch size, split percentage, etc.

Data Preparation

Model Training

After we prepared the data, we then began the process of training our model using the fit() method in the API.

Model Fitting

From the statistics in the figure above, we can see that our validation loss continues to decrease with each epoch of training. The resulting validation loss of our initial 25 epochs still left room for improvement, so we trained the model for an additional 25 epochs.

Visualize the results

Next, we will validate the model by visualizing a few samples from the validation data set we prepared by simply calling show_results() using the API.

Results achieved after training the model for 50 epochs
Optical imagery vs translated imagery using the model we trained

After training for 50 epochs, the results in the screenshot above indicate that our model was trained well and can realistically convert SAR to RGB images, as well as RGB images to SAR.

Inferencing

Once we were satisfied with the results of our trained model, we performed inferencing on a larger scale to convert SAR imagery to RGB. We did this by using the Classify Pixels using Deep Learning tool available in ArcGIS Pro.

Classify pixels using deep learning tool in ArcGIS Pro

The resulting inferenced imagery is presented in the figure below. You can observe that the SAR imagery is now more interpretable by humans after being translated to optical imagery using the model.

SAR to RGB image translated using the process

Conclusion

In this post, we have seen a practical application of using generative deep learning to convert Synthetic-aperture Radar (SAR) imagery to optical RGB imagery. This is made possible through the image-to-image translation models like CycleGAN in the arcgis.learn module of ArcGIS API for Python.

Earth observation is an important, yet challenging task, especially on cloudy days. SAR to RGB image translation using models like CycleGAN can be a great tool to overcome the limitations of optical imagery. The exercise shows how generative deep learning models can help us reap the benefits of SAR imagery even on cloudy days.

Acknowledgment

We wish to acknowledge Capella Space for making the SAR imagery available for this study. Capella has recently unveiled the world’s highest-resolution commercial SAR imagery that allows us to monitor our planet in all-weather and in all-light conditions.

About the authors

Product Engineer at Esri R&D Center in New Delhi. Believer in the fact that Geospatial technology together with AI can solve the world's biggest problems.

Connect:

Director of Esri R&D Center, New Delhi & development lead of ArcGIS AI technologies and ArcGIS API for Python. Applying deep learning to the Science of Where!

Connect:

Leave a Reply

Please Login to comment

Next Article

What’s new in ArcGIS Velocity (April 2021)

Read this article