ArcGIS Pro

What’s new for deep learning in the Image Analyst extension of ArcGIS Pro 3.1

There have been many deep learning capabilities that were added to the Image Analyst extension for ArcGIS Pro over the last few releases. The emphasis for the ArcGIS Pro 3.1 release was to improve user workflows and to increase productivity. Here are our top three new deep learning features in the 3.1 release:

Want to learn more? Continue reading.

Pixel classification using time-series imagery

In this release, we added support for the Pixel-Set Encoders and Temporal Self-Attention (PSETAE) architecture to train models for time series classification. PSETAE is a transformer based deep learning model, originally developed by Garnot et al for agricultural parcel classification into different crop types using time-series satellite images.

To support this workflow, we have added support for multidimensional inputs and the PSETAE architecture for the following tools:

Example workflow: Crop classification on time-series imagery

We have aerial images for different time periods, and we want to perform crop classification. Our first step is to prepare the data for training, where we will use the Export Training Data for Deep Learning tool. The input time-series imagery can be either be a composite of rasters or a multi-dimensional raster, from the required time periods. To create training labels for a multi-dimensional raster, use the Label objects for Deep Learning tool (available on the Imagery tab within the Image Classification group). Pixels are labelled into different classes, based on the available information. The figure below shows labeling of different crop types. The tool will write out stacked images as training chips. The training chips will have all the information from all the time slices.

Samples of labels from different classes overlayed on time series raster
Samples of labels from different classes overlayed on time series raster

Once we have our training data, we can run the Training Deep Learning tool with the new PSETAE model architecture. Finally, we can use the Classify Pixels using Deep Learning tool to perform the inferencing. The Classify Pixels using Deep Learning tool also supports multi-dimensional datasets, which can be a CRF or a mosaic dataset.

The final output will be a classified raster of the different crop types.

Crop Field Classification on Time Series Data
Crop Field Classification on Time Series Data

Pixel classification on features with image attachments

Features with attachments is an information model that spans across the ArcGIS platform. Many of the Esri apps, such as ArcGIS QuickCapture and ArcGIS Survey 123, use this approach to collect data and photographs. We enhanced the Classify Pixel using Deep Learning tool to allow features classes with image attachments as an input. The output will be a feature class with images, whose pixels are classified or transformed as per the input deep learning model. You can learn more about adding attachments to a feature in this video.

Example: Blurring sensitive information on features with image attachments

Let’s consider that we collect street view images for various purposes, such as asset inspections, landscape design, or traffic management. This information can be collected using car mounted cameras. However, the images may contain human faces and vehicle license plates. For privacy purposes, we want all faces and license plates to be blurred out before allowing this data to be public. We can use  ArcGIS pre-trained models (Face Blurring and License Plate Blurring) to address these privacy concerns.

We will use the GPS locations to create points for each image and attached the images to the corresponding points features. And of course we will make sure to blur the faces and license plates in the image attachments.

Features with image attachments
Features with image attachments

The image above shows an example of a point features with image attachments. We will use this point feature class as an input in the Classify Pixels using Deep Learning tool.  The final output is a feature class with images having faces and license plates blurred out. We can use ModelBuilder or a Python script to streamline all the steps together in the workflow.

This is just one example of how pixel classification using features with image attachments can help your workflows. We are confident that this enhancement can help your workflows too.

Features with image attachments with blurred face and license plates
Features with image attachments with blurred face and license plates

Interactive segmentation for labeling

And finally, another productivity enhancement we like a lot is interactive segmentation for labeling. We like this enhancement because data labeling is one of the most tedious and time-consuming aspects of deep learning workflows. Organizations spends large amounts of resources to get data labeled. In ArcGIS Pro 3.1, we have added a new feature called Auto Detect in our Label Objects for Deep Learning pane. This feature can detect objects and automatically draw the polygons or segments on the target image. You can just select your label and click on the features.

In below animation, we are clicking on the houses and pools, and the tool is creating polygons around the objects automatically.

Auto Detect features
Auto Detect

What do you think about these enhancements? Do you have any use cases that can benefit from these new features? Please share your ideas to improve deep learning workflows in ArcGIS.

About the authors

Pavan Yadav

Pavan Yadav is a product engineer with the ArcGIS Imagery team. He is passionate about GeoAI, more specifically Deep Learning, and applying them to solve GIS, Image Analysis, and Image processing problems.


Simon W

Simon is an Esri product engineer on the Raster team. He works mainly with Pixel Editor, raster visualization, georeferencing, and documentation.

Inline Feedbacks
View all comments

Next Article

Join Hacktoberfest 2023: Celebrate Open Source with Us!

Read this article