ArcGIS API for Python

How we did it: Integrating ArcGIS and deep learning at UC 2018

At the plenary session of this year’s Esri User Conference, we demonstrated an integration of ArcGIS software with the latest innovations in deep learning to perform detection of swimming pools using aerial imagery. We then went a step further to identify pools in a state of neglect, which will empower health inspectors to help prevent the spread of vector-borne diseases.

Today, we’ll show you how we did it, and how ArcGIS made the process much easier. Our notebooks and script files are published here on GitHub; for complete methodology details, see our post on Medium.

The demo tackled two challenges facing the city of Redlands, California. Let’s take a closer look at them.

Challenge #1: Update assessor data

Tax assessors at local government agencies must often rely on expensive and infrequent surveys, leading to assessment inaccuracies. Swimming pools are an important part of assessment records because they impact the value of the property. Finding pools that are not on the assessment roll (such as those recently constructed) will be valuable to the assessor, and will ultimately mean additional revenue for the community.

Visualizing pools in ArcGIS Pro
Visualizing residential parcels in ArcGIS Pro; parcels assessed with pools are in blue. Note that some parcels with pools are missing, indicating outdated data.

Doing this through GIS and AI would certainly reduce the expensive human labor involved in updating the records through field visits of each property.

Challenge #2: Identify neglected pools

If a swimming pool is neglected, as when the property is foreclosed, it often turns a green color and becomes inviting to mosquitoes – standing water with no inputs or outputs. The sheer volume of properties affected in warmer climates, even as the market rebounded, has made the detection of these risky pools challenging for many organizations.

Public health and mosquito control agencies are tasked with protecting the public from vector-borne diseases, including viruses carried by mosquitoes like West Nile and Chikungunya. These agencies need a simple solution that helps them locate neglected pools from imagery and then use this intelligence to drive field activity and mitigation efforts.

A healthy swimming pool (left) and a possibly neglected pool (right) seen from above.

Creating our training data

There’s a golden rule of deep learning: The more training data you give, the better the results. Deep learning models become powerful and accurate when they are fed huge quantities of data to learn from. To build our model, we needed to teach it what a swimming pool looks like from above, and then train it with a lot of imagery.

When we started, we couldn’t find an openly available labeled dataset for swimming pool detection using aerial imagery, so we labeled around 2,000 swimming pools in cities in southern California. Using ArcGIS Pro made the process faster.

In Pro, we have access to a host of aerial, satellite and drone imagery from Esri and its partners; we used the Esri World Imagery basemap for labeling. Pro includes a simple interface to label data, as well as advanced GIS functionality, including tools for reviewing data to manage its quality. ArcGIS Pro’s geoprocessing tools allowed us to create buffers and bounding boxes around labeled pool locations, and it includes the Export Training Data for Deep Learning tool, which creates the labeled image chips like the ones our model needed.

Export Training Data for Deep Learning tool
Working with the Export Training Data for Deep Learning tool in ArcGIS Pro.

We created a shapefile containing the labeled pool locations using ArcGIS Pro and chipped out 224 px by 224 px images from the imagery using the locations in the shapefile. Our GitHub repository includes a notebook demonstrating this approach.

Choosing our imagery

When training a deep learning model, it’s important to use the most current and spatially accurate imagery. The image resolution at which you perform the training and inference and the imagery bands you use are also critical decision points.

The ArcGIS platform provides access to the Living Atlas of the World, which includes imagery from Landsat, Sentinel and NAIP (National Agriculture Imagery Program) satellites. High-resolution imagery is also available through Esri partners, including 7cm imagery from Nearmap and Vexcel.

NAIP imagery is acquired every two years during the growing seasons throughout the continental U.S. This imagery has a 1-meter resolution and comes in four bands – red, green, blue, and near-infrared. We chose NAIP imagery because it’s free for U.S. counties and states, and covers the whole “lower 48.”

Effectively distinguishing clean (healthy) and green (neglected) pools, on the other hand, requires a closer lens and more recent data. Nearmap and Vexcel imagery is collected much more often and provides a much higher resolution. We chose to use Nearmap imagery for our neglected-pool detection, as it’s included in the Esri World Imagery basemap for the area of Redlands.

Training the model

We used a Single Shot MultiBox Detector (SSD)-inspired architecture with focal loss to train our pool detector. Integration with deep learning libraries like PyTorch and fast.ai made this process fast and efficient.

For the imagery our model would use in training, we created chips from NAIP Color Infrared imagery. We were able to get decent results with around 2,000 chips, but the model made mistakes in detecting all pools. At this point, we recalled the golden rule!

We did heavy data augmentation by taking 50 random “jitters” of images around each pool location. Using this technique, we were able to convert those 2,000 images to 100,000 images. Upon training the complete model again, our validation tests were much more satisfactory.

Training the model
A result slide from the training stage of our model. Note the use of the near-infrared band in the NAIP Color Infrared band combination.

Once our model was fully trained and giving us good results, we tested it on a larger area of the NAIP imagery. We split the larger image into smaller chips, as the model requires. All these chips were simultaneously passed as a batch to the model, and its pool-detection predictions were gathered, combined and visualized in ArcGIS.

To run our model on the entire city, we used the capabilities of ArcGIS API for Python. We took the extent of the city of Redlands and exported NAIP images from that area. We then used the simple pipeline described above to collect predictions on all chips within each exported image, and converted them into a feature layer by transforming from image to geographic coordinates.

 

Identifying parcels with unassessed pools

With our pool-detection model ready, we applied it to solve our first challenge: identifying pools in Redlands that are not listed on their parcel’s tax assessment. The Join Features tool in ArcGIS Online helped us here.

We began with a feature layer of all Redlands parcels. The attribute table of that layer includes the pool_ field, which indicates whether the county assessor includes a swimming pool for a given parcel. Next, we added a feature layer with the results of our pool-detection model, and used Join Features to add the relevant parcel fields to that layer.

We then created a third feature layer with each parcel for which the pool_ field was empty, but within which our model had detected a pool.

Pool layers view
Zooming in on our finished layers. The red parcels have a pool detected, but are not assessed as having one by the county.

This new layer identified nearly 600 new pools that were not marked correctly in the county database. ArcGIS Online then made it easy to visualize our results. You can explore this web map containing our three feature layers:

 

 

Clean or green?

Now that our model could accurately identify swimming pools from aerial imagery, we could tackle our second challenge – to label them as clean or green (neglected pools, sometimes also referred to as “zombie pools”). Green pools often contain algae and can be breeding grounds for mosquitoes and other insects. Mosquito control agencies need a simple solution that helps them locate such pools and drive field activity and mitigation efforts.

Clean and green pools

We trained our model using recent high-resolution Nearmap imagery (7 cm resolution) and a large data set of manually labeled clean or green pools. As before, we took multiple “jittered” images of each green pool to increase the number of training images the network could study.

Once the model was ready to distinguish clean and green pools, we ran the inference pipeline on our Redlands city imagery. Take a look at this web map, which shows zombie pools potentially at risk of being mosquito breeding grounds!

We are seeing excellent results as we integrate AI techniques into the ArcGIS platform. This includes machine learning and deep learning algorithms that can do automated object detection, classification and extraction ... delivering a highly accurate model that can facilitate deeper analysis and predict precise outcomes, at minimal cost.

Jay Theodore Chief Technology Officer, ArcGIS Enterprise

The benefits of ArcGIS integration

A primary goal of this project was to apply the latest research in deep learning and use it to solve real-world problems, be it to update county records or to assist mosquito abatement drives. The ArcGIS platform includes a host of capabilities – online mapping, big data analysis, collaboration, and field mobility – to help achieve these goals.

Once we had obtained the locations of the detected swimming pools, it was easy to use the analysis tools in ArcGIS Online to identify parcels that were not being accessed correctly. The Spatial DataFrame in ArcGIS API for Python enabled us to generate Pythonic reports of such properties, and to create information products such as GIS layers to be visualized in web maps or used for further analysis.

The API’s map widget for Jupyter notebooks allowed us to visualize detected pools and residential parcels, and provided renderers and symbology to make it easy to understand the generated maps. The maps could then be saved as web maps and shared with collaborators.

Esri has also recently introduced (as beta) an Image Visit configurable app template that lets image analysts visually inspect the results of an object detection workflow and categorize each as either a correct or an incorrect detection. A live demo of the configured web app is here. This information could then also be fed into better training or to filter the results and prioritize field activities.

Image Visit app
The Image Visit app enables visual inspection of the neglected pools detected by our deep learning model.

We could then use the field mobility capabilities of the ArcGIS platform. Workforce for ArcGIS allows for creation of assignments for mobile workers, such as inspectors in mosquito control agencies. We used the recently introduced apps module in ArcGIS API for Python to automate creation of Workforce assignments for mosquito-control field workers based on the results of the neglected pool detection analysis.

Workforce inspection assignments
Viewing pool inspection assignments in Workforce for ArcGIS. At 10.6.1, Workforce is now available as a web app.

We have only scratched the surface of what’s possible. This is just one application of GIS and AI integration – there are countless others waiting to be powered by these amazing technologies. Stay tuned for more exciting stuff coming out soon!

This post was written with Divyansh Jha, a deep learning intern in our New Delhi office.

About

Development Lead - ArcGIS API for Python. Applying deep learning to the Science of Where @Esri. https://twitter.com/geonumist

Connect:

Next Article

What’s new in ArcGIS Enterprise 10.6.1: Sync collaboration workspaces on demand

Read this article