ArcGIS Pro

Dev Summit 2020: Use deep learning tools with full motion video

Many researchers use ArcGIS data science and deep learning tools to solve geospatial problems.

In this demo, Shannon shows how drone footage is used with predictive modeling tools in ArcGIS to conduct an automated and repeatable catfish survey.

What was once a manual and tedious process to detect and identify objects such as individual fish can now be automated to save time and effort.


Using ArcGIS Notebooks, Shannon demonstrates how to train a catfish detector model using a training set derived from ArcGIS Pro’s tools for full motion video.

The trained model effectively detects objects in an entire video and creates a .csv file containing the location and size of the detected objects in each frame of the video.


At a high level, Shannon covers the following workflow:

  1. Create training data
    • Use the Motion Imagery toolset to extract .jpg files from multiplexed videos.
    • Export selected frames as georeferenced images
    • Count, measure, label, and map each object within georeferenced images and store results in a feature class
  2. Train a model
  3. Conduct subsequent surveys

Background: One researcher’s geospatial problem and solution

While Blue catfish (Ictalurus furcatus)  are native to the Mississippi River drainage, they’ve been introduced over the last several decades to a multitude of other areas to encourage sport fishing.

Among reasons they’re popular for sport fishing – they’re uncommonly big (they can reach 50 pounds with regularity) and some say they’re outrageously scrumptious. 

While in some circles they may be desirablein others they’re a nuisance.

Characteristics such as their adaptability, sheer size, longevity, and tendency to rapidly overpopulate have contributed to their classification as a problematic and invasive species.

Perhaps most problematic is their undiscerning palate, which has in turn contributed to declining populations of Blue crabs (Callinectes sapidus) and other species.

This, of course, is concerning to conservationists and to commercial fisheries who have historically relied on their commercial viability. 

To effectively study the blue catfish population, researchers use several methods.

Of them, electrofishing uses a non-invasive way to apply low voltage frequency to coax catfish from the river bottoms to the surface.

Once fish are visible at the surface, researchers can sample and quantify their population characteristics. 

William Shuart from Virginia Commonwealth University Life Sciences has done such research. In his work, William used a drone to capture electrofishing in progress.

He used ArcGIS Pro to visualize, analyze, and georeference several frames of collected footageWithin each targeted frame, he identified and classified objects, stored attributes in a feature class as training data, and exported results using deep learning tools in ArcGIS. 

From there, he used training data as input to run a predictive catfish detector model.

After validating the model’s results and efficacyit could then be used with drone footage from subsequent electrofishing events. The results of each subsequent study could then be shared with key stakeholders, such as local anglers and conservationists concerned about species loss. 

Learn more

Introduction to Full Motion Video

Working with Full Motion Video in ArcGIS web course 

About the authors

Sara is a lead product engineer on the ArcGIS Enterprise team. She enjoys spending time at the lake, non-fiction, and cooking.

Director of Esri R&D Center, New Delhi & development lead of ArcGIS AI technologies and ArcGIS API for Python. Applying deep learning to the Science of Where!

Notify of
1 Comment
Inline Feedbacks
View all comments

Next Article

Accessibility essentials for GIS and mapping

Read this article