Analytics

Parallel Geoprocessing in ArcGIS Pro

ArcGIS Pro is a powerful multithreaded application with unparalleled (no pun intended) spatial analysis and geoprocessing capabilities. Over the last year, including the 2.0 and 2.1 Pro releases, nearly 300 new geoprocessing tools were added which solve new spatial problems, make your work easier, and enable processing of new data types. Some of the new tools include:

Keep checking What’s New in ArcGIS Pro in the coming weeks to see the full list of new analysis and geoprocessing capabilities added to Pro 2.1 and previous releases.

While the quantity of new functionality is undeniably impressive, you might be wondering “What is ArcGIS Pro doing to make my analysis faster?”

Each release a number of tools are enhanced to provide better performance. The Calculate Field tool was enhanced in Pro 2.1 to support Arcade expressions, which are considerably faster compared to calculations in ArcMap. For example, a date-time calculation on 20,000 features in ArcMap took nearly one minute, while Pro with Arcade takes less than 3 seconds.

Calculate Field with Arcade
Calculate Field in ArcGIS Pro 2.1 using Arcade

 

 

♫ Shave and a haircut, 64-bits! ♫

As discussed in this FAQ, geoprocessing in ArcGIS Pro is 64-bit and occurs on a dedicated geoprocessing thread. For most geoprocessing tools performance is roughly equivalent between ArcGIS Pro and ArcMap*, while many tools have been improved and are faster. *ArcMap comparisons use the 64-bit background geoprocessing setup

64-bit geoprocessing does not automatically make tools faster, only able to use more memory for data processing. 64-bit geoprocessing is more robust, results will be more accurate, and processes that previously hung, crashed, or ran out of memory may be able to complete successfully. Some ArcGIS Pro users on twitter recently shared their success with running a large geoprocessing job.

 

 

Listen to this yarn about multiple threads

Likewise, multithreading does not make geoprocessing faster. The multithreading in ArcGIS Pro makes it possible for the geoprocessing thread to run independently from other threads. This means that when a tool is running you can do other things in Pro such as add new data to a map, navigate and make selections in the map, change a layer’s symbology or labels, create and modify a layout, georeference an image, and even find and fill in another geoprocessing tool.

While multithreading is important and will make your work in Pro more productive, many people incorrectly assume it is a performance enhancing drug that will make every tool run in world record time.

 

 

Meanwhile, in a parallel universe…

One technique geoprocessing uses to improve performance is parallel processing. Parallel processing is a computing technique which splits up a big job into many smaller jobs and allows multiple CPUs, cores, or processes to work on the big job at the same time, often resulting in faster processing time. More and more geoprocessing tools leverage parallel processing in each new Pro release. In ArcGIS Pro 2.1, 70 geoprocessing tools can run using parallel processing to split a job across multiple cores or processes on your desktop machine. This is more than twice the number of tools that use parallel processing in ArcMap 10.6.

Parallel processing in ArcGIS Pro is controlled by the Parallel Processing Factor. Some tools will use parallel processing by default, while others require you to set the parallel processing factor to turn it on. Refer to each tool’s help documentation to determine how to control parallel processing for the selected tool.

While there are many geoprocessing tools that do not support parallel processing, we’ve worked on some high priority analysis and data crunching tools to ensure that these processes can leverage modern hardware and have the best possible performance with improved scalability.

The following geoprocessing tools support parallel processing in ArcGIS Pro 2.1.

 

Analysis Tools


3D Analyst Tools


Conversion Tools


Spatial Statistics and Space Time Pattern Mining


Spatial Analyst Tools


Geostatistical Analyst Tools


Data Management Tools (Raster and Mosaic Dataset tools)


Image Analyst Tools


GeoAnalytics Tools

In addition to the ArcGIS Desktop tools listed above, ArcGIS Pro can also run tools on ArcGIS Enterprise servers. The ArcGIS GeoAnalytics Server provides a number of tools that perform feature and big data analysis using distributed computing on one or more machines. You can run and use these tools in Pro just like any other geoprocessing tools, using either your local or enterprise data sources as input, as well as big data file shares and feature layers in your portal. The GeoAnalytics tools are available in Pro if you are connected to an ArcGIS Enterprise that has a GeoAnalytics server, and your user account has sufficient privileges.


Example

I was recently running a Dissolve operation on 20,000 overlapping polygons. While this isn’t a huge process, it still took several minutes to complete. I was wishing it could run faster by splitting the operation up and using all 4 cores on my desktop machine. While the Dissolve tool does not support parallel processing, the Pairwise Dissolve tool does. Using the Pairwise Dissolve tool with it’s default Parallel Processing Factor setting (full parallel processing–the tool distributes the work to all machine cores), the same dissolve process took less than 3 seconds.

pairwise dissolve
Pairwise Dissolve completed in under 3 seconds

Now that’s unparalleled performance!

About the author

Drew Flater is a group lead product engineer at Esri, working on the spatial analysis and geoprocessing capabilities of ArcGIS.

0 Comments
Inline Feedbacks
View all comments

Next Article

ArcGIS Monitor: Analysis Elements for Enterprise portal content

Read this article