Geoprocessing, Resolution, Tolerance, and Hair

Thanks to Ken Hartling from the Geoprocessing team for the story and steps

Over the years, our Geoprocessing team has done many investigations that have come in as apparently gnarly, seemingly inexplicable, or at a minimum vexing performance problems or issues where output seems amiss.

The great thing is that they have already provided the world with a sweet rundown of what you need to know in this help topic called Spatial reference and geoprocessing.  There is also the scintillating companion topic called The properties of a spatial reference.

I’m not here to re-write that which is already written so here is a synopsis.  People often sling around the terms coordinate system and spatial reference interchangeably.  They are not the same thing – a spatial reference is made up of the following:

1. A coordinate system
2. XY resolution and, optionally, M and Z resolution and domain
3. XY tolerances and, optionally, M and Z tolerances

With that preamble out of the way, why is all of this important?  Remember that we started this whole thing by saying that we may run into problems with geoprocessing.  The main point we want to make here is that spatial reference has a lot to do with geoprocessing so it should be on our list of things to check if we ever experience weirdness.

And what about hair?  What is that all about?  Well, in thinking about how to best convey the meaning of what one-tenth of one millimeter actually looks like, I found myself doing some internet research and the most relatable thing I came up with was hair!  The average (as with anything, there is a range – some hair is thinner and some is thicker) human hair is around 0.1 mm thick.  So next time you’re creating data, if you ever find yourself thinking that it might be a good idea to set a finer resolution, scratch your head and remember hair.  Are you really measuring something that needs to be stored more precisely than the thickness of a piece of hair?

How do we know if something is bad?

To know if something is bad, we must know what is good.  A feature class with default Resolution and Tolerance will look like this:

XY Resolution should be 0.0001 meters (1/10 millimeters) or its equivalent in map units. For example, if a feature class is stored in state plane feet, the default precision will be 0.0003281 feet (0.003937 inches). If coordinates are in latitude-longitude, the default resolution is 0.000000001 degrees.

XY Tolerance should be 0.001 meters or its equivalent in map units.  This is 10 times larger than the default resolution value and is recommended in most cases.

If you find that any dataset participating in your geoprocessing workflow is not set to the default resolution and tolerance, here is what to do.

1. Since these settings can only be specified during feature class creation, create a new feature class.

2. During creation, model the new feature class on the old.

3. Use the original feature class’s coordinate system as a template for the new feature class.

4. THIS IS THE MOST IMPORTANT STEP. Click the Reset to Default button for Tolerance,

… and check the box to ‘Accept default resolution and domain extent (recommended)’.

We now have a new empty feature class with the correct schema (all the fields and definitions from the original), the same coordinate system as the original, and most importantly, default resolution, tolerance and domain for the rest of the spatial reference.

6. Load the original data into the new feature class.

From the Catalog Pane in ArcGIS Pro, right-click the new feature class and choose Load Data.

This will launch the Append tool where the data from the original dataset can be loaded into the new feature class.  (HINT: In ArcMap this will launch the Simple Data Loader)

7. Once the data is loaded, run Check Geometry to see if any geometries need to be fixed with Repair Geometry. *The Check Geometry and Repair Geometry tools only accept shapefiles, personal geodatabases, and file geodatabases as input. Other formats are not valid input to these two tools.

8. Run the geoprocessing workflow again with the new feature class that has repaired spatial reference and geometries to see if the problem goes away.

If everything works better and is good in the world, yeah, we’re done!

If the processing issue does not go away, we’ll have to get this to Esri Technical Support to help investigate.

About the author

Kory Kramer

Kory Kramer is a Product Manager for ArcGIS Pro, collaborating across multiple teams to contribute to the direction of product development. He is dedicated to listening to users and finding ways to convert that feedback into a product that continues to add customer value. In his free time, he tries to get outside as much as possible – often on trails, sometimes on a bike. He has recently started composting and is trying to get bats to take up residence in his backyard bat house.

Connect: