The goal of this article is to share simple coding patterns for effectively performing multiprocessing for geoprocessing. The article will cover relevant considerations and limitations, which are important when attempting to implement multiprocessing.<\/p>\n
1. Processing large numbers of datasets<\/h2>\n
The first example performs a specific operation on a large number of datasets, in a workspace or set of workspaces. In cases where there are large numbers of datasets, taking advantage of multiprocessing can help get the job done faster. The following code demonstrates a multiprocessing module used to define a projection, add a field, and calculate the field for a large list of shapefiles. This Python code will create a pool of processes equal to the number of CPUs or CPU cores available. This pool of processes will then be used to processes the feature classes.<\/p>\n
import os
\nimport re
\nimport multiprocessing
\nimport arcpy<\/p>\n
def update_shapefiles(shapefile):<\/p>\n
# Define the projection to wgs84 \u2014 factory code is 4326.<\/p>\n
arcpy.management.DefineProjection(shapefile, 4326)<\/p>\n
# Add a field named CITY of type TEXT.<\/p>\n
arcpy.management.AddField(shapefile, \u2018CITY\u2019, \u2018TEXT\u2019)<\/p>\n
# Calculate field \u2018CITY\u2019 stripping \u2018_base\u2019 from the shapefile name.<\/p>\n
city_name = shapefile.split(\u2018_base\u2019)[0]
\ncity_name = re.sub(\u2018_\u2019, \u2018 \u2018, city_name)
\narcpy.management.CalculateField(shapefile, \u2018CITY\u2019, \u2018\u201d{0}\u201d\u2018.format(city_name.upper()), \u2018PYTHON\u2019)<\/p>\n
# End update_shapefiles<\/p>\n
def main():<\/p>\n
# Create a pool class and run the jobs\u2013the number of jobs is equal to the number of shapefiles<\/p>\n
workspace = r\u2019C:GISDataUSAusa\u2019
\narcpy.env.workspace = workspace<\/p>\n
fcs = arcpy.ListFeatureClasses(\u2018*\u2019)<\/p>\n
fc_list = [os.path.join(workspace, fc) for fc in fcs]<\/p>\n
pool = multiprocessing.Pool()<\/p>\n
pool.map(update_shapefiles, fc_list)<\/p>\n
# Synchronize the main process with the job processes to ensure proper cleanup.<\/p>\n
pool.close()<\/p>\n
pool.join()<\/p>\n
# End main<\/p>\n
if __name__ == \u2018__main__\u2019:
\nmain()<\/p>\n<\/div>\n
2. Processing a individual dataset with a lot of features and records<\/h2>\n
This second example looks at geoprocessing tools analyzing an individual dataset with a lot of features and records. In this situation, we can benefit from multiprocessing by splitting data into groups to be processed simultaneously. For example, finding identical features may be faster when you split a large feature class into groups, based on spatial extents. The following code uses a pre-defined fishnet of polygons covering the extent of 1 million points (Figure 1).<\/p>\n
![\"\"](\"http:\/\/edn1.esri.com\/blogimages\/gp\/082911_2339_Multiproces1.png\")
<\/p>\n
Figure 1: A fishnet of polygons covering the extent of one million points.<\/i><\/p>\n import multiprocessing def find_identical(oid):<\/p>\n # Create a feature layer for the tile in the fishnet.<\/p>\n tile = arcpy.management.MakeFeatureLayer(r\u2019c:testingtesting.gdbfishnet\u2019, \u2018layer{0}\u2019.format(oid[0]),<\/p>\n \u201c\u201dOID = {0}\u201d”\u201d.format((oid[0])))<\/p>\n # Get the extent of the feature layer and set the extent environment.<\/p>\n tile_row = arcpy.SearchCursor(tile)<\/p>\n geometry = tile_row.next().shape<\/p>\n arcpy.env.extent = geometry.extent<\/p>\n # Execute Find Identical<\/p>\n identical_table = arcpy.management.FindIdentical(r\u2019c:testingtesting.gdbrandom1mil\u2019, r\u2019c:cursortestingidentical{0}.dbf\u2019.format(oid[0]),\u00a0 \u2018Shape\u2019) # End find_identical<\/p>\n def main():<\/p>\n # Create a list of OID\u2019s used to chunk the inputs<\/p>\n fishnet_rows = arcpy.SearchCursor(r\u2019c:testingtesting.gdbfishnet\u2019, \u201d, \u201d, \u2018OID\u2019)<\/p>\n oids = [[row.getValue(‘OID’)] for row in fishnet_rows]<\/p>\n # Create a pool class and run the jobs\u2013the number of jobs is equal to the length of the oids list<\/p>\n pool = multiprocessing.Pool()<\/p>\n result_tables = pool.map(find_identical, oids)<\/p>\n # Merge the all the temporary output tables \u2014 this is optional. Omitting this can increase performance.<\/p>\n arcpy.management.Merge(result_tables, r\u2019C:cursortestingctesting.gdbfind_identical\u2019)<\/p>\n # Synchronize the main process with the job processes to ensure proper cleanup.<\/p>\n pool.close() # End main<\/p>\n if __name__ == \u2018__main__\u2019:<\/p>\n main()<\/p>\n<\/div>\n There are tools that do not require data be split spatially. The Generate Near Table example below, shows the data processed in groups of\u00a0250000 features\u00a0by selecting them based on object ID ranges.<\/p>\n import multiprocessing def generate_near_table(oid_range):<\/p>\n i = oid_range[0]<\/p>\n j = oid_range[1]<\/p>\n lyr = arcpy.management.MakeFeatureLayer(r\u2019c:testingtesting.gdbrandom1mil\u2019, \u2018layer{0}\u2019.format(i),<\/p>\n \u201c\u201d”OID >= {0} AND OID <= {1}\u201d”\u201d.format(i, j))<\/p>\n gn_table = arcpy.analysis.GenerateNearTable(lyr, r\u2019c:testingtesting.gdbrandom10000\u2032,<\/p>\n r\u2019c:testingoutnear{0}.dbf\u2019.format(i)) # End generate_near_table function<\/p>\n def main():<\/p>\n oid_ranges = [[0, 250000], [250001, 500000], [500001, 750000], [750001, 1000001]]<\/p>\n arcpy.env.overwriteOutput = True<\/p>\n # Create a pool class and run the jobs<\/p>\n pool = multiprocessing.Pool()<\/p>\n result_tables = pool.map(generate_near_table, oid_ranges)<\/p>\n # Merge resulting tables is optional. Can add overhead if not required.<\/p>\n arcpy.management.Merge(result_tables, r\u2019c:cursortestingctesting.gdbgenerate_near_table\u2019)<\/p>\n # Synchronize the main process with the job processes to ensure proper cleanup.<\/p>\n pool.close() # End main<\/p>\n if __name__ == \u2018__main__\u2019:<\/p>\n main()<\/p>\n<\/div>\n Here are some important considerations before deciding to use multiprocessing:<\/p>\n The scenario demonstrated in the first example, will not work with feature classes in a file geodatabase because each update must acquire a schema lock on the workspace. A schema lock effectively prevents any other process from simultaneously updating the FGDB. This example will work with shapefiles and ArcSDE geodatabase data.<\/p>\n For each process, there is a start-up cost loading the arcpy library (1-3 seconds). Depending on the complexity and size of the data, this can cause the multiprocessing script to take longer to run than a script without multiprocessing. In many cases, the final step in the multiprocessing workflow is to aggregate all results together, which is an additional cost.<\/p>\n Determining if multiprocessing is appropriate for your workflow can often be a trial and error process. This process can invalidate the gains made using multiprocessing in a one off operation; however, the trial and error process may be very valuable if the final workflow is to be run multiple times, or applied to similar workflows using large data. For example, if you are running the Find Identical tool on a weekly basis, and it is running for hours with your data, multiprocessing may be worth the effort.<\/p>\n Whenever possible, take advantage of the \u201cin_memory\u201d workspace for creating temporary data to improve performance. However, depending on the size of data being created in-memory, it may be necessary to write temporary data to disk. Temporary datasets cannot be created in a file geodatabase because of schema locking. Deleting the in-memory dataset when you are finished can prevent out of memory errors.<\/p>\n These are just a few examples showing how multiprocessing can be used to increase performance and scalability when doing geoprocessing. However, it is important to remember that multiprocessing does not always mean better performance.<\/p>\n
\nimport arcpy<\/p>\n
\nreturn identical_table.getOutput(0)<\/p>\n
\npool.join()<\/p>\n
\nimport arcpy<\/p>\n
\nreturn gn_table.getOutput(0)<\/p>\n
\npool.join()<\/p>\nConsiderations<\/h2>\n
Summary<\/h2>\n
"}],"related_articles":"","card_image":false,"wide_image":false},"yoast_head":"\n