{"id":182521,"date":"2012-09-26T17:59:32","date_gmt":"2012-09-26T17:59:32","guid":{"rendered":"http:\/\/www.esri.com\/arcgis-blog\/?post_type=blog&p=182521"},"modified":"2018-12-18T11:22:35","modified_gmt":"2018-12-18T19:22:35","slug":"multiprocessing-with-arcgis-approaches-and-considerations-part-1","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1","title":{"rendered":"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)"},"author":5121,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_searchwp_excluded":""},"categories":[23341],"tags":[31181,24321,24341],"industry":[],"product":[36991],"class_list":["post-182521","blog","type-blog","status-publish","format-standard","hentry","category-analytics","tag-arcpy","tag-geoprocessing","tag-python","product-arcgis-desktop"],"acf":{"short_description":"The multiprocessing Python module provides functionality for distributing work between multiple processes on a given machine.","flexible_content":[{"acf_fc_layout":"content","content":"

The\u00a0multiprocessing<\/em>\u00a0Python module provides functionality for distributing work between multiple processes on a given machine, taking advantage of multiple CPU cores and larger amounts of available system memory. When analyzing or working with large amounts of data in ArcGIS, there are scenarios where multiprocessing can improve performance and scalability. However, there are many cases where multiprocessing can negatively affect performance, and even some instances where it should not be used.
\nThere are two approaches to using multiprocessing for improving performance or scalability:<\/p>\n

    \n
  1. Processing many individual datasets.<\/li>\n
  2. Processing datasets with many features.<\/li>\n<\/ol>\n

    The goal of this article is to share simple coding patterns for effectively performing multiprocessing for geoprocessing. The article will cover relevant considerations and limitations, which are important when attempting to implement this approach.<\/p>\n

    1. Processing large numbers of datasets<\/strong><\/h2>\n

    The first example covers performing a specific operation on a large number of datasets, in a workspace or set of workspaces. In cases where there are large numbers of datasets, taking advantage of multiprocessing can help get the job done faster. The following code demonstrates a multiprocessing module used to define a projection, add a field, and calculate the field for a large list of shapefiles. This Python code is a simple pattern, which will create a pool of processes equal to the number of CPUs or CPU cores available. This pool of processes will then be used to processes the feature classes.<\/p>\n

    \n
    \n\n\n\n
    \n
    1<\/div>\n
    2<\/div>\n
    3<\/div>\n
    4<\/div>\n
    5<\/div>\n
    6<\/div>\n
    7<\/div>\n
    8<\/div>\n
    9<\/div>\n
    10<\/div>\n
    11<\/div>\n
    12<\/div>\n
    13<\/div>\n
    14<\/div>\n
    15<\/div>\n
    16<\/div>\n
    17<\/div>\n
    18<\/div>\n
    19<\/div>\n
    20<\/div>\n
    21<\/div>\n
    22<\/div>\n
    23<\/div>\n
    24<\/div>\n
    25<\/div>\n
    26<\/div>\n
    27<\/div>\n
    28<\/div>\n
    29<\/div>\n
    30<\/div>\n
    31<\/div>\n
    32<\/div>\n
    33<\/div>\n
    34<\/div>\n
    35<\/div>\n
    36<\/div>\n
    37<\/div>\n
    38<\/div>\n
    39<\/div>\n
    40<\/div>\n<\/td>\n
    		\n
    \n
    import<\/code> os<\/code><\/div>\n
    import<\/code> re<\/code><\/div>\n
    import<\/code> multiprocessing<\/code><\/div>\n
    import<\/code> arcpy<\/code><\/div>\n
    <\/div>\n
    def<\/code> update_shapefiles(shapefile):<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>'''Worker function'''<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Define the projection to wgs84 -- factory code is 4326.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>arcpy.management.DefineProjection(shapefile, <\/code>4326<\/code>)<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Add a field named CITY of type TEXT.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>arcpy.management.AddField(shapefile, <\/code>'CITY'<\/code>, <\/code>'TEXT'<\/code>)<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Calculate field 'CITY' stripping '_base' from<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># the shapefile name.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>city_name <\/code>=<\/code> shapefile.split(<\/code>'_base'<\/code>)[<\/code>0<\/code>]<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>city_name <\/code>=<\/code> re.sub(<\/code>'_'<\/code>, <\/code>' '<\/code>, city_name)<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>arcpy.management.CalculateField(shapefile, <\/code>'CITY'<\/code>,<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'\"{0}\"'<\/code>.<\/code>format<\/code>(city_name.upper()), <\/code>'PYTHON'<\/code>)<\/code><\/div>\n
    <\/div>\n
    # End update_shapefiles<\/code><\/div>\n
    def<\/code> main():<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>''' Create a pool class and run the jobs.'''<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># The number of jobs is equal to the number of shapefiles<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>workspace <\/code>=<\/code> 'C:\/GISData\/USA\/usa'<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>arcpy.env.workspace <\/code>=<\/code> workspace<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>fcs <\/code>=<\/code> arcpy.ListFeatureClasses(<\/code>'*'<\/code>)<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>fc_list <\/code>=<\/code> [os.path.join(workspace, fc) <\/code>for<\/code> fc <\/code>in<\/code> fcs]<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool <\/code>=<\/code> multiprocessing.Pool()<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool.<\/code>map<\/code>(update_shapefiles, fc_list)<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Synchronize the main process with the job processes to<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># ensure proper cleanup.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool.close()<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool.join()<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># End main<\/code><\/div>\n
    <\/div>\n
    if<\/code> __name__ <\/code>=<\/code>=<\/code> '__main__'<\/code>:<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>main()<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

    2. Processing a individual dataset with a lot of features and records<\/strong><\/h2>\n

    This second example looks at Geoprocessing tools analyzing on an individual dataset with a lot of features and records. In this situation we can benefit from multiprocessing, by splitting data into groups to be processed simultaneously.<\/p>\n

    \n
    \n
    \n
    \"\"<\/a><\/dt>\n
    Figure 1: A fishnet of polygons covering the extent of one million points<\/dd>\n<\/dl>\n<\/div>\n<\/div>\n

    For example, finding identical features may be faster when you split a large feature class into groups, based on spatial extents.
    \nThe following code uses a pre-defined fishnet of polygons covering the extent of 1 million points (Figure 1).<\/p>\n

    \n
    \n\n\n\n
    \n
    1<\/div>\n
    2<\/div>\n
    3<\/div>\n
    4<\/div>\n
    5<\/div>\n
    6<\/div>\n
    7<\/div>\n
    8<\/div>\n
    9<\/div>\n
    10<\/div>\n
    11<\/div>\n
    12<\/div>\n
    13<\/div>\n
    14<\/div>\n
    15<\/div>\n
    16<\/div>\n
    17<\/div>\n
    18<\/div>\n
    19<\/div>\n
    20<\/div>\n
    21<\/div>\n
    22<\/div>\n
    23<\/div>\n
    24<\/div>\n
    25<\/div>\n
    26<\/div>\n
    27<\/div>\n
    28<\/div>\n
    29<\/div>\n
    30<\/div>\n
    31<\/div>\n
    32<\/div>\n
    33<\/div>\n
    34<\/div>\n
    35<\/div>\n
    36<\/div>\n
    37<\/div>\n
    38<\/div>\n
    39<\/div>\n
    40<\/div>\n
    41<\/div>\n
    42<\/div>\n
    43<\/div>\n
    44<\/div>\n
    45<\/div>\n
    46<\/div>\n
    47<\/div>\n
    48<\/div>\n
    49<\/div>\n
    50<\/div>\n
    51<\/div>\n
    52<\/div>\n
    53<\/div>\n
    54<\/div>\n
    55<\/div>\n<\/td>\n
    		\n
    \n
    import<\/code> multiprocessing<\/code><\/div>\n
    import<\/code> numpy<\/code><\/div>\n
    import<\/code> arcpy<\/code><\/div>\n
    arcpy.env.overwriteOutput <\/code>=<\/code> True<\/code><\/div>\n
    <\/div>\n
    def<\/code> find_identical(oid):<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>'''Worker function to perform Find Identical, and return'''<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>'''a list of numpy arrays as the results.'''<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Create a feature layer for the tile in the fishnet.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>tile <\/code>=<\/code> arcpy.management.MakeFeatureLayer(<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'C:\/testing\/testing.gdb\/fishnet'<\/code>,<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'layer{0}'<\/code>.<\/code>format<\/code>(oid[<\/code>0<\/code>]), <\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>\"\"\"OID = {0}\"\"\"<\/code>.<\/code>format<\/code>((oid[<\/code>0<\/code>])))<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Get the extent of the feature layer and set the environment<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># to use during Find Identical.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>tile_row <\/code>=<\/code> arcpy.da.SearchCursor(tile, <\/code>'shape@'<\/code>)<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>geometry <\/code>=<\/code> tile_row.<\/code>next<\/code>()[<\/code>0<\/code>]<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>arcpy.env.extent <\/code>=<\/code> geometry.extent<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Execute Find Identical<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>identical_table <\/code>=<\/code> arcpy.management.FindIdentical(<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'C:\/testing\/testing.gdb\/Points'<\/code>, <\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'in_memory\/identical'<\/code>, <\/code>'Shape'<\/code>)<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Convert the resulting table into a numpy array and return<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>result_array <\/code>=<\/code> arcpy.da.TableToNumPyArray(identical_table, [<\/code>\"*\"<\/code>])<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>return<\/code> result_array<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># End find_identical<\/code><\/div>\n
    <\/div>\n
    def<\/code> main():<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Create a list of OID's used to chunk the inputs<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>fishnet_rows <\/code>=<\/code> arcpy.SearchCursor(<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'C:\/testing\/testing.gdb\/fishnet'<\/code>, '<\/code>', '<\/code>', '<\/code>OID')<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>oids <\/code>=<\/code> [[row.getValue(<\/code>'OID'<\/code>)] <\/code>for<\/code> row <\/code>in<\/code> fishnet_rows]<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Create a pool class and run the jobs--<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># the number of jobs is equal to the length of the oids list<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool <\/code>=<\/code> multiprocessing.Pool()<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>result_arrays <\/code>=<\/code> pool.<\/code>map<\/code>(find_identical, oids)<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Concatenate the resulting arrays and create an output table<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># reporting any identical records.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>result_array <\/code>=<\/code> numpy.concatenate(result_arrays,axis<\/code>=<\/code>0<\/code>)<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>arcpy.da.NumPyArrayToTable(result_array, <\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'C:\/testing\/testing.gdb\/identical'<\/code>)<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Synchronize the main process with the job processes to ensure<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># proper cleanup.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool.close()<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool.join()<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># End main<\/code><\/div>\n
    <\/div>\n
    if<\/code> __name__ <\/code>=<\/code>=<\/code> '__main__'<\/code>:<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>main()<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

    The example above splits the data using spatial extents. However, there are tools that do not require data be split spatially. The Generate Near Table example below, shows the data processed in groups of 250000 features by selecting them based on object ID ranges.<\/p>\n

    \n
    \n\n\n\n
    \n
    1<\/div>\n
    2<\/div>\n
    3<\/div>\n
    4<\/div>\n
    5<\/div>\n
    6<\/div>\n
    7<\/div>\n
    8<\/div>\n
    9<\/div>\n
    10<\/div>\n
    11<\/div>\n
    12<\/div>\n
    13<\/div>\n
    14<\/div>\n
    15<\/div>\n
    16<\/div>\n
    17<\/div>\n
    18<\/div>\n
    19<\/div>\n
    20<\/div>\n
    21<\/div>\n
    22<\/div>\n
    23<\/div>\n
    24<\/div>\n
    25<\/div>\n
    26<\/div>\n
    27<\/div>\n
    28<\/div>\n
    29<\/div>\n
    30<\/div>\n
    31<\/div>\n
    32<\/div>\n
    33<\/div>\n
    34<\/div>\n
    35<\/div>\n
    36<\/div>\n
    37<\/div>\n
    38<\/div>\n<\/td>\n
    		\n
    \n
    import<\/code> multiprocessing<\/code><\/div>\n
    import<\/code> numpy<\/code><\/div>\n
    import<\/code> arcpy<\/code><\/div>\n
    def<\/code> generate_near_table(ranges):<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>i, j <\/code>=<\/code> ranges[<\/code>0<\/code>], ranges[<\/code>1<\/code>]<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>lyr <\/code>=<\/code> arcpy.management.MakeFeatureLayer(<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'c:\/testing\/testing.gdb\/random1mil'<\/code>,<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'layer{0}'<\/code>.<\/code>format<\/code>(i), <\/code>\"\"\"OID &gt;= {0} AND<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>OID &lt;= {1}\"\"\"<\/code>.<\/code>format<\/code>((i, j)))<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>gn_table <\/code>=<\/code> arcpy.analysis.GenerateNearTable(<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>lyr, <\/code>'c:\/testing\/testing.gdb\/random300'<\/code>,<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>'in_memory\/outnear{0}'<\/code>.<\/code>format<\/code>(i))<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>result_array <\/code>=<\/code> arcpy.da.TableToNumPyArray(gn_table, [<\/code>\"*\"<\/code>])<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>arcpy.management.Delete(gn_table)<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>return<\/code> result_array<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># End generate_near_table function<\/code><\/div>\n
    <\/div>\n
    def<\/code> main():<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>ranges <\/code>=<\/code> [[<\/code>0<\/code>, <\/code>250000<\/code>], [<\/code>250001<\/code>, <\/code>500000<\/code>], [<\/code>500001<\/code>, <\/code>750000<\/code>],<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>[<\/code>750001<\/code>, <\/code>1000001<\/code>]]<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Create a pool class and run the jobs--the number of jobs is<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># equal to the length of the oids list<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool <\/code>=<\/code> multiprocessing.Pool()<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>result_arrays <\/code>=<\/code> pool.<\/code>map<\/code>(generate_near_table, ranges)<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Concatenate the resulting arrays and create an output table<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># reporting any identical records.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>result_array <\/code>=<\/code> numpy.concatenate(result_arrays,axis<\/code>=<\/code>0<\/code>)<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>arcpy.da.NumPyArrayToTable(result_array, <\/code>'c:\/testing\/testing.gdb\/gn3'<\/code>)<\/code><\/div>\n
    <\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Synchronize the main process with the job processes to<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># Ensure proper cleanup.<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool.close()<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>pool.join()<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code># End main<\/code><\/div>\n
    <\/div>\n
    if<\/code> __name__ <\/code>=<\/code>=<\/code> '__main__'<\/code>:<\/code><\/div>\n
    \u00a0\u00a0\u00a0\u00a0<\/code>main()<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

    Considerations<\/strong><\/h2>\n

    Here are some important considerations before deciding to use multiprocessing:<\/p>\n

      \n
    1. The scenario demonstrated in the first example, will not work with feature classes in a file geodatabase because each update must acquire a schema lock on the workspace. A schema lock effectively prevents any other process from simultaneously updating the FGDB. This example will however, work with shapefiles and ArcSDE geodatabase data.<\/li>\n
    2. For each process, there is a start-up cost in loading the arcpy library (1-3 seconds). Depending on the complexity and size of the data, this can cause the multiprocessing script to take longer to run than a script without multiprocessing. In many cases, the final step in the multiprocessing workflow is to aggregate all results together which is an additional cost.<\/li>\n
    3. Determining if multiprocessing is appropriate for your workflow can often be a trial and error process. This process can invalidate the gains made using multiprocessing in a one off operation; however, the trial and error process may be very valuable if the final workflow is to be run multiple times, or applied to similar workflows using large data. For example, if you are running the Find Identical tool on a weekly basis, and it is running for hours with your data, multiprocessing may be worth the effort.<\/li>\n
    4. Whenever possible, take advantage of the \u201cin_memory\u201d workspace for creating temporary data. It can improve performance rather than writing data to disk. However, depending on the size of data being created in-memory, it may be necessary to write temporary data to disk. Temporary datasets cannot be created in a file geodatabase because of schema locking. Deleting the in-memory dataset when you are finished can prevent out of memory errors.<\/li>\n<\/ol>\n

      Summary<\/strong><\/h2>\n
        \n
      1. \n
          These are just a few examples showing how multiprocessing can be used to increase performance and scalability when doing geoprocessing. However, it is important to remember that multiprocessing does not always mean better performance.<\/ol>\n<\/li>\n<\/ol>\n

           <\/p>\n

            \n
          1. \n
              The multiprocessing module was included in Python 2.6 and the examples above will work in ArcGIS 10.1. For more information about the multiprocessing module, refer the Python documentation,<\/ol>\n<\/li>\n<\/ol>\n

              http:\/\/docs.python.org\/library\/multiprocessing.html<\/a><\/p>\n

                \n
              1. \n
                  Please provide any feedback and comments to this blog posting, and stay tuned for another posting coming soon about \u201cBeing successful processing large complex data with the geoprocessing overlay tools\u201d.<\/ol>\n<\/li>\n<\/ol>\n

                  This post was contributed by Jason Pardy, a product engineer on the analysis and geoprocessing team.<\/strong><\/em><\/p>\n"}],"authors":[{"ID":5121,"user_firstname":"","user_lastname":"","nickname":"jpardy84","user_nicename":"jpardy84","display_name":"jpardy","user_email":"jpardy@esri.com","user_url":"","user_registered":"2018-03-02 00:16:49","user_description":"","user_avatar":""}],"related_articles":"","card_image":false,"wide_image":false},"yoast_head":"\nMultiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1\" \/>\n<meta property=\"og:site_name\" content=\"ArcGIS Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/esrigis\/\" \/>\n<meta property=\"article:modified_time\" content=\"2018-12-18T19:22:35+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@ESRI\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":[\"Article\",\"BlogPosting\"],\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1\"},\"author\":{\"name\":\"jpardy\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/424b923d0991fee8d20fd49f9912af94\"},\"headline\":\"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)\",\"datePublished\":\"2012-09-26T17:59:32+00:00\",\"dateModified\":\"2018-12-18T19:22:35+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1\"},\"wordCount\":7,\"publisher\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#organization\"},\"keywords\":[\"ArcPy\",\"geoprocessing\",\"python\"],\"articleSection\":[\"Analytics\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1\",\"name\":\"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)\",\"isPartOf\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#website\"},\"datePublished\":\"2012-09-26T17:59:32+00:00\",\"dateModified\":\"2018-12-18T19:22:35+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.esri.com\/arcgis-blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#website\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/\",\"name\":\"ArcGIS Blog\",\"description\":\"Get insider info from Esri product teams\",\"publisher\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.esri.com\/arcgis-blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#organization\",\"name\":\"Esri\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2018\/04\/Esri.png\",\"contentUrl\":\"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2018\/04\/Esri.png\",\"width\":400,\"height\":400,\"caption\":\"Esri\"},\"image\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/esrigis\/\",\"https:\/\/x.com\/ESRI\",\"https:\/\/www.linkedin.com\/company\/5311\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/424b923d0991fee8d20fd49f9912af94\",\"name\":\"jpardy\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ef5394b9654988f205fc1c43beebaef1eea90faf08e750f1b4d89434fc733f9b?s=96&d=blank&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ef5394b9654988f205fc1c43beebaef1eea90faf08e750f1b4d89434fc733f9b?s=96&d=blank&r=g\",\"caption\":\"jpardy\"},\"url\":\"\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1","og_locale":"en_US","og_type":"article","og_title":"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)","og_url":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1","og_site_name":"ArcGIS Blog","article_publisher":"https:\/\/www.facebook.com\/esrigis\/","article_modified_time":"2018-12-18T19:22:35+00:00","twitter_card":"summary_large_image","twitter_site":"@ESRI","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["Article","BlogPosting"],"@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1#article","isPartOf":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1"},"author":{"name":"jpardy","@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/424b923d0991fee8d20fd49f9912af94"},"headline":"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)","datePublished":"2012-09-26T17:59:32+00:00","dateModified":"2018-12-18T19:22:35+00:00","mainEntityOfPage":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1"},"wordCount":7,"publisher":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/#organization"},"keywords":["ArcPy","geoprocessing","python"],"articleSection":["Analytics"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1","url":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1","name":"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)","isPartOf":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/#website"},"datePublished":"2012-09-26T17:59:32+00:00","dateModified":"2018-12-18T19:22:35+00:00","breadcrumb":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-desktop\/analytics\/multiprocessing-with-arcgis-approaches-and-considerations-part-1#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.esri.com\/arcgis-blog\/"},{"@type":"ListItem","position":2,"name":"Multiprocessing with ArcGIS \u2013 Approaches and Considerations (Part 1)"}]},{"@type":"WebSite","@id":"https:\/\/www.esri.com\/arcgis-blog\/#website","url":"https:\/\/www.esri.com\/arcgis-blog\/","name":"ArcGIS Blog","description":"Get insider info from Esri product teams","publisher":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.esri.com\/arcgis-blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.esri.com\/arcgis-blog\/#organization","name":"Esri","url":"https:\/\/www.esri.com\/arcgis-blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2018\/04\/Esri.png","contentUrl":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2018\/04\/Esri.png","width":400,"height":400,"caption":"Esri"},"image":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/esrigis\/","https:\/\/x.com\/ESRI","https:\/\/www.linkedin.com\/company\/5311\/"]},{"@type":"Person","@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/424b923d0991fee8d20fd49f9912af94","name":"jpardy","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/ef5394b9654988f205fc1c43beebaef1eea90faf08e750f1b4d89434fc733f9b?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ef5394b9654988f205fc1c43beebaef1eea90faf08e750f1b4d89434fc733f9b?s=96&d=blank&r=g","caption":"jpardy"},"url":""}]}},"text_date":"September 26, 2012","author_name":"jpardy","author_page":false,"custom_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/08\/Newsroom-Keyart-Wide-1920-x-1080.jpg","primary_product":"ArcMap","tag_data":[{"term_id":31181,"name":"ArcPy","slug":"arcpy","term_group":0,"term_taxonomy_id":31181,"taxonomy":"post_tag","description":"","parent":0,"count":32,"filter":"raw"},{"term_id":24321,"name":"geoprocessing","slug":"geoprocessing","term_group":0,"term_taxonomy_id":24321,"taxonomy":"post_tag","description":"","parent":0,"count":129,"filter":"raw"},{"term_id":24341,"name":"python","slug":"python","term_group":0,"term_taxonomy_id":24341,"taxonomy":"post_tag","description":"","parent":0,"count":171,"filter":"raw"}],"category_data":[{"term_id":23341,"name":"Analytics","slug":"analytics","term_group":0,"term_taxonomy_id":23341,"taxonomy":"category","description":"","parent":0,"count":1326,"filter":"raw"}],"product_data":[{"term_id":36991,"name":"ArcMap","slug":"arcgis-desktop","term_group":0,"term_taxonomy_id":36991,"taxonomy":"product","description":"","parent":36981,"count":325,"filter":"raw"}],"primary_product_link":"https:\/\/www.esri.com\/arcgis-blog\/?s=#&products=arcgis-desktop","_links":{"self":[{"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/blog\/182521","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/blog"}],"about":[{"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/types\/blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/users\/5121"}],"replies":[{"embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/comments?post=182521"}],"version-history":[{"count":0,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/blog\/182521\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/media?parent=182521"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/categories?post=182521"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/tags?post=182521"},{"taxonomy":"industry","embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/industry?post=182521"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/product?post=182521"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}