"}],"short_description":"Leverage Apache Arrow to transport data at increased efficiency between ArcGIS Pro and open-source components.","flexible_content":[{"acf_fc_layout":"content","content":"ArcGIS continues to grow as a geospatial data science platform, incorporating specialized geospatial data science tools with open-source components. With this ever-increasing network and the ever-increasing volume of data we need some way to efficiently connect between these various components. Apache Arrow may help.<\/p>\n
Apache Arrow<\/a> is a burgeoning, ambitious, open-source project by Wes McKinley and partners. For some time now it has been slowly finding its way into various popular data and analytics platforms<\/a>. In short, Apache Arrow is an in-memory, columnar, cross-platform, cross-language, and open-source data representation that allows you to efficiently transfer data between components. It is intended to sit low in the stack:<\/p>\n"},{"acf_fc_layout":"quote","author_name":"The Apache Organization","author_profession_organization":"","image":false,"text":"[Apache Arrow] is designed to both improve the performance of analytical algorithms and the efficiency of moving data from one system or programming language to another."},{"acf_fc_layout":"content","content":" In other words, unlike user-facing Pandas and Spark data frames, Apache Arrow\u2019s data representation is intended to sit behind the scenes at a lower level, efficiently running the logistics regardless of platform or language.<\/p>\n One of the most powerful promises of Arrow is to serve as a sort of Esperanto (or common language) for data transport\u2014a super-efficient, often zero-copy vehicle that can thread the interfaces between various platforms, including ArcGIS Pro.<\/p>\n In this blog, you\u2019ll learn how to leverage Apache Arrow to improve your workflows across components like Pandas (including Spatially Enabled Data Frames and Geopandas), Spark, Parquet, and ArcPy.<\/p>\n We added support for reading and writing Arrow Tables to ArcPy at ArcGIS Pro 2.9. With the release of ArcGIS Pro 3.2, we improved upon this feature by adding support for additional data types and geometry encodings. This allows you to connect your ArcGIS Pro workflows with other data and analytics platforms by transporting your geospatial data using Arrow Tables. As Apache Arrow grows in popularity and adoption, support for it will expand on other platforms. So, if you\u2019re searching for an efficient path for bringing your geospatial data from other projects into ArcGIS (or vice versa), leveraging ArcPy\u2019s integration with Arrow may, in some cases, offer the best solution.<\/p>\n Apache Arrow\u2019s interface for Python is provided by the PyArrow<\/a> library.<\/p>\n Arrow Tables<\/a> are a tabular data representation composed of columns, in which each column has a field name, data type, and the data itself (as well as optional metadata, more on this later).<\/p>\n To convert a Featureclass to an Arrow Table, you can use the The geometry column in the resulting Arrow Table will be encoded in the EsriShape<\/em> binary format. This format is efficient and lossless, but it is also incompatible with most other analytics platforms. When you need the exported geometry data to be compatible with another platform, you can choose a different geometry encoding with the optional Most geospatial analytics and data platforms will support reading or writing at least one of these formats.<\/p>\n Reading an Arrow Table into ArcGIS Pro is done by passing the Arrow Table into ArcPy can read Arrow Tables with all the same geometry encodings it can write (EsriShape, EsriJSON, GeoJSON, WKT, and WKB). However, when the Arrow Table did not originate from ArcPy, you may need to do some additional prep work to ensure the table can be successfully read by ArcPy. You\u2019ll learn about that in the next section.<\/p>\n ArcPy uses metadata keys embedded with the Arrow Table columns to determine how to interpret the data. The metadata is stored as part of a table\u2019s schema. When using Arrow Tables as a vehicle for moving data between different geospatial data and analytics platforms, it is important to understand the schema specification for Apache Arrow Tables that ArcPy supports. You can view an Arrow Table\u2019s schema using its schema method.<\/p>\n For the Arrow Table from the previous example, Note the metadata attached to the Shape field. The You can find additional information about the required schema and the mappings and metadata for the supported field data types in the Type conversions section of the Apache Arrow in ArcGIS documentation<\/a>.<\/p>\n The schema profile for Apache Arrow Tables supported by ArcPy is not the only Arrow Table schema profile for geospatial data, there is also the GeoArrow specification<\/a>. ArcPy also supports reading Arrow Tables with a GeoArrow schema. However, ArcPy will not create Arrow Tables with the GeoArrow schema.<\/p>\n Some platforms may not preserve an Arrow Table\u2019s original schema or produce an Arrow Table with a schema ArcPy understands. In cases where they don\u2019t, you will need to reconstruct the schema either from scratch or using the original schema.<\/p>\n While Apache Arrow is an efficient but temporary in-memory data structure for fast operations, Apache Parquet<\/a> is an on-disk data structure for space efficient long-term storage. In short, Apache Arrow is for processing and moving of data, and Apache Parquet is for storage. The two formats are optimized for compatibility. This compatibility means that the schema will be preserved when writing an Arrow Table to a parquet file for long-term storage.<\/p>\n Here’s how you can move geospatial data between Parquet files and ArcGIS Pro using Apache Arrow:<\/p>\n"},{"acf_fc_layout":"sidebar","content":"","image_reference":false,"layout":"code_snippet","image_reference_figure":"","snippet":"import os\r\nimport pyarrow.parquet as pq\r\n\r\n# Using the `nsowlnests_at` created previously from ArcPy:\r\n\r\n# Write to parquet for long-term storage.\r\nws = r'C:\\data'\r\nnsowlnests_pq = os.path.join(ws, \"northern_spotted_owl_nests.parquet\")\r\n# _pq for Parquet\r\n\r\npq.write_table(nsowlnests_at, nsowlnests_pq)\r\n\r\n# After some time in storage...\r\n\r\n# Read (retrieve from storage) the parquet file to an Arrow Table. \r\n# The original Arrow Table\u2019s schema is preserved.\r\nretrieved_at = pq.read_table(nsowlnests_pq)\r\n\r\n# Now use `retrieved_at` in ArcPy.","spotlight_name":"","section_title":"","position":"Center","spotlight_image":false},{"acf_fc_layout":"content","content":" The Pandas<\/a> DataFrame is a table-like in-memory data structure with an interface for data analysis. The Pandas team plans to completely back Pandas with Apache Arrow (instead of NumPy) when Pandas 3.0 is released. With the recently released Pandas 2.0, backing a DataFrame with Apache Arrow is optional. ArcGIS Pro 3.2 ships with Pandas version 2.0.2, so you can try this out yourself.<\/p>\n In the following example, we will use Arrow to move geospatial data between a Pandas DataFrame and ArcGIS Pro, and leverage the new Arrow backed data types in Pandas:<\/p>\n"},{"acf_fc_layout":"sidebar","content":"","image_reference":false,"layout":"code_snippet","image_reference_figure":"","snippet":"import pandas as pd\r\nimport pyarrow as pa\r\n\r\n# Using the `nsowlnests_at` created previously from ArcPy:\r\n\r\n# Store the Arrow Table\u2019s schema in `schema` for later, because\r\n# it will not be preserved during the conversion to a Pandas DataFrame.\r\nschema = nsowlnests_at.schema\r\n\r\n# Define a data type mapping (to arrow data types) for Pandas to use.\r\ndtype_mapping = {\r\n pa.int8(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.int8()),\r\n pa.int16(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.int16()),\r\n pa.int32(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.int32()),\r\n pa.int64(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.int64()),\r\n pa.uint8(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.uint8()),\r\n pa.uint16(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.uint16()),\r\n pa.uint32(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.uint32()),\r\n pa.uint64(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.uint64()),\r\n pa.float32(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.float32()),\r\n pa.float64(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.float64()),\r\n pa.float64(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.float64()),\r\n pa.bool_(): pd.core.arrays.arrow.dtype.ArrowDtype(pa.bool_()),\r\n pa.binary():pd.core.arrays.arrow.dtype.ArrowDtype(pa.binary()),\r\n pa.string():pd.core.arrays.arrow.dtype.ArrowDtype(pa.string())\r\n}\r\n\r\n# Convert the Arrow Table to a Pandas DataFrame using `dtype_mapping`.\r\nnsowlnests_pdf = nsowlnests_at.to_pandas(types_mapper=dtype_mapping.get) \r\n# _pdf for Pandas DataFrame\r\n\r\n# After some processing performed on the Pandas DataFrame...\r\n\r\n# Convert the Pandas DataFrame back to an Arrow Table,\r\n# applying the schema stored earlier.\r\nretrieved_at = pa.Table.from_pandas(nsowlnests_pdf, schema=schema)\r\n\r\n# Now use `retrieved_at` in ArcPy.\r\n","spotlight_name":"","section_title":"","position":"Center","spotlight_image":false},{"acf_fc_layout":"content","content":" In testing, the While moving data between ArcGIS and Pandas can be useful, Pandas has no inherent geospatial data processing and analysis capabilities. For this, you will need to look to the ArcGIS API Spatially Enabled DataFrame in the next section.<\/p>\n The ArcGIS API for Python<\/a>\u2019s Spatially Enabled DataFrame (SEDF) is built on top of Pandas. Essentially, it extends the Pandas DataFrame with geospatial capabilities, with interoperability between SEDF and ArcPy. An SEDF can be created from a Featureclass using the ArcGIS API, and ArcPy can directly read the SEDF format as input to geoprocessing tools, so you don\u2019t necessarily need to use Arrow. However, you can use Arrow in this transaction as well. By converting the SEDF to an Arrow Table first, and then using the Arrow Table with ArcPy instead of the SEDF, testing resulted in roughly a 14 percent boost in performance (again, your mileage may vary).<\/p>\n The below code shows how you can leverage Arrow to move geospatial data between the ArcGIS API SEDF and ArcGIS Pro to gain a slight performance boost:<\/p>\n"},{"acf_fc_layout":"sidebar","content":"","image_reference":false,"layout":"code_snippet","image_reference_figure":"","snippet":"import arcgis\r\nimport pandas as pd\r\n\r\n# Convert a Featureclass to an ArcGIS API SEDF.\r\nnsowlnests_path = r'C:\\data\\forestry.gdb\\northern_spotted_owl_nests'\r\nnsowlnests_sedf = pd.DataFrame.spatial.from_Featureclass(nsowlnests_path)\r\n\r\n# After some geospatial processing performed on the ArcGIS API SEDF...\r\n\r\n# Convert the ArcGIS API SEDF to an Arrow Table.\r\nretrieved_at = nsowl_nests_sedf.spatial.to_arrow()\r\n\r\n# Now use `retrieved_at ` in ArcPy.\r\n","spotlight_name":"","section_title":"","position":"Center","spotlight_image":false},{"acf_fc_layout":"content","content":"Leverage Apache Arrow in ArcPy<\/h1>\n
Apache Arrow in Python<\/h3>\n
Arrow Tables<\/h3>\n
Write an Arrow Table from ArcPy<\/h2>\n
arcpy.da.TableToArrowTable<\/code> function.<\/p>\n"},{"acf_fc_layout":"sidebar","content":"","image_reference":false,"layout":"code_snippet","image_reference_figure":"","snippet":"import arcpy\r\n\r\nnsowlnests_path = r'C:\\data\\forestry.gdb\\northern_spotted_owl_nests'\r\n\r\n# Convert a Featureclass to an Arrow Table.\r\nnsowlnests_at = arcpy.da.TableToArrowTable(nsowlnests_path) # _at for Arrow Table","spotlight_name":"","section_title":"","position":"Center","spotlight_image":false},{"acf_fc_layout":"content","content":"geometry_encoding<\/code> parameter, which supports the additional geometry encodings EsriJSON<\/em>, GeoJSON<\/em>, WKT<\/em>, and WKB<\/em>. These are publicly documented formats for representing geometries, which you can read about at the following sites:<\/p>\n\n
Read an Arrow Table to ArcPy<\/h2>\n
arcpy.management.CopyRows<\/code> (for Tables) or arcpy.management.CopyFeatures<\/code> (for Featureclasses). In fact, geoprocessing tools accept Arrow Tables as a data source, so you can directly use an Arrow Table as input to a geoprocessing tool.<\/p>\n"},{"acf_fc_layout":"sidebar","content":"","image_reference":false,"layout":"code_snippet","image_reference_figure":"","snippet":"buffer = '0.7 MILES'\r\noutfc = r'C:\\data\\forestry.gdb\\protected_areas'\r\n\r\n# Use the Arrow Table in a Geoprocessing tool.\r\narcpy.analysis.Buffer(nsowlnests_at, outfc, buffer)\r\n","spotlight_name":"","section_title":"","position":"Center","spotlight_image":false},{"acf_fc_layout":"content","content":"Interoperability with other analytics components<\/h1>\n
<Arrow Table Object>.schema<\/code><\/p>\n'nsowlnests_at'<\/code>, which contains the columns OID, Shape, NEST_ID, TREE_SPECIES, NEST_HEIGHT_M, LAST_ACTIVE_YEAR, and NOTES, the schema looks like this:<\/p>\n"},{"acf_fc_layout":"sidebar","content":"","image_reference":false,"layout":"code_snippet","image_reference_figure":"","snippet":"OID: int64 not null\r\n -- field metadata --\r\n esri.oid: 'esri.int64'\r\nShape: binary\r\n -- field metadata --\r\n esri.sr_wkt: 'GEOGCS[\"GCS_WGS_1984\",DatUM[\"D_WGS_1984\",SPHEROID[\"WGS_19' + 111\r\n esri.encoding: 'EsriShape'\r\nNEST_ID: int32\r\nTREE_SPECIES: string\r\nNEST_HEIGHT_M: int32\r\nLAST_ACTIVE_YEAR: int32\r\nNOTES: string","spotlight_name":"","section_title":"","position":"Center","spotlight_image":false},{"acf_fc_layout":"content","content":"esri.sr_wkt<\/code> key defines the coordinate system of the geometry stored in this column using Well-known-text of Coordinate Reference Systems<\/a> (WKT CRS). The esri.encoding<\/code> key specifies the geometry encoding of the data, in this case EsriShape. The data type of the Shape field is binary. Note that different geometry encodings may require different field data types. For example, if the Shape field held GeoJSON encoded geometry instead, it would need to be of data type string.<\/p>\nParquet<\/h2>\n
Pandas DataFrames<\/h2>\n
from_pandas<\/code> operation sees a significant performance boost of roughly 40 percent from the Pandas DataFrame being backed with Arrow data types rather than NumPy, but your mileage may vary. Pandas 3.0 is expected to standardize this once it is released.<\/p>\nSpatially Enabled DataFrames<\/h2>\n