Version 1.8.3 of the ArcGIS API for Python is here
The Python API is designed to serve a wide spectrum of users – from GIS administrators, to Python developers and data scientists. Our users have trusted the API and use it to administer their web GIS, publish, update and clone a wide variety of GIS data, build sophisticated analytical models and automate their mission critical workflows. Over the last few years, the API has grown from a quaint little library to one with several different modules. Now we have over 2300 methods and functions!
The picture above (presented at Python Libraries for Spatial Data Science talk) distills some of the key capabilities of the API. At version 1.8.3, we have tried to enhance each of these aspects and this blog is a rundown of the top features.
Framework improvements
One of the biggest improvements we performed behind the scenes was migrating the HTTP communication modules to use requests (a popular 3rd party module) from built-in urllib module. There is no change to your scripts, except that now and going forward we can better support some of your custom and complex authentication systems. For JSON parsing, we switched to UltraJSON. Combined, we were able to improve support for fundamental features such as multipart uploads while extracting performance improvements. Apart from this, we have added support for Python 3.8, JupyterLab 1x, 2x and Pandas v1.0.
GIS Administration
ArcGIS Notebooks and the Notebook Server play a key role in enabling spatial data science capability of the ArcGIS Platform. The arcgis.gis.nb module has been enhanced allowing you to manage notebook servers. You can now inspect the instances that are running and their properties. This module also enables powerful workflows such as executing notebooks on demand and scheduling notebooks. In the future, we plan to support managing different versions of notebook items.
There have been numerous small updates and fixes to other parts of the GIS administration API. When creating users, for instance, you can now use smart defaults you as an admin have defined for your org. A new AGOLUsageReports class allows better compilation of usage reports in ArcGIS Online, something our users have been asking for.
On the security front, the API now supports authentication using API keys. This is a new login mechanism targeted toward GIS developers and you will be learning more about it in the coming months.
Content management
We are excited to introduce the new `arcgis.apps.dashboard` module that allows users to compose ArcGIS Dashboards apps using Python! A lot of data scientists want a medium that is simpler than notebooks to communicate their results and we hope this module can help them quickly compose such apps.
Content migration and cloning is a very important workflow for our users, and we are happy to introduce a new ‘GroupMigrationManager’ workflow that allows group admins to migrate content from one group to another across organizations. Long term users of the API would be familiar with `clone_items` function that is native to the Python API. It has been enhanced to support a slew of new app types along with various bug fixes.
This release also adds new publishing workflows – such as publishing directly from content deposited into a data store and the ability to set desired item ids for certain workflows and environments.
On the OGC and FOSS4G fronts, we have added a new `arcgis.mapping.ogc` sub-module to work with OGC layers. Another exciting development is the ability to directly read in-memory GeoPandas GeoDataFrame objects into ArcGIS! The API now supports reading geopackage files and newer columnar file formats like feather. We are researching some of these new patterns to increase the performance of the Spatially enabled DataFrame (SeDF) objects.
Spatial analysis
Let’s talk about Spatially enabled DataFrames (SeDF). You can now pass SeDF objects as inputs to spatial analysis tools. This allows you to easily enable data engineering workflows to use the powerful spatial analysis tools available as ready-to-use tools on your web GIS. Another feature is the ability to sanitize column names before saving the DataFrame. SeDF objects now support a new `renderer` property to store symbology or drawing info. When you read a feature layer, the renderer will now pick up the symbology defined on that web layer and utilize that to visualize the layer on a map. You can override it with your custom symbols at any time.
Now almost all web tools in Python API support asynchronous processing. Async processing allows you to submit big and long running tasks to the server and get back a GPJob object. This essentially frees up the Python kernel, allowing you to run subsequent commands while the server keeps working on the submitted job. You can poll the job object anytime to know the status or cancel the job.
Raster analysis
While raster is part of spatial analysis section covered above, a lot of work went into it that we thought it warrants a new section of its own. In major news, we added two new classes – `Raster` and `RasterCollection` that allows you to work with a wide variety of imagery data in a seamless manner. You can read and work with local file-based raster data just as well as those published as imagery layers (services). When you read local raster data, the API chooses ArcPy as the processing engine and if you read an Image service, it uses the ArcGIS Image Server as the back-end engine. This allows users to work with one API immaterial of where the data is stored and processed in the back-end.
A slew of new raster functions and distributed raster analytics tools for applications such as trend analysis, multidimensional analysis, change detection etc. have been added.
Deep Learning
A significant feature of the API is its ability to train deep learning models for geospatial applications using the `arcgis.learn` module. The API now supports over 30 models for various advanced workflows. These models can work with a wide variety of data such as planimetric & oriented imagery, point cloud, feature, tabular, time-series data and unstructured text. This release adds models for key workflows such as edge detection, change detection, road extraction and image translation.
A new computer-vision model allows users to go from scanned paper maps to georeferenced vector layers! This release challenges the somewhat common myth in geospatial industry that deep learning is best suited only for imagery data by forging ahead with models that work on tabular and time-series data. The `FullyConnectedNetwork` model allows you to feed tabular and raster layers into a deep neural network, letting it learn and model even the most complex of patterns. The `TimeSeriesModel` allows you to perform time-series forecasting.
The new `MLModel` class allows you to integrate machine learning models from the popular scikit-learn library and the new `arcgis.learn.text` submodule now houses a number of models for natural language processing (NLP). What does NLP have to do with GIS you ask? For starters, you can extract addresses and location information from unstructured text, predict missing parts of an incomplete address, and auto-correct errors in addresses making them suitable for geocoding.
To simplify the installation of the libraries and frameworks needed to light up deep learning, we have added new deep learning installers for ArcGIS Pro and Enterprise and a new conda metapackage for independent installation.
A significant addition around this release is the availability of ready-to-use deep learning models in Esri Living Atlas. Since these models have been previously trained by Esri, you can skip over the tedious and data-intensive task of cleaning and preparing large volumes of data necessary to train your models or procuring the hardware necessary for such tasks. You can download the models and proceed to perform inference.
Mapping and visualization
The map widget in Jupyter notebooks can now link with other map widgets in the same notebook. You can thus synchronize navigation between two or more widgets allowing you to load and visualize different layers side by side for the same extent. Once linked any interaction in one of the linked maps will cause other maps to follow suit as shown in the animation below.
You are not limited to linking just 2D maps. The synced navigation works just as well between 2D and 3D modes of widgets allowing you to easily view both perspective and planimetric rendering of the same study area.
By now, you must be familiar with the ability to take screenshots of your map widgets. While screenshots embed a static version of the map as an image in the notebook, with this release, you can embed live, interactive versions of the map. Thus, when you export your notebook as a HTML file, you now have the ability to choose whether to embed your maps as static images or as dynamic web maps. When the map in a notebook is just not big enough to fully appreciate the geography, you can use the existing export to HTML function to generate a full size web page of your map.
Did I mention you can load local file-based raster files on the map widget? Yes, you can now simply add the new Raster class as a layer to your maps and easily visualize your imagery and raster data.
Fans of JupyterLab IDE? We now support versions 1x and 2x of lab and you can now search for the ‘arcgis’ extension within the lab and install it without breaking a sweat. JupyterLab is a cool new interface and I recommend you check it out if you haven’t already.
Documentation
The year 2020 has been a very busy year for us in terms of documentation. We started on a mission to overhaul some of our older guides and replace them with comprehensive, tutorial style notebooks. So far, we have rewritten guides for network analysis, GeoEnrichment, orthomapping and the geometry modules. You’ll notice these guides begin by providing sufficient foundational information followed by a layout diagram which describes what different parts of the module are and how they work with rest of the API. These guides are also numbered in parts, demonstrating a clear progression for our new users.
We have also embarked on a long-term project to improve the API reference by adding contextual snippets for parameters, cross referencing keywords, adding schematic images for certain tools and example Python scripts.
The ‘try-it-live’ experience has now been migrated to a new home in ArcGIS Online. When you click on the ‘Try it live’ button on sample notebooks, you will be taken to a notebook that runs on ArcGIS Online! Simply login with your credentials and test drive the samples.
Conclusion
Wow! That was a long drive through the various modules of the API and I have only covered the key features. How about I leave you with a single slide overview of the above?
Thirsty for more details? Check out our very detailed release notes.
2020 has been challenging to each of us in so many unique and different ways. I am proud of our team that came together and accomplished all of the above. We are passionate about what we do and are determined to bring you a first-class Python API for geospatial analysis.
From all of us at the Python API team, thank you for being our users. We wish you a safe and happy holiday season.
Article Discussion: