ArcGIS Living Atlas

Don't Let Your Content Go Stale!

Constantly keeping your layers updated can be a laborious task.  To keep some of our layers updated in the ArcGIS Living Atlas of the World we use python scripts.  These scripts are easy to use and available to you.  The scripts check to make sure your layers are using the most recent underlying source data each time they are executed.

Using the script, let’s walk through an example using Water Temperature from NOAA and the web upload method.

Step 1: Download the Script

Overwrite Feature Service Script
Overwrite Feature Service Script

Extract and place the script in: “c:\scripts\OverwriteFS”

Step 2: Create a copy of the script and name it “OverwriteFS”

When importing a script, Python does not like multiple periods in the filename.

Create a copy of the script and name it “OverwriteFS”.
Create a copy of the script and name it “OverwriteFS”.

Step 3: Download water temperature data from source URL

Water Temperature Data downloaded from NOAA
Water Temperature Data downloaded from NOAA

Step 4: Add Item to ArcGIS Online

Navigate to your content page and select “Add Item > From My Computer” (You must have publisher credentials to publish data).

 

Configure the field names that contain the coordinate information and add the appropriate tags:

 

The newly created content item for Water Temperature should look like this:

take note of the item ID in URL (highlighted in image above).

Step 5: Launch Python 3

If you have ArcPro installed and are relying on the associated python library the location is most likely here: “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe” although this location may be different on your computer.  More Info.

When the console launches type the following:

import arcgis
arcgis.GIS(“ArcGIS Online ORG URL”,”username”,”PASSWORD”,profile=”profile name”)

The highlighted portion is return indicating that the profile was created. (above)
1. ArcGIS Online Org
2. Username
3. Password
4. Profile Name

More Info: https://developers.arcgis.com/python/guide/working-with-different-authentication-schemes/

Step 6: Run script to keep data updated
Each one of these methods can be used to keep your data updated.

Method 1: Using ArcGIS Pro Python Command Line
Launch the Python Command Line from ArcGIS Pro “View” Menu.

First, register the directory where we saved the OverwriteFS.py script.
To do this, type the following:

import sys
sys.path
sys.path.append(r”C:\Scripts\OverwriteFS”)

Then import and execute the Overwrite FS.py script

import OverwriteFS
import arcgis
gis = arcgis.GIS(profile = “Keith VanGraafeiland“)
item = gis.content.get(“326829d3d88b4c7093c1abad69da5553“)
OverwriteFS.overwriteFeatureService(item,”https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius“)

*Items in bold are user/item specific.

Running Script from ArcGIS Pro Python Command Line
Running Script from ArcGIS Pro Python Command Line

Method 2 – Using Python
Calling and executing the script from python command line

Navigate to: “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3”
Run (double click) python.exe
First, register the directory where we saved the OverwriteFS.py script.
Type the following:

import sys
sys.path
sys.path.append(r”C:\Scripts\OverwriteFS”)

Then import and execute the Overwrite FS.py script

import arcgis
import OverwriteFS
gis = arcgis.GIS(profile = “Keith VanGraafeiland“)
item = gis.content.get(“326829d3d88b4c7093c1abad69da5553“)
OverwriteFS.overwriteFeatureService(item,”https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius“)

*Items in bold are user/item specific.

Running Script from Python
Running Script from Python

Method 3 – Command Prompt
Launch Command Prompt and call the Python Program to execute the script:

“C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe” “C:\Scripts\OverwriteFS\OverwriteFS.py” “Keith VanGraafeiland” “326829d3d88b4c7093c1abad69da5553” “SOS_WaterTemperature_Collection_Latest” “https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius

*Items in bold are user/item specific.

Running the script from the Command Prompt
Running the script from the Command Prompt

The layer will be checked/updated each time the script is ran.

Step 7: Setting up script to run in Task Scheduler

Launch Task scheduler and “Create a Basic Task”.

 

Adding a basic task in Task Scheduler
Adding a basic task in Task Scheduler

Create a new Basic Task and name it appropriately.
Under security options: check “Run whether user is logged on or not” (so it will run without requiring the user to be logged in)
Select “Do not store password” (so it will run without requiring a stored user account password – which can change from time to time)

Add an action in Task Scheduler
Add an action in Task Scheduler

Inputs for 1 and 2 in image above:

1. “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe”
2. “C:\Scripts\OverwriteFS\OverwriteFS.py” “Keith VanGraafeiland” “326829d3d88b4c7093c1abad69da5553” “SOS_WaterTemperature_Collection_Latest” “https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius

*Items in bold are user/item specific.

Triggers in Task Scheduler
Triggers in Task Scheduler

Schedule the task to run as often as you like using the triggers tab.
Click “OK” to save the properties.
In the Task Scheduler Library – Right click and run the task to test and make sure it completes successfully.  When completed the task should indicate “Operation Completed Successfully”.

It’s possible to setup your MAC to run Automator to do the same lift as window’s task manager as well.

Running Scheduled Task
Running Scheduled Task

Check the content item, now you can see the “Data Update Details” reflect the most recent update.

Updated Content Item
Updated Content Item

Since this layer (Water Temperature) is constantly being refreshed from a sensor network, the updates will occur every time it checks against the source data.   This script can also be used against data that is updated less frequently to ensure the hosted feature class is using the most recent version of the source data.  This script and methodology can prove to be very useful for keeping your data current and reliable.

Pros: Easy, Simple
Cons: Can be lengthy overwrite if large or complex data

Water Temperatures (°C) off the Mid-Atlantic Coast of the United States. (Data Courtesy of NOAA Co-Ops)
Water Temperatures (°C) off the Mid-Atlantic Coast of the United States. (Data Courtesy of NOAA Co-Ops)

This script is really easy to use.  If you or your organization relies on having updated layers this is a good method to automate that process.  There are many options for running the script to update the data, I prefer using task manager.  This also works with other formats.  I used a .CSV, but it works with other formats such as shapefiles, geodatabase, and geoJSON.  With the data being updated reliably and regularly – now you have the ability to detail out the associated content item and prepare the data for nomination to Living Atlas.

We have an update coming for the script.  It will include some new enhancements and additional functionality that includes:

Links:

Query for Data: https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service%3DSOS%26request%3DGetObservation%26version%3D1.0.0%26observedProperty%3Dsea_water_temperature%26offering%3Durn%3Aioos%3Anetwork%3ANOAA.NOS.CO-OPS%3AAll%26responseFormat%3Dtext%2Fcsv%26unit%3DCelsius

Overwrite Feature Service Script: http://esriurl.com/OverwriteFS

Other “Live Feeds” scripts are available here: http://www.esriurl.com/LiveFeed

About the author

Keith is a Product Engineer at Esri. He serves as the Ocean Curator for the Living Atlas of the World team. Keith works to create foundational layers that can help marine researchers, scientist, and others gain a better understanding of our oceans.

Connect:

Next Article

Updated classic story map transition timeline

Read this article