Easy Access to Near Real-Time Data
Looking at a methodology that generates aggregated live feeds for ArcGIS clients
Esri has developed a methodology that makes near real-time data and the results of analysis available through all ArcGIS clients. Called aggregated live feeds, this technique was developed by Derrick Burke and Paul Dodd of Esri. It aggregates information captured from the Internet or Web sites; pushes that information into an ArcSDE database; and publishes this information through ArcGIS Server as a Web mapping service that can be accessed by ArcGIS Desktop, ArcGIS Server, and ArcGIS Explorer. Because these operations are performed on the server side, any client can use them without the need for custom code.
Aggregated live feeds make near real-time data and the results of analysis available through all ArcGIS clients.
Aggregated live feeds are used in applications ranging from homeland security to environmental analysis. For example, a lightweight browser application created for modeling contaminant leaks calls analysis capabilities from an ArcGIS Server geoprocessing service to perform plume modeling based on an aggregated ArcGIS Server service that contains the latest wind velocity and direction information. The analysis produces a plume that can be chained to other ArcGIS Server analyses such as identifying the demographics of the affected area. This methodology uses simple batch scripting with a handful of public domain command line utilities to download and preprocess the feed data. The scripts use ArcSDE command line functions to push this data into the database. The scripts also incorporate logic that tracks the process to ensure they run as expected. If a load fails, an alert e-mail is sent to the administrator. Scripts can run at varying intervals--every 5 minutes, every 30 minutes, once an hour, or once a day--depending on the application. This takes the load off clients. Continuously polling for fresh data can carry a heavy penalty, especially in browser-based applications. Processing feed data on the server side allows the client to poll for data only when needed, via standard ArcGIS Server protocols. Managing these feeds centrally can ease the demand on network resources by using a few systems
36 ArcUser Spring 2009
Near real-time data, such as this severe weather warning information, can be made available to ArcGIS Explorer and other ArcGIS clients, without requiring custom coding, using aggregated live feeds.
Name Description A tool that is part of the SE_TOOLKIT by Vince Angelo. This tool will load ASCII files (such as comma-separated values [CSV] files) into ArcSDE. A full featured command line e-mail tool used to send alerts when a load script fails. BLAT is also used in the Service Monitor. A tool written by Srinivasa Rao that converts Google KML files to shapefiles. Stream Editor used to edit text files through the command line. A command line tool that will wait for another process or a specific number of seconds. It is used to pause a script and monitor the ArcSDE sdemono status command through the Service Monitor. A powerful HTTP, HTTPS, and FTP command line tool used to download data from the Internet. Exports XML formatted content to a CSV file. URL
ASC2SDE
arcscripts.esri.com/details.asp?dbid=14767
BLAT
www.blat.net/
KML2SHP sed
arcscripts.esri.com/details.asp?dbid=14980 www.gnu.org/software/sed/
WAIT
dpotter.net/Technical/2007/09/utility-to-waitfor-a-process-waitexe/
GNU Wget XML2CSV
www.gnu.org/software/wget/index. html#downloading www.a7soft.com/xml2csv.html
Resources
www.esri.com