"},{"ID":342532,"user_firstname":"Sarah","user_lastname":"Battersby","nickname":"Sarah Battersby","user_nicename":"sbattersby","display_name":"Sarah Battersby","user_email":"sbattersby@esri.com","user_url":"","user_registered":"2023-07-17 22:33:34","user_description":"Sarah is a Product Manager for ArcGIS GeoAnalytics Engine. She has a PhD in Geography \/ Cognitive Science from UC Santa Barbara, and enjoys finding ways to make spatial technologies easier to use, understand, and trust.","user_avatar":"
"}],"short_description":"Enable efficient data integration and scalability with ArcGIS GeoAnalytics Engine and AWS Glue.","flexible_content":[{"acf_fc_layout":"content","content":"When your organization\u2019s data grows and diversifies, geospatial analysis faces two key challenges: efficient data integration and scalability. ArcGIS GeoAnalytics Engine<\/a> \u2013 a cloud-native library with a comprehensive set of \u00a0spatial functions and tools \u2013 addresses these challenges by moving geoanalytics workflows directly to where your data resides – in data lakes, data warehouses, or ArcGIS. GeoAnalytics Engine is fully integrated with Apache Spark, thus can process and analyze spatial datasets at scale with advanced geospatial operations including space-time pattern mining, track data analysis, geocoding, reverse geocoding, network analysis, and spatial modeling.<\/p>\n Accessing and preparing large, diverse datasets from various sources can be painstaking and time-consuming \u2013 that is where AWS Glue comes in. AWS Glue<\/a> is a serverless data integration service designed to simplify the discovery, preparation, transfer, and integration of your datasets from various sources for analytics, machine learning, and application development. It automatically scales compute resources, supports Spark and PySpark jobs, and runs your extract, transform, and load (ETL) tasks, making it more efficient to clean, organize, and securely move datasets between sources like Amazon S3<\/a> and Redshift<\/a>.<\/p>\n GeoAnalytics Engine in AWS Glue offers your organization a streamlined geospatial data integration with both data lakes and ArcGIS. With that, you are now better equipped to orchestrate your ETL pipeline that can run highly scalable geospatial analysis both on-demand and on-schedule.<\/p>\n Note that while ArcGIS GeoAnalytics Engine is certified and tested with AWS EMR, it is not specifically tested with AWS Glue. At this time, network analysis and geocoding tools are not supported in Glue.<\/p>\n Let\u2019s showcase how you can get up and running with GeoAnalytics Engine in AWS Glue Studio<\/a> to run geoanalytics workflows \u00a0interactively or on schedule<\/a>. First, follow the guideline here<\/a> to download ArcGIS GeoAnalytics Engine distribution. Next, sign in to your AWS Management Console<\/a> and upload those distribution files to your S3 bucket. After that navigate to AWS Glue Studio and start or upload a Notebook with an appropriate IAM role<\/a> so that it has permissions for data access and ETL job.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2789472,"id":2789472,"title":"Picture1","filename":"Picture1.png","filesize":26706,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture1.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/geoanalytics-engine\/analytics\/solving-big-data-geoanalytics-challenges-with-arcgis-geoanalytics-engine-in-aws-glue\/picture1-120","alt":"","author":"323502","description":"","caption":"","name":"picture1-120","status":"inherit","uploaded_to":2789462,"date":"2025-05-10 21:17:04","modified":"2025-05-10 21:17:04","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":637,"height":157,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture1-213x157.png","thumbnail-width":213,"thumbnail-height":157,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture1.png","medium-width":464,"medium-height":114,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture1.png","medium_large-width":637,"medium_large-height":157,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture1.png","large-width":637,"large-height":157,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture1.png","1536x1536-width":637,"1536x1536-height":157,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture1.png","2048x2048-width":637,"2048x2048-height":157,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture1.png","card_image-width":637,"card_image-height":157,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture1.png","wide_image-width":637,"wide_image-height":157}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" IAM Role Configuration for Secure Access:<\/strong><\/p>\n Let\u2019s\u00a0 start with an empty notebook. In the notebook cell, first, configure the compute resources on the backend including the Glue version, worker type and number of workers, using the following commands:<\/p>\n For more information about scaling your AWS Glue for Apache Spark jobs, check out this blog post<\/a>.<\/p>\n Given that we have already uploaded GeoAnalytics Engine installation files to S3, next, we import those necessary JARs and Python Wheel (.whl) files to the Spark environment, using the following commands:<\/p>\n Now, you have to set several Spark properties, as shown below, before a Spark context is created.<\/p>\n Once these properties are set, you will be able to import geoanalytics in your Spark session and authorize the module using a username and password, a license file, or an API key. Next, initialize a Glue session:<\/p>\n We will retrieve a JSON credentials file stored in our S3 bucket and then use those credentials to authenticate the geoanalytics module.<\/p>\n Alternatively, if you want to authenticate the module using a license file stored in your S3 bucket:<\/p>\n You can also use an API key for an active GeoAnalytics Engine subscription \u2013 see here<\/a>.<\/p>\n Let\u2019s take a look at how you can use AWS Glue and ArcGIS GeoAnalytics Engine together to tackle your analytics problems.\u00a0 We\u2019ll do this with an example exploring real-time streaming of flight data.<\/p>\n Real-time streaming data needs to be processed instantly or on schedule for optimal decision-making, efficiency, cost savings, ensuring safety and security. FlightAware<\/a> is a leading provider of real-time flight tracking data. Thanks to the integration of ArcGIS Velocity<\/a> and FlightAware, organizations can now bring live, global aircraft positions (per second) directly into ArcGIS Online<\/a> as feature layer, as well as into big data storages like S3 as formats like parquet.<\/p>\n Let\u2019s read FlightAware\u2019s flight position data (over 200 million records) using data exported periodically from ArcGIS Velocity to our S3 bucket.\u00a0 We read this into a Spark DataFrame in a Glue notebook. Using GeoAnalytics Engine’s ST-like SQL functions<\/a>, create point geometries and filter the data to include only flights inbound to Ronald Reagan Washington National Airport.<\/p>\n Next, using ReconstructTracks<\/a> tool, connect all time-sequential points for each flight into tracks and summarize records within each track.<\/p>\n You can export the tracks<\/em> DataFrame to ArcGIS Online as a hosted feature layer and create a flight track app to visually explore durations of each flight (Figure 1). These near-real-time flight tracks can be further analyzed alongside weather data for hazard detection and alerting. In the event of an aircraft incident or near-miss, investigators can reconstruct flight paths to gain insights into the circumstances surrounding the event.<\/p>\n Figure 1. Daily tracks and duration of flights (Feb 6-9, 2025) to Ronald Reagan Washington National Airport (Code: DCA). Data source: FlightAware<\/a><\/p>\n By analyzing real-time flight tracking and historical flight data, geo-enriched with other context-aware information such as weather events, and gaining predictive insights, companies can identify problem areas such as delays, flight incidents, route inefficiencies, or recurring issues with specific airports or airlines. These insights enable proactive decision-making, optimizing operations, improving customer experiences, streamlining supply chain management, and more.<\/p>\n As your streaming and historical datasets continuously flow in, you can execute this notebook workflow on-demand and also based on a schedule using AWS Glue triggers<\/a>. This allows for easy and reliable time-based job scheduling (e.g., every hour, day, week, or month), making it more efficient to automate data workflows and reducing the need for manual oversight or external scheduling systems.<\/p>\n This blog offers a step-by-step guide on setting up and executing ArcGIS GeoAnalytics Engine workflows in AWS Glue, emphasizing the advantages of utilizing both for big data ETL and spatial analytics jobs via an aviation industry use case using FlightAware Firehose\u2120 Flight Data Feed.<\/p>\n By leveraging AWS Glue’s fully managed, serverless environment, and the Apache Spark-based \u00a0functionalities of GeoAnalytics Engine, you can process and analyze large spatial datasets without the need for infrastructure management. Whether you\u2019re working with real-time streaming data, very large historical datasets spanning years to decades, or complex geospatial operations, this integrated solution streamlines your ETL processes, improves speed, all while providing robust tools for advanced spatial analysis.<\/p>\n We hope this guide inspires you to explore how GeoAnalytics Engine and AWS Glue can address your geospatial data challenges. For more details, check out the official documentation on GeoAnalytics Engine<\/a> and AWS Glue.<\/a> We\u2019d love to hear how you’re using these tools in your own projects and workflows\u2014feel free to share your experiences with us!<\/p>\n WARNING:<\/strong> The example code in this blog demonstrates concepts only. Production deployments should undergo comprehensive security reviews and follow your organizations security policies and compliance requirements.<\/p>\n"}],"related_articles":"","show_article_image":false,"card_image":false,"wide_image":false},"yoast_head":"\nWhy Integrate ArcGIS GeoAnalytics Engine with AWS Glue?<\/h2>\n
Getting Started in AWS Glue<\/h2>\n
\n
\n
Configure Spark Environment<\/h2>\n
\n
%glue_version x.x\r\n%number_of_workers 5\r\n%worker_type G.1X<\/pre>\n<\/blockquote>\n
Import and Configure GeoAnalytics Engine Distribution<\/h2>\n
\n
%extra_jars s3:\/\/ga-engine\/Install_GeoAnalytics\/geoanalytics_2.12-1.6.0.jar\r\n%extra_py_files s3:\/\/ga-engine\/Install_GeoAnalytics\/geoanalytics-1.6.0-py3-none-any.whl<\/pre>\n<\/blockquote>\n
\n
%%configure\r\n{\u00a0\u00a0 \"--conf\": \"spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.kryo.registrator=com.esri.geoanalytics.KryoRegistrator --conf spark.plugins=com.esri.geoanalytics.Plugin\"}<\/pre>\n<\/blockquote>\n\n
import sys\r\nfrom awsglue.transforms import *\r\nfrom awsglue.utils import getResolvedOptions\r\nfrom awsglue.context import GlueContext\r\nfrom awsglue.job import Job\r\n\r\nfrom pyspark.context import SparkContext\r\nsc = SparkContext.getOrCreate()\r\nglueContext = GlueContext(sc)\r\nspark = glueContext.spark_session\r\njob = Job(glueContext)<\/pre>\n<\/blockquote>\n
Import geoanalytics<\/em> library and authenticate<\/h3>\n
\n
import geoanalytics\r\nimport json\r\nimport boto3<\/pre>\n
# Initialize S3 client<\/em>\r\ns3_client = boto3.client('s3')\r\nbucket_name = 'your-s3-bucket-name'\r\nfile_key = 'gae_credentials.json'\r\n\r\nresponse = s3_client.get_object(Bucket=bucket_name, Key=file_key)\r\ncreds_json = json.loads(response['Body'].read().decode('utf-8'))\r\n\r\n# Authenticate<\/em>\r\ngeoanalytics.auth(username = creds_json['geoanalytics']['username'], \\\r\npassword = creds_json['geoanalytics']['password'])<\/pre>\n<\/blockquote>\n\n
geoanalytics.auth(license_file=\"s3:\/\/your-secure-bucket\/path\/to\/license.ecp\")<\/pre>\n<\/blockquote>\n
Use Case: Analyzing Real-Time Flight Data<\/h2>\n
\n
path = \"s3:\/\/ga-engine\/Datasets\/FlightAware\/feed\/*.parquet\"\r\ndf = spark.read.format(\"parquet\").load(path)\r\ndf_dca = (df\r\n .withColumn(\"point\", ST.transform(ST.point(\"lon\", \"lat\", 4326), 8857))\r\n .filter(df.dest == \"KDCA\")\r\n .filter(\"point IS NOT NULL\")\r\n .select(\"id\", \"aircrafttype\", \"orig\", \"dest\", \"ident\", \"heading\", \"point\", \"clock\", \"alt\"))<\/pre>\n<\/blockquote>\n
\n
from geoanalytics.tools import ReconstructTracks\r\ntracks = ReconstructTracks() \\\r\n .setTrackFields(\"id\") \\\r\n .setDistanceMethod(distance_method=\"Planar\") \\\r\n .addSummaryField(summary_field=\"alt\", statistic=\"Min\") \\\r\n .addSummaryField(summary_field=\"alt\", statistic=\"Max\") \\\r\n .addSummaryField(summary_field=\"alt\", statistic=\"Mean\",\r\nalias=\"avg_alt\") \\\r\n .run(dataframe=df1) \\\r\n .show(5)<\/pre>\n<\/blockquote>\n"},{"acf_fc_layout":"image","image":{"ID":2789502,"id":2789502,"title":"Picture4","filename":"Picture4.png","filesize":65266,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture4.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/geoanalytics-engine\/analytics\/solving-big-data-geoanalytics-challenges-with-arcgis-geoanalytics-engine-in-aws-glue\/picture4-68","alt":"","author":"323502","description":"","caption":"","name":"picture4-68","status":"inherit","uploaded_to":2789462,"date":"2025-05-10 21:27:13","modified":"2025-05-10 21:27:13","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":624,"height":108,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture4-213x108.png","thumbnail-width":213,"thumbnail-height":108,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture4.png","medium-width":464,"medium-height":80,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture4.png","medium_large-width":624,"medium_large-height":108,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture4.png","large-width":624,"large-height":108,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture4.png","1536x1536-width":624,"1536x1536-height":108,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture4.png","2048x2048-width":624,"2048x2048-height":108,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture4.png","card_image-width":624,"card_image-height":108,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/Picture4.png","wide_image-width":624,"wide_image-height":108}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"Export flight tracks to ArcGIS Online and visualize<\/h3>\n
\n
tracks.write.format(\"feature-service\")\u00a0 \\\r\n .option(\"gis\", \"myGIS\") \\\r\n .option(\"serviceName\", \"flightAware_DCA_Tracks\") \\\r\n .option(\"layerName\", \"flightAware_DCA_Tracks\")\u00a0 \\\r\n .save()<\/pre>\n<\/blockquote>\n"},{"acf_fc_layout":"image","image":{"ID":2789552,"id":2789552,"title":"flight tracks animation","filename":"flight-tracks-animation.gif","filesize":944339,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/flight-tracks-animation.gif","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/geoanalytics-engine\/analytics\/solving-big-data-geoanalytics-challenges-with-arcgis-geoanalytics-engine-in-aws-glue\/flight-tracks-animation","alt":"","author":"323502","description":"","caption":"","name":"flight-tracks-animation","status":"inherit","uploaded_to":2789462,"date":"2025-05-11 01:49:03","modified":"2025-05-11 01:49:03","menu_order":0,"mime_type":"image\/gif","type":"image","subtype":"gif","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1392,"height":772,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/flight-tracks-animation-213x200.gif","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/flight-tracks-animation.gif","medium-width":464,"medium-height":257,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/flight-tracks-animation.gif","medium_large-width":768,"medium_large-height":426,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/flight-tracks-animation.gif","large-width":1392,"large-height":772,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/flight-tracks-animation.gif","1536x1536-width":1392,"1536x1536-height":772,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/flight-tracks-animation.gif","2048x2048-width":1392,"2048x2048-height":772,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/flight-tracks-animation-826x458.gif","card_image-width":826,"card_image-height":458,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/05\/flight-tracks-animation.gif","wide_image-width":1392,"wide_image-height":772}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"Automating the Workflow with AWS Glue Triggers<\/h2>\n
Conclusion: Unlocking Scalable Geospatial Intelligence<\/h2>\n