"},{"ID":6831,"user_firstname":"Noah","user_lastname":"Slocum","nickname":"noahmead","user_nicename":"noahmead","display_name":"Noah Slocum","user_email":"NSlocum@esri.com","user_url":"","user_registered":"2018-03-02 00:18:50","user_description":"I am a product engineer on the GeoAnalytics team at Esri in Redlands, CA","user_avatar":"
"}],"short_description":"Learn how you can perform big data analysis using ArcGIS GeoAnalytics Engine.","flexible_content":[{"acf_fc_layout":"content","content":"ArcGIS <\/span>GeoAnalytics<\/span> Engine<\/span><\/a>\u00a0extends<\/span> Apache Spark <\/span>with<\/span> a collection of <\/span>over 140 <\/span>spatial SQL functions and spatial analysis tools. <\/span>It is<\/span> a<\/span> powerful <\/span>python<\/span> library<\/span> t<\/span>hat <\/span>allow<\/span>s<\/span> you to <\/span>carry out your spatial analysis<\/a> workflows within your Cloud platform<\/span><\/span>s<\/span><\/span><\/span> or on-premises<\/span> Spark<\/span> infrastructure.<\/span><\/span>\u00a0<\/span><\/p>\n In the demo for this year\u2019s <\/span><\/span>Developer Summit <\/span>p<\/span>lenary<\/span>, we <\/span>leverage<\/span> GeoAnalytics<\/span> Engine within an \u202fAzure Databricks<\/a><\/span>\u00a0<\/span><\/span>hosted<\/span> <\/span>PySpark<\/span><\/span><\/span> notebook to perform custom big data analysis.<\/span><\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"kaltura","video_id":"1_ajqnojd3","time":true,"start":"1726","stop":"1992"},{"acf_fc_layout":"content","content":" This demo <\/span>identif<\/span>ies<\/span> potential sites for new stores<\/span> by<\/span> investigat<\/span>ing<\/span> changes in consumer spending at brick-and-mortar stores throughout the United States<\/span>.<\/span> <\/span>We <\/span>used <\/span>the <\/span>Spend datase<\/span>t<\/span> from Esri partner <\/span>SafeGraph<\/span><\/a>. <\/span>SafeGraph<\/span> Spend<\/span><\/a> <\/span>i<\/span>s<\/span> a <\/span>points of interest (POIs)<\/span> dataset<\/span> provid<\/span>ing<\/span><\/span> anonymized, permissioned, and aggregated <\/span><\/span>credit and debit <\/span>transaction <\/span>information<\/span>. It<\/span> includes <\/span>date, time, location,<\/span> details of businesses across the United States, as well as consumer spe<\/span>n<\/span>ding metrics <\/span>for online and in-person<\/span> transactions<\/span>.<\/span> By<\/span><\/span> joining <\/span>spending data<\/span> to<\/span> <\/span>Census Block groups<\/span>, we can <\/span>gain<\/span> a<\/span>n <\/span><\/span>understanding of<\/span> how <\/span>consumer spending behavior<\/span>s change<\/span> in different region<\/span>s<\/span> across <\/span>the United States<\/span>.<\/span><\/span>\u00a0<\/span><\/p>\n Prepare data and workspace<\/span><\/b>\u00a0<\/span><\/p>\n We used the following workflow to prepare the data and workspace:<\/span>\u00a0<\/span><\/p>\n The Spend dataset used in this demo has 49 million spending records from January 1<\/span>st<\/span>, 2019 to January 1<\/span>st<\/span>, 2023. We can visualize the spatial distribution of the spending records by aggregating the data into bins.<\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1884642,"id":1884642,"title":"Figure 1","filename":"fig1.png","filesize":351282,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig1.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/geoanalytics-engine\/analytics\/dev-summit-2023-in-person-spending-analysis-with-geoanalytics-engine\/fig1-7","alt":"","author":"335752","description":"","caption":"Figure 1 Spatial distribution of the SafeGraph spending records across the Continental US ","name":"fig1-7","status":"inherit","uploaded_to":1884632,"date":"2023-03-23 02:09:02","modified":"2023-03-23 02:11:05","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":683,"height":429,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig1-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig1.png","medium-width":416,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig1.png","medium_large-width":683,"medium_large-height":429,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig1.png","large-width":683,"large-height":429,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig1.png","1536x1536-width":683,"1536x1536-height":429,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig1.png","2048x2048-width":683,"2048x2048-height":429,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig1.png","card_image-width":683,"card_image-height":429,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig1.png","wide_image-width":683,"wide_image-height":429}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" Analyzing the change in in-person spending with US census block groups\u00a0<\/span><\/b>\u00a0<\/span><\/p>\n To analyze the change in in-person spending at the block group level, we load the US census block group dataset<\/span>\u00a0from the Living Atlas into a PySpark DataFrame and then aggregate the spending records into block groups by calculating the total amount of in-person spending per year from 2019 to 2022. The table is pivoted for in-person spending in 2019 and 2022, and then we calculate the percent change in spending from 2019 to 2022.<\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"content","content":" We<\/span> next<\/span> remove<\/span> potential<\/span> outliers<\/span>. To <\/span>find<\/span> the outliers, we <\/span>calculate<\/span> the<\/span> interquartile range <\/span>(IQR)<\/span> to<\/span> find<\/span> the difference between <\/span>the <\/span>third<\/span> and <\/span>first<\/span> quartiles<\/span> (Q3<\/span> and Q1<\/span>) of<\/span> the dataset<\/span>. Any spending record<\/span> with change<\/span> that fa<\/span>l<\/span>ls below the lower bound (Q1 \u2013 1.5IQR) or <\/span>a<\/span>bove the upper bound (Q3 + 1.5IQR) is considered an outlier and removed from the spending dataset. Then w<\/span>e v<\/span>isualize the spatial pattern of the<\/span> in-person<\/span> spending <\/span>change<\/span> at <\/span>the <\/span>census <\/span>block<\/span> group<\/span> level<\/span>.<\/span><\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1884792,"id":1884792,"title":"Figure 2","filename":"fig2.png","filesize":300712,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig2.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/geoanalytics-engine\/analytics\/dev-summit-2023-in-person-spending-analysis-with-geoanalytics-engine\/fig2-10","alt":"","author":"335752","description":"","caption":"Figure 2 Spatial pattern for the percent change in in-person spending at census block group level","name":"fig2-10","status":"inherit","uploaded_to":1884632,"date":"2023-03-23 03:02:45","modified":"2023-03-23 03:03:18","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":684,"height":429,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig2-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig2.png","medium-width":416,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig2.png","medium_large-width":684,"medium_large-height":429,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig2.png","large-width":684,"large-height":429,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig2.png","1536x1536-width":684,"1536x1536-height":429,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig2.png","2048x2048-width":684,"2048x2048-height":429,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig2.png","card_image-width":684,"card_image-height":429,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig2.png","wide_image-width":684,"wide_image-height":429}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" In this demo, we deployed the cluster with 10 workers running in parallel, which <\/span>has<\/span> 352 cores and 2816GB executor memory in total. It takes about 30 seconds to join the 49 million spending <\/span><\/span>records<\/span><\/span><\/span> within <\/span>240,000<\/span> block group polygons<\/span>. <\/span>We also tested with more data, to <\/span>ensure this scaled. We <\/span>simulated<\/span> 1.5 billion<\/span> spending records. <\/span>Joining 1.5 billion spending records with 240,000 block groups completed on the same environment <\/span>within 6 minutes.<\/span><\/span>\u00a0<\/span><\/p>\n Finding sites for new stores using the Find Similar Locations tool<\/span><\/b>\u00a0<\/span><\/p>\n As shown in Figure 2, in-person spending data is only available for some census block groups, but how can we evaluate every census block group to select suitable sites for a new store?\u00a0<\/span>\u00a0<\/span><\/p>\n To answer that, we use the Find Similar Locations<\/a> tool to find census block groups most similar to the block groups with the largest increase in in-person spending since 2020. The Find Similar locations tool requires a reference DataFrame, which represents what we want to match to. We create this by selecting ten top-performing census block groups, which are the 10 block groups with the largest increase in in-person spending after removing outliers.\u00a0 Second, we prepare both the reference and search DataFrames by choosing 40 economic and demographic variables relating to population, household, industry and income (e.g., Population Density, Average Household Size, Wholesale Trade, and Median Disposable Income) from the census block group feature service layer. To look for locations in the search DataFrame with similar demographics to the ten reference records, we run the Find Similar Locations tool to calculate cosine similarity (ranges from 1.0 for perfect similarity to -1.0 for perfect dissimilarity) among the census block groups. Figure 3 shows an example map of the potential block groups near Indianapolis, Indiana<\/span>\u00a0with cosine similarity index larger than 0.8.<\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"content","content":" Conclusion<\/span><\/b>\u00a0<\/span><\/p>\n This demo showcases the power and versatility of the ArcGIS GeoAnalytics Engine for spatial analysis. By leveraging the SafeGraph Spend dataset and the US census block group dataset, we were able to gain insights into in-person spending patterns and identify potential sites for new stores using the Find Similar Locations tool. With the ability to process large amounts of spatial data quickly and efficiently, GeoAnalytics Engine can help businesses and organizations looking to make data-driven decisions. Refer to this <\/span>PySpark<\/span>\n
# Aggregate the spending records to US census block groups\r\nsg_spend_blocks = AggregatePoints() \\\r\n .setPolygons(block_groups) \\ \r\n .setTimeStep(interval_duration=1, interval_unit=\"years\") \\ \r\n .addSummaryField(\"in_person_spend\", \"Sum\", alias=\"sum_inperson_spend\") \\ \r\n .run(sg_spend)\r\n\r\n# Calculate the percent change in in-person spending from 2019 to 2022\r\nspend_blocks_pivoted = sg_spend_blocks.filter(F.col(\"COUNT\") > 10).groupBy(blocks_fields) \\ \r\n .pivot(\"step_start\").agg(F.first(\"sum_inperson_spend\").alias(\"inperson\")) \\ \r\n .select(blocks_fields + [\"2019-01-01 00:00:00\", \"2022-01-01 00:00:00\"]) \\ \r\n .withColumn(\"inperson_pct_19_22\", F.round((F.col(\"inperson_spend_2022\") - F.col(\"2022-01-01 00:00:00\"))\/F.col(\"2019-01-01 00:00:00\") *100, 2)) \r\n<\/code><\/pre>\n"},{"acf_fc_layout":"content","content":"# Select the 10 block groups with the highest averaged increase in spending from 2019 to 2022\u00a0\r\ntop10_blocks = spend_blocks_cleaned.orderBy(F.col(\"inperson_pct_19_22\").desc()).limit(10)\u00a0\r\n\r\n# Prepare the reference and search dataframes for similarity search\u00a0\r\nblocks_df = block_groups.select([F.col(item) for item in blocks_fields])\u00a0\r\ntop10_df = blocks_df.join(top10_blocks, [\"ID\"], \"inner\").select(blocks_df[\"*\"])\u00a0\r\n\r\n# Run Find Similar Locations tool for site selection, and get the similarity score\u00a0\r\nfsl_blocks = FindSimilarLocations() \\\u00a0\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .setAnalysisFields(*analysis_fields) \\\u00a0\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .setMostOrLeastSimilar(most_or_least_similar=\"MostSimilar\") \\\u00a0\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .setMatchMethod(match_method=\"AttributeProfiles\") \\\u00a0\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .setNumberOfResults(number_of_results=search_df.count()) \\\u00a0\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .setAppendFields(\"ObjectID\", \"Shape\", \"ID\", \"NAME\", \"ST_ABBREV\", \"STATE_NAME\") \\\u00a0\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 .run(reference_dataframe= top10_df, search_dataframe= blocks_df)\r\n<\/code><\/pre>\n"},{"acf_fc_layout":"image","image":{"ID":1884812,"id":1884812,"title":"Figure3","filename":"fig3.png","filesize":128098,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig3.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/geoanalytics-engine\/analytics\/dev-summit-2023-in-person-spending-analysis-with-geoanalytics-engine\/fig3-9","alt":"","author":"335752","description":"","caption":"Figure 3 Block group samples most similar to top-performing block groups in Indiana region","name":"fig3-9","status":"inherit","uploaded_to":1884632,"date":"2023-03-23 03:07:22","modified":"2023-03-23 03:07:41","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":684,"height":496,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig3-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig3.png","medium-width":360,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig3.png","medium_large-width":684,"medium_large-height":496,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig3.png","large-width":684,"large-height":496,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig3.png","1536x1536-width":684,"1536x1536-height":496,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig3.png","2048x2048-width":684,"2048x2048-height":496,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig3-641x465.png","card_image-width":641,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/fig3.png","wide_image-width":684,"wide_image-height":496}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"