{"id":545492,"date":"2022-11-01T19:59:16","date_gmt":"2022-11-02T02:59:16","guid":{"rendered":"https:\/\/www.esri.com\/about\/newsroom\/?post_type=arcnews&#038;p=545492"},"modified":"2023-12-11T16:36:06","modified_gmt":"2023-12-12T00:36:06","slug":"bringing-spatial-analysis-of-big-data-to-the-cloud","status":"publish","type":"arcnews","link":"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud","title":{"rendered":"Bringing Spatial Analysis of Big Data to the Cloud"},"author":5752,"featured_media":0,"menu_order":0,"template":"","format":"standard","meta":{"_acf_changed":false,"sync_status":"","episode_type":"","audio_file":"","podmotor_file_id":"","podmotor_episode_id":"","castos_file_data":"","cover_image":"","cover_image_id":"","duration":"","filesize":"","filesize_raw":"","date_recorded":"","explicit":"","block":"","itunes_episode_number":"","itunes_title":"","itunes_season_number":"","itunes_episode_type":"","_links_to":"","_links_to_target":""},"categories":[481962,369842,23392],"tags":[178702,871,170502,278902,281],"arcnews_issues":[481982],"class_list":["post-545492","arcnews","type-arcnews","status-publish","format-standard","hentry","category-arcgis-geoanalytics-engine","category-big-data-advanced-analytics","category-cloud-computing","tag-analysis","tag-big-data","tag-cloud","tag-data-science","tag-location-intelligence","arcnews_issues-fall-2022","arcnews_sections-esri-technology"],"acf":{"short_description":"With ArcGIS GeoAnalytics Engine, it takes minutes\u2014rather than hours or days\u2014to process massive volumes of spatial data.","pdf":{"host_remotely":false,"file":"","file_url":""},"flexible_content":[{"acf_fc_layout":"content","content":"Because of its size, big data can be difficult to store and complex to process using traditional data processing software. Rather than migrating big data to specialized computing environments, organizations typically store and analyze this data in managed clouds.\r\n\r\nArcGIS GeoAnalytics Engine brings the power of Esri\u2019s <a href=\"https:\/\/www.esri.com\/en-us\/capabilities\/spatial-analytics-data-science\/overview\">spatial analytics<\/a> capabilities to where organizations\u2019 cloud-based big data lives: in data lakes, data warehouses, and databases. Supported cloud environments include Microsoft Azure Synapse Analytics, Amazon EMR, and Google Cloud Dataproc."},{"acf_fc_layout":"image","image":544572,"image_position":"right","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"Data scientists and GIS analysts access GeoAnalytics Engine directly from within Apache Spark, the large-scale data processing engine that\u2019s designed for big data analysis. This makes performing spatial analysis on big data faster and more efficient while going well beyond the basics."},{"acf_fc_layout":"content","content":"<h2>Conducting Analysis Where Data Is Stored<\/h2>\r\nIn the past, data had to be moved to where analytics was accessible, usually in specialized analysis environments. But migrating massive data is cost prohibitive and time-consuming and creates data silos.\r\n\r\nThis is primarily why data scientists adopted Spark\u2014an open-source analytics engine used to process large amounts of data\u2014as their big data environment of choice. It employs cluster computing to increase big data processing speeds while hosting various libraries of analytic functions that are delivered directly to data where it is stored.\r\n\r\nGeoAnalytics Engine is native to Spark, so it leverages Spark\u2019s computing power while rapidly processing massive volumes of spatial data. Without GeoAnalytics Engine, processing big datasets can take hours or even days. But benchmark testing done by Esri shows that the performance of GeoAnalytics Engine is 10 to 100 times faster than other open-source spatial analysis options."},{"acf_fc_layout":"content","content":"<h2>Processing 16 Billion Records in Five Minutes<\/h2>\r\nGovernment agencies and commercial organizations often work with tens of billions of records to gain actionable intelligence from data. Cellular network coverage data, for example, is huge and can reveal a wealth of information if the right spatial analytics is applied to it.\r\n\r\nReal-world uses of anonymized cell coverage data include determining where mobile networks have satisfactory or unsatisfactory coverage and finding out how many people lingered at a specific site for a particular amount of time. Cell Analytics, from Esri partner Ookla, collects big data on how cell networks around the world are performing each day. Taking a dataset of about 16 billion depersonalized records from Cell Analytics (the cellular coverage dataset from Speedtest), a team of data scientists at Esri used the Find Hot Spots and Find Dwell Locations tools in GeoAnalytics Engine to identify patterns of cell signal strength and human presence and mobility. It took the team less than five minutes to extract, transform, load, and analyze 16 billion records. The team was then able to quickly build interactive dashboards, web and mobile apps, map-based stories, and analytical models to share actionable information with stakeholders.\r\n\r\nIn this scenario, if the data scientists had used traditional spatial analysis packages, they would have needed to geospatially index the data, which takes a significant amount of time. GeoAnalytics Engine enables users to skip that step and employ geospatial data immediately, streamlining the process of getting from raw data to actionable results.\r\n\r\nThis means that data analysis can begin right away. Users are able to focus on supporting the mission at hand rather than losing valuable time on moving and preparing data. And once generated, analysis results are easy to communicate so that stakeholders can act."},{"acf_fc_layout":"image","image":544582,"image_position":"left","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"<h2>Seeing the Full Picture<\/h2>\r\nGeoAnalytics Engine enables users to create comprehensive analyses of specific situations. It has a library of more than 120 functions and analysis tools\u2014ranging from simple transformation and spatial aggregation tools to advanced statistical algorithms that aren\u2019t available in open-source packages\u2014in a standard big data analysis workflow. Thus, data scientists and GIS analysts no longer have to patch together spatial analysis packages to get the full picture of a situation.\r\n\r\nTo conduct a full-picture analysis with GeoAnalytics Engine, data scientists at Esri obtained public information from the City of New York\u2019s open data website to see where noise complaints occur in high numbers. City officials could use the results of an analysis like this to identify where more noise-tampering resources need to be deployed.\r\n\r\nIn New York, residents can call or send a message to the city\u2019s 311 customer service center to make noise complaints (and access other nonemergency city services). The Esri team obtained 27 million noise complaint records for a 10-year period to perform the analysis.\r\n\r\nIf team members had relied on traditional analytics to try to answer their primary question, they could have used the 311 data to determine whether noise complaints had increased, decreased, or stayed the same, but it would have been much more difficult to find out where and when the complaints had occurred and how long it had taken to respond to them. That\u2019s where spatial analysis comes in. Using GeoAnalytics Engine to process the data, the team generated a hexbin map to show clusters of 311 noise complaints along with their corresponding response times. Darker bins on the map reveal areas where it took longer for city officials to respond to noise complaints, suggesting less efficient 311 service."},{"acf_fc_layout":"content","content":"<h2>Continuing to Evolve Big Data Spatial Analytics<\/h2>\r\nAs organizations obtain ever-larger volumes of spatial data that need to be processed and analyzed, the capabilities of GeoAnalytics Engine will only continue to grow. Future releases will focus on adding tools and functions, advancing how data comes into and is shared out of GeoAnalytics Engine, and enhancing visualization capabilities.\r\n\r\n<a href=\"https:\/\/www.esri.com\/en-us\/arcgis\/products\/arcgis-geoanalytics-engine\/overview?aduc=PublicRelations&amp;aduca=MIGeoAnalyticsEngine&amp;aduco=arcnews-fall&amp;adum=Blog&amp;sf_id=7015x000001XrJWAA0&amp;utm_source=PublicRelations\">Get started with ArcGIS GeoAnalytics Engine<\/a>."}],"references":null},"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Bringing Spatial Analysis of Big Data to the Cloud | ArcNews | Fall 2022<\/title>\n<meta name=\"description\" content=\"With ArcGIS GeoAnalytics Engine, it takes minutes\u2014rather than hours or days\u2014to process massive volumes of spatial data.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Bringing Spatial Analysis of Big Data to the Cloud | ArcNews | Fall 2022\" \/>\n<meta property=\"og:description\" content=\"With ArcGIS GeoAnalytics Engine, it takes minutes\u2014rather than hours or days\u2014to process massive volumes of spatial data.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud\" \/>\n<meta property=\"og:site_name\" content=\"Esri\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/esrigis\/\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-12T00:36:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.esri.com\/about\/newsroom\/app\/uploads\/2022\/10\/arcnews-banner-bringingspatial-wide.jpg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Bringing Spatial Analysis of Big Data to the Cloud | ArcNews | Fall 2022\" \/>\n<meta name=\"twitter:description\" content=\"With ArcGIS GeoAnalytics Engine, it takes minutes\u2014rather than hours or days\u2014to process massive volumes of spatial data.\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.esri.com\/about\/newsroom\/app\/uploads\/2022\/10\/arcnews-banner-bringingspatial-wide.jpg\" \/>\n<meta name=\"twitter:site\" content=\"@Esri\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\n\t    \"@context\": \"https:\/\/schema.org\",\n\t    \"@graph\": [\n\t        {\n\t            \"@type\": \"WebPage\",\n\t            \"@id\": \"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud\",\n\t            \"url\": \"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud\",\n\t            \"name\": \"Bringing Spatial Analysis of Big Data to the Cloud | ArcNews | Fall 2022\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\/\/www.esri.com\/about\/newsroom\/#website\"\n\t            },\n\t            \"datePublished\": \"2022-11-02T02:59:16+00:00\",\n\t            \"dateModified\": \"2023-12-12T00:36:06+00:00\",\n\t            \"description\": \"With ArcGIS GeoAnalytics Engine, it takes minutes\u2014rather than hours or days\u2014to process massive volumes of spatial data.\",\n\t            \"breadcrumb\": {\n\t                \"@id\": \"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud#breadcrumb\"\n\t            },\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"ReadAction\",\n\t                    \"target\": [\n\t                        \"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"BreadcrumbList\",\n\t            \"@id\": \"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud#breadcrumb\",\n\t            \"itemListElement\": [\n\t                {\n\t                    \"@type\": \"ListItem\",\n\t                    \"position\": 1,\n\t                    \"name\": \"Home\",\n\t                    \"item\": \"https:\/\/www.esri.com\/about\/newsroom\"\n\t                },\n\t                {\n\t                    \"@type\": \"ListItem\",\n\t                    \"position\": 2,\n\t                    \"name\": \"ArcNews Articles\",\n\t                    \"item\": \"https:\/\/www.esri.com\/about\/newsroom\/arcnews\"\n\t                },\n\t                {\n\t                    \"@type\": \"ListItem\",\n\t                    \"position\": 3,\n\t                    \"name\": \"Bringing Spatial Analysis of Big Data to the Cloud\"\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"WebSite\",\n\t            \"@id\": \"https:\/\/www.esri.com\/about\/newsroom\/#website\",\n\t            \"url\": \"https:\/\/www.esri.com\/about\/newsroom\/\",\n\t            \"name\": \"Esri\",\n\t            \"description\": \"Esri Newsroom\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"SearchAction\",\n\t                    \"target\": {\n\t                        \"@type\": \"EntryPoint\",\n\t                        \"urlTemplate\": \"https:\/\/www.esri.com\/about\/newsroom\/?s={search_term_string}\"\n\t                    },\n\t                    \"query-input\": {\n\t                        \"@type\": \"PropertyValueSpecification\",\n\t                        \"valueRequired\": true,\n\t                        \"valueName\": \"search_term_string\"\n\t                    }\n\t                }\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"Person\",\n\t            \"@id\": \"https:\/\/www.esri.com\/about\/newsroom\/#\/schema\/person\/41c803b2ea8734c36f9c4e9586d1449d\",\n\t            \"name\": \"Amy Ambard\",\n\t            \"image\": {\n\t                \"@type\": \"ImageObject\",\n\t                \"inLanguage\": \"en-US\",\n\t                \"@id\": \"https:\/\/www.esri.com\/about\/newsroom\/#\/schema\/person\/image\/\",\n\t                \"url\": \"https:\/\/secure.gravatar.com\/avatar\/f356480172f8ad0bc8d72b855e84171c52f1944c7c7779f3e425d73bf3efa3c7?s=96&d=blank&r=g\",\n\t                \"contentUrl\": \"https:\/\/secure.gravatar.com\/avatar\/f356480172f8ad0bc8d72b855e84171c52f1944c7c7779f3e425d73bf3efa3c7?s=96&d=blank&r=g\",\n\t                \"caption\": \"Amy Ambard\"\n\t            },\n\t            \"url\": \"\"\n\t        }\n\t    ]\n\t}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Bringing Spatial Analysis of Big Data to the Cloud | ArcNews | Fall 2022","description":"With ArcGIS GeoAnalytics Engine, it takes minutes\u2014rather than hours or days\u2014to process massive volumes of spatial data.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud","og_locale":"en_US","og_type":"article","og_title":"Bringing Spatial Analysis of Big Data to the Cloud | ArcNews | Fall 2022","og_description":"With ArcGIS GeoAnalytics Engine, it takes minutes\u2014rather than hours or days\u2014to process massive volumes of spatial data.","og_url":"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud","og_site_name":"Esri","article_publisher":"https:\/\/www.facebook.com\/esrigis\/","article_modified_time":"2023-12-12T00:36:06+00:00","og_image":[{"url":"https:\/\/www.esri.com\/about\/newsroom\/app\/uploads\/2022\/10\/arcnews-banner-bringingspatial-wide.jpg","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_title":"Bringing Spatial Analysis of Big Data to the Cloud | ArcNews | Fall 2022","twitter_description":"With ArcGIS GeoAnalytics Engine, it takes minutes\u2014rather than hours or days\u2014to process massive volumes of spatial data.","twitter_image":"https:\/\/www.esri.com\/about\/newsroom\/app\/uploads\/2022\/10\/arcnews-banner-bringingspatial-wide.jpg","twitter_site":"@Esri","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud","url":"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud","name":"Bringing Spatial Analysis of Big Data to the Cloud | ArcNews | Fall 2022","isPartOf":{"@id":"https:\/\/www.esri.com\/about\/newsroom\/#website"},"datePublished":"2022-11-02T02:59:16+00:00","dateModified":"2023-12-12T00:36:06+00:00","description":"With ArcGIS GeoAnalytics Engine, it takes minutes\u2014rather than hours or days\u2014to process massive volumes of spatial data.","breadcrumb":{"@id":"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.esri.com\/about\/newsroom\/arcnews\/bringing-spatial-analysis-of-big-data-to-the-cloud#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.esri.com\/about\/newsroom"},{"@type":"ListItem","position":2,"name":"ArcNews Articles","item":"https:\/\/www.esri.com\/about\/newsroom\/arcnews"},{"@type":"ListItem","position":3,"name":"Bringing Spatial Analysis of Big Data to the Cloud"}]},{"@type":"WebSite","@id":"https:\/\/www.esri.com\/about\/newsroom\/#website","url":"https:\/\/www.esri.com\/about\/newsroom\/","name":"Esri","description":"Esri Newsroom","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.esri.com\/about\/newsroom\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.esri.com\/about\/newsroom\/#\/schema\/person\/41c803b2ea8734c36f9c4e9586d1449d","name":"Amy Ambard","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.esri.com\/about\/newsroom\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/f356480172f8ad0bc8d72b855e84171c52f1944c7c7779f3e425d73bf3efa3c7?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f356480172f8ad0bc8d72b855e84171c52f1944c7c7779f3e425d73bf3efa3c7?s=96&d=blank&r=g","caption":"Amy Ambard"},"url":""}]}},"sort_order":"5","_links":{"self":[{"href":"https:\/\/www.esri.com\/about\/newsroom\/wp-json\/wp\/v2\/arcnews\/545492","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.esri.com\/about\/newsroom\/wp-json\/wp\/v2\/arcnews"}],"about":[{"href":"https:\/\/www.esri.com\/about\/newsroom\/wp-json\/wp\/v2\/types\/arcnews"}],"author":[{"embeddable":true,"href":"https:\/\/www.esri.com\/about\/newsroom\/wp-json\/wp\/v2\/users\/5752"}],"version-history":[{"count":0,"href":"https:\/\/www.esri.com\/about\/newsroom\/wp-json\/wp\/v2\/arcnews\/545492\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.esri.com\/about\/newsroom\/wp-json\/wp\/v2\/media?parent=545492"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.esri.com\/about\/newsroom\/wp-json\/wp\/v2\/categories?post=545492"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.esri.com\/about\/newsroom\/wp-json\/wp\/v2\/tags?post=545492"},{"taxonomy":"arcnews_issues","embeddable":true,"href":"https:\/\/www.esri.com\/about\/newsroom\/wp-json\/wp\/v2\/arcnews_issues?post=545492"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}