New Ways to Access Big Data in the Cloud

As organizations require ever more storage for their data, data storage technologies need to adapt to provide adequate data retention and processing. Cloud data warehouses do this, making data such as point-of-sale information, telemetry data from sensors, and sales leads generated by websites easier to access and process. For Esri users, ArcGIS Pro 2.9 (and later) and ArcGIS Enterprise 10.9.1 (and later) offer support for connecting to cloud data warehouses and publishing that data.

The Advantages of Cloud Data Warehouses

Anyone who works with constant streams of data—whether cataloging sales transactions for a grocery store chain or tracking data produced by delivery truck fleets—needs a data storage solution that can keep up with enormous intakes of structured data.

Traditional databases may run into challenges when serving this data to a wide audience. If the data is stored on premises, an existing deployment would likely need to be scaled up, which comes with considerable cost. If a dataset is used as a source for an app that’s available worldwide, making the data accessible at that scale becomes a significant challenge.

An illustration of a satellite, an airplane, and a map with a signal coming out of it with arrows pointing from them to a cloud with cylinders in it. More arrows point from the cloud to a desktop computer that shows the ArcGIS Enterprise and ArcGIS Pro logos along with a map. Additional arrows point from the computer screen to a smaller computer screen with a map on it, a dashboard, and a mobile device with a map on it
When structured data from different sources is stored in a cloud data warehouse, users of ArcGIS Pro 2.9 (and later) and ArcGIS Enterprise 10.9.1 (and later) can publish this data as map image layers.

Cloud data warehouses offer several advantages over other forms of structured data storage, including the following:

How to Access Data Stored in a Cloud Data Warehouse

The ability to connect to and use data from cloud data warehouses was implemented in ArcGIS Pro 2.9 and ArcGIS Enterprise 10.9.1 on Windows, Linux, and Kubernetes. The software supports connections to three cloud data warehouses: Google BigQuery, Snowflake, and Amazon Redshift.

Adding this data to ArcGIS Pro is like adding data from any other database. One of the main challenges of working with cloud data warehouses, however, is access. There is a cost associated with accessing data stored in cloud data warehouses, which is something developers need to keep in mind when building web maps or apps that rely on this data.

To balance cost and access, data publishers can publish data from cloud data warehouses to ArcGIS Enterprise as map services. There are three ways to do this, based on how frequently users need to retrieve the data stored in cloud data warehouses:

Keeping Up with the Evolution of Big Data

As much of the world moves toward wider adoption of Internet of Things (IoT) technology and Web 3.0, the amount of data being produced will only continue to scale up. Cloud data warehouses are a widely adopted standard for users who work with an immense amount of structured data.

ArcGIS Pro 2.9 (and later) and ArcGIS Enterprise 10.9.1 (and later) offer several ways to access data in cloud data warehouses, allowing users to more easily—and cost-effectively—explore, visualize, and share large volumes of structured data. To learn more about how Esri supports connecting to cloud data warehouses, read the following blog posts on ArcGIS Blog: