Streamline Data Integration with ArcGIS Data Pipelines

Data is essential for making informed decisions, but it can be a challenge to integrate and transform data into a usable format. For use cases where data preparation consumes a significant amount of time, when there are issues with cleaning and preparing data, or when datasets need to be combined from multiple sources, ArcGIS Data Pipelines can help with data integration and preparation.

Accessed from the app launcher in ArcGIS Online, Data Pipelines provides a new data integration capability in ArcGIS Online that makes it faster and easier to access, prepare, and maintain data for mapping, analysis, and reporting. It streamlines a user’s extract, transform, and load (ETL) workflow in ArcGIS Online by going beyond traditional ETL tools. With its drag-and-drop interface and task scheduling functionality, users can create data pipelines without writing any code and automatically keep data up-to-date. Data Pipelines enhances data-driven decision-making in ArcGIS Online and enables users to maximize the value of their geospatial data.

A flowchart with arrows shows the progression from four text boxes entitled Files, Cloud Storage, Cloud Data Warehouses, and ArcGIS Feature Layers through a text box that reads ArcGIS Data Pipelines and culminating in a box that reads ArcGIS Online. All six of the boxes are blue, with symbology relating to the box titles.
ArcGIS Data Pipelines provides data integration and data transformation capabilities for ArcGIS Online.

Benefits of Using Data Pipelines

By streamlining data integration for ArcGIS Online, Data Pipelines centralizes data from various sources into a single, accessible location. This saves time and effort so that users can focus on deriving valuable insights from the data rather than on data preparation and cleaning tasks.

Data Pipelines also expands access to data stores by supporting data from cloud-based data storage options such as Amazon S3, Azure Blob, Google BigQuery, and Snowflake. Users can leverage previously inaccessible or difficult-to-integrate data to enrich analysis and decision-making capabilities. They can also bring in files from their local drive or via a URL, or feature layers from ArcGIS Online. Access to a broader range of data sources can help foster data-driven decision-making across an organization and between organizations.

The application’s data transformation tools enable users to get data into an ideal state for mapping, analysis, and reporting. Users can clean, transform, and enrich data to ensure accuracy, consistency, and completeness.

An intuitive graphical interface eliminates the need for specialized coding skills by enabling users to build and manage data pipelines via a drag-and-drop authoring experience. This low-code approach allows users of all skill levels to construct and manage data pipelines.

Automation is another key advantage of Data Pipelines. Its scheduling functionality facilitates running data pipelines on a recurring basis, keeping datasets up-to-date and synchronized with the latest changes in the source data.

Data Pipelines offers a comprehensive set of benefits for ETL processes, including streamlined data integration, expanded data access, data transformation tools, democratization of data management, and workflow automation. By leveraging these capabilities, organizations can unlock more insights with their data, helping with data-driven decision-making and gaining a competitive edge in today’s datacentric landscape.

Key Use Cases

While Data Pipelines was in beta, organizations used the application to streamline data integration in exciting and innovative ways, such as the following:

A map of California, with some nearby states also shown, is labeled Table to Boundaries. Above the map is a flowchart showing progressions from Homelessness data and Boundary areas to Write results.
With Data Pipelines, users can geospatially enable a subset of tabular data using the Join tool and then write the results out to a feature layer.
A map of the Chicago area, with some nearby towns also shown, is labeled Prepare. Small yellow circles are scattered around the map. Above the map is a flowchart showing the progression from Connect to Integrate. On the right and left are toolbar columns.
Access, prepare, and maintain data for mapping, analysis, and reporting with Data Pipelines.

These are just a few of the workflows that ArcGIS Data Pipelines can help users achieve. By leveraging its capabilities, users can streamline data management processes and enhance geospatial workflows without writing a single line of code or relying on repetitive, manual efforts.

Getting Started with ArcGIS Data Pipelines

For organizations that are looking for a tool to streamline data management and integration workflows in ArcGIS Online, Data Pipelines is available now. It does not require an additional license. Instead, it consumes credits based on the amount of time spent working in the app.