ArcGIS Online

Introducing Data Pipelines in ArcGIS Online (beta release)

Is data preparation the most time-consuming part of your workflow? Do you have datasets you want to bring to, and use in ArcGIS? Is your data difficult to clean? If these problems sound familiar to you, keep reading to learn about the new data preparation and data ingest capability of ArcGIS Online: Data Pipelines.

What is Data Pipelines?

Data Pipelines is a new native data integration capability of ArcGIS Online that makes it faster and easier to access, prepare, and integrate data.

With Data Pipelines, you can:

Check out the video below to see Data Pipelines in action.

 

How do you build a Data Pipeline?

Data pipeline workflows consist of the following element types:

The image below shows an example workflow using the three elements:

  1. An input–In this case, a connection to a public Google BigQuery dataset for Citi Bike locations.
  2. A tool–This workflow uses the Create geometry tool to define the geometry for each location from coordinates.
  3. An output–In this example, writing the Citi Bike point locations to a feature layer.
The three elements of a data pipeline workflow: inputs, tools, and outputs.

You can also make a more complicated data pipeline, with multiple inputs, tools, and outputs.

Interactive data prep

Data Pipelines provides an interactive experience for investigating your data while building out your prep workflow. While working with your data, you may want to check that each step is completed as you expect. You can do this through the preview option. At each step, you can visualize your data in a table or a map to get a better understanding of how it has been processed, or identify any remaining processing steps that you need to do.

Data Pipelines interactive preview experience to investigate your data in a table, map, or schema view.

Previewing means you can easily identify any remaining steps in preparing your data. If you realize you’ve missed a step, you can update exiting tool parameters or add or delete tools in your diagram.

Get started with Data Pipelines

You’ve seen how you can now connect to an external data source, use a suite of data preparation tools, and save the results in ArcGIS Online. Data Pipelines takes one of the most challenging parts of your GIS workflows, and simplifies it by giving you the power to easily prepare and ingest your data. We hope that you feel empowered to try it out. To get started with Data Pipelines, check out the following resources:

More information

Data Pipelines consumes credits based on the amount of time the editor session is active (active means in a connecting or connected state). While the session is active, you can continuously preview and run your data pipeline workflow. To learn more about credits in ArcGIS Online, see the Understand credits topic. For more specific details on how and when Data Pipelines consumes credits, see the Data Pipelines FAQ on credit consumption.

For more information and additional details about Data Pipelines, see the Data Pipelines documentation. Consider checking out the FAQ topic to find answers to any specific questions. For any other questions or suggestions, please post in the Esri Community forum where one of the Data Pipelines team members will be happy to help you out.

If you’re interested in data preparation workflows in ArcGIS Pro, see the blog Explore and Prepare Your Data with ArcGIS Pro Data Engineering.

About the authors

Bethany (she/her) is a Product Engineer on the Data Pipelines team and the GeoAnalytics team. Her background is in biology and GIS with experience in data management and spatial-temporal analysis.

Connect:

Corinne is a Product Marketing Manager on Esri’s Spatial Analytics & Data Science team. She has a background in marketing and business analytics with experience working in the technology and geospatial industries.

Connect:
0 Comments
Inline Feedbacks
View all comments

Next Article

Basemap Releases Include Over 300 New and Updated Communities

Read this article