ArcGIS Data Pipelines is Now Available in ArcGIS Enterprise

Do you spend hours repeating the same data preparation workflow or struggle to automate it effectively? Does your organization rely on scripts that are difficult to create, maintain, or hand off to others?

If you are looking for a faster, more collaborative way to integrate and prepare data for your mapping and analytics workflows in ArcGIS Enterprise, this blog is for you.

Use the links below to jump to a specific section:

• ArcGIS Data Pipelines streamlines data integration workflows
• Save time and reduce complexity
• Connect to data
• Perform data processing
• Write prepared data out
• Automate data updates
• Licensing and deployment considerations
• More information

ArcGIS Data Pipelines streamlines data integration workflows

Data preparation can be a time-consuming part of any GIS workflow. However, ArcGIS Data Pipelines helps you move faster with a visual diagramming app that supports automation and runs natively in ArcGIS Enterprise.

ArcGIS Data Pipelines is a no-code, visual data engineering capability that streamlines data integration, preparation, and automation in ArcGIS. First introduced in ArcGIS Online and later released as a beta feature in ArcGIS Enterprise 12.0, it is now available in ArcGIS Enterprise 12.1 on Windows and Linux and is ready for production use.

Watch the video below to see ArcGIS Data Pipelines in action.

Learn about what’s new since the ArcGIS Enterprise 12.0 beta release of Data Pipelines.

Save time and reduce complexity

ArcGIS Data Pipelines is included with ArcGIS Enterprise Advanced. This gives you access to a purpose-built app for preparing and integrating data with ArcGIS Enterprise. It also supports automation, allowing you to schedule data pipelines to run on a regular interval without extra infrastructure overhead.

Unlike script-based workflows, Data Pipelines provides a visual representation of each step. As a result, you can more easily create, understand, and share workflows across your team. To validate your workflow, you can preview your data at every step, including attributes, geometry, and schema.

In addition, you can add notes directly to your data pipelines to document decisions, simplify long-term maintenance, and enhance cross-team collaboration.

Whether you need to combine disparate datasets, clean messy data, or transform files into feature layers for mapping and analysis, ArcGIS Data Pipelines helps you spend less time preparing data and more time generating insights.

Connect to data

Inputs are used to load data into your data pipeline for downstream processing, and there are many input source types available. ArcGIS Data Pipelines can connect to external data sources as well as data stored in your ArcGIS content.

While the list of supported sources will expand over time, the following input types are supported in ArcGIS Data Pipelines for ArcGIS Enterprise 12.1:

• Files uploaded to your content, accessed from a file share, or returned from a URL or API request
• Files stored in an Amazon S3 bucket or Microsoft Azure Storage container
• Tables from Google BigQuery or Snowflake, two cloud data warehouses
• Feature layers from ArcGIS, including both hosted feature layers and referenced layers

When you connect to files stored in a file share or a cloud storage container, you can bring in multiple files that share a common schema and format as a single dataset. This is especially useful when working with partitioned data.

Unlike scripts, which often require you to hard-code database credentials, Data Pipelines stands out. In ArcGIS Data Pipelines, all connection information for databases and APIs is encrypted and stored securely as either a data store item or service connection item, respectively.

Over time, more input sources will be made available, including a connector to Databricks and connectors to relational databases like PostgreSQL, SQL Server, and Oracle.

For more information about the available inputs, see Dataset configuration.

Perform data processing

ArcGIS Data Pipelines includes essential data preparation tools to help you build, blend, and format vector and tabular datasets.

In ArcGIS Enterprise 12.1, these tools are organized into the following categories:

• Clean: Filter by attribute, Filter by extent, Select fields, Remove duplicates, Simplify geometry, and Clip
• Construct: Calculate field, Create geometry, and Create date time
• Format: Update fields, Unnest field, Map fields, Project geometry, and Pivot
• Integrate: Join, Merge, Dissolve, and Summarize attributes

In the example below, two datasets are joined using a matching field. You can also join datasets based on spatial or temporal relationships.

Some of the most commonly used tools include:

• Select fields to limit the columns returned
• Filter by attribute to return only records that meet the condition of a query
• Unnest field to extract values stored in complex field types (map, struct, array)
• Calculate field to calculate values for a new or existing field using Arcade
• Create geometry to create a geometry field from one or more fields

Together, these tools allow you to blend and build datasets using batch processing and automation without writing code.

To learn more about each tool, refer to the Data processing help topic.

Write prepared data to your content

After you connect to your data and apply processing tools, you can configure one or more feature layer outputs. When you run a data pipeline, it processes the data and creates or updates layers in your content.

As shown below, the Feature layer output allows you to define how results are written. You can create a new feature layer, replace data in an existing layer, or add and update existing records.

All output feature layers from ArcGIS Data Pipelines leverage the hosting server’s relational store. To learn more about writing results, see the Feature layer output documentation.

In the future, additional outputs will be supported.

Automate data updates

Working with the latest data is critical, especially in systems like ArcGIS Enterprise that are being used to support decision making and daily business operations. With ArcGIS Data Pipelines, you can schedule your data pipelines to run automatically, keeping data up to date and reliable for use.

ArcGIS Data Pipelines allows you to create and manage scheduled tasks, automating data pipelines to run on an interval of your choosing. For example, if you create a data pipeline that ingests data from an API that updates every day at 6 p.m., you could schedule the data pipeline to run daily at 6:15 p.m.

Tasks can be created or updated from within the editor when a data pipeline is open. Or they can be managed centrally from the Manage scheduling page. For each run, you can review results, check run duration, and inspect any messages generated during execution.

When scheduling a data pipeline to run automatically, it is generally recommended to configure all Feature layer outputs to use the Replace or Add and update output method. This is because the Create with overwrite option allows for schema changes, which could result in broken downstream workflows such as pop-ups or filters.

To learn more about scheduling a data pipeline task, refer to this documentation.

In the future, you will also be able to automate data pipeline run in ArcGIS Enterprise using Microsoft Power Automate and the ArcGIS API for Python.

Licensing and deployment considerations

ArcGIS Data Pipelines is included with ArcGIS Enterprise Advanced, beginning with ArcGIS Enterprise 12.1 on Windows and Linux. In the future, it will also be available with ArcGIS Enterprise on Kubernetes.

To configure ArcGIS Data Pipelines, an administrator must set up a Data Pipelines Server site, configure the ArcGIS Web Adaptor, and federate it with ArcGIS Enterprise. This site can run on a single machine or multiple machines for high availability.

Once configured, Creator or higher user types with the appropriate privileges can access the Data Pipelines app and begin creating and automating workflows. By default, these privileges are included in the Publisher role, and custom roles can also be used to control access.

As noted above, ArcGIS Data Pipelines Server must be installed on a separate machine, without other ArcGIS software, for production environments, and it requires federation with ArcGIS Enterprise.

Administrators can deploy ArcGIS Data Pipelines Server on a single machine or deploy it across multiple machines. Scaling horizontally across multiple server machines allows you to support more concurrent data pipeline authors working in the editor, more simultaneous scheduled task runs, and it enables high availability, preventing downtime in the event of a machine failure. There are no licensing restrictions on the number of machines or cores, so you can scale the site to meet your workload requirements.

As an administrator, you can configure system properties to adjust security settings and manage resources. For example, you can define an allowed hosts list that can be used with the URL input to read in data, which is empty by default, or you can limit the file share inputs that can be used in data pipeline workflows within your organization to only a list of approved directories. By default, all file shares that are accessible by the ArcGIS Data Pipelines Server account will be available for use in the File share input.

To learn more, refer to the help on ArcGIS Server administrator operations.

More information

To get started, review the documentation and FAQ for answers to common questions.

Administrators may also want to review the installation guide and system properties that can be updated, while Data Pipelines users may want to follow this step-by-step tutorial: Get started with ArcGIS Data Pipelines in ArcGIS Enterprise.

There is much more to come for ArcGIS Data Pipelines. We value your feedback on new features and enhancements that can help improve your data preparation workflows. Share your ideas with us or ask us a question in the ArcGIS Data Pipelines community to continue the conversation.

The ArcGIS Enterprise 12.1 release delivers other exciting updates beyond ArcGIS Data Pipelines. To learn about the other highlights, read the What’s New in ArcGIS Enterprise 12.1 blog.