Preparing data for mapping and visualization often means pulling information from several systems, cleaning it up, and getting it into a consistent format. However, that can also mean a lot of repetitive data preparation tasks.
ArcGIS Data Pipelines helps you build workflows to connect, prepare, and integrate data in ArcGIS. You can also schedule your workflows to run automatically instead of repeating the same preparation steps manually. With ArcGIS Enterprise 12.2, ArcGIS Data Pipelines adds direct connections to relational databases, new data preparation tools, file outputs, expanded administration capabilities, and ArcGIS REST API support. Check out the video below to see some of these new features in action, or keep reading to learn more.
Connect directly to relational databases
ArcGIS Data Pipelines now supports tables from Microsoft SQL Server, PostgreSQL, and Oracle as inputs. This enables you to work directly with data in these databases alongside data from other sources such cloud object stores, and cloud data warehouses, and APIs.
For example, you can join a table from your PostgreSQL database with a feature layer from ArcGIS Living Atlas, enhancing your organizational data with authoritative spatial information curated by ArcGIS.
Prepare data with new tools and more control
With this release, the following new tools give you more options for preparing your data:
- Find and replace helps standardize text values across one or more string fields. Use it to replace inconsistent terms, update outdated names, or clean up null, empty, and whitespace values.
- Create buffers creates polygon areas around point, line, or polygon geometries. Buffer distances can be consistent across features or vary for individual features.
Additionally, some existing tools have been enhanced with new capabilities:
- Update fields can convert geometry, date, and date-only fields to strings using a selected format. For example, you can choose WKT when converting geometry to text.
- Filter by attribute now supports entering SQL directly, in addition to using the guided query builder. This gives SQL users a familiar way to build more complex filters.
- Calculate field now includes the ArcGIS Arcade Assistant (Beta), which helps you generate Arcade expressions from natural-language prompts.
New and improved outputs
The new File output lets you save data pipeline results to your ArcGIS Enterprise content as a CSV, GeoJSON, or Apache Parquet file. You can create a new file item or overwrite an existing one, giving you more flexibility in how prepared data is stored and reused without publishing it as a feature layer.
Additionally, Feature layer outputs have been enhanced with improved cancellation behavior. If an underlying feature service append job becomes unresponsive, the output can now cancel automatically rather than continuing to wait for the service to become available.
Manage Data Pipelines Server from the app
New administration capabilities in the Data Pipelines app make it easier to monitor and manage your Data Pipelines Server site. Administrators can now view and filter logs, review active compute resources, and update system and log properties directly from the app. For example, you can easily configure the list of allowed hosts to enable access to data from external URLs or APIs. Or, modify system constraints to manage the resources available to your Data Pipelines Server site.
Additionally, the new Max cores per worker setting lets administrators define the number of processor cores available to each compute resource. You can use this setting along with others, like Max workers per machine and Max memory per worker (MiB), to scale and optimize your Data Pipelines Server site. Together, these settings help you allocate resources to power larger workflows or support more users and data pipeline workflows running at the same time.
To learn more, see Manage ArcGIS Data Pipelines Server using the Data Pipelines app.
Automate workflows with APIs
You can now run and automate data pipeline workflows using the ArcGIS API for Python or the ArcGIS REST APIs. This makes it easier to incorporate data integration into existing scripts, scheduled processes, and other automated workflows.
Additionally, the ArcGIS REST APIs support administering the ArcGIS Data Pipelines Server site, allowing administrators to manage server operations programmatically alongside other ArcGIS Enterprise administration tasks.
Conclusion
With ArcGIS Enterprise 12.2, you can now connect directly to more organizational data, use new tools to prepare it, choose to deliver results as feature layers or files, and leverage new capabilities to monitor, manage, and automate your work.
More information
Explore the What’s new in ArcGIS Data Pipelines 12.2 documentation for the full release details and links to learn more about the new capabilities.
You can also learn more about connecting to your data, explore available data preparation tools, and review how to manage ArcGIS Data Pipelines Server.
The ArcGIS Enterprise 12.2 release includes additional enhancements beyond ArcGIS Data Pipelines. To learn about the other highlights, read the What’s New in ArcGIS Enterprise 12.2 blog.
We value your feedback on features and enhancements that can improve your data preparation workflows. Share your ideas or ask a question in the ArcGIS Data Pipelines community.
Commenting is not enabled for this article.