"},{"ID":263392,"user_firstname":"Albert","user_lastname":"Schelin","nickname":"Albert Schelin","user_nicename":"aschelin","display_name":"Albert Schelin","user_email":"ASchelin@esri.com","user_url":"","user_registered":"2021-08-12 21:11:17","user_description":"Albert is a Product Marketing Manager at Esri, promoting ArcGIS Web Editor and ArcGIS Data Pipelines. When he isn't promoting Esri's products, you can find him enjoying a cup of single origin light roast coffee at a local coffee shop.","user_avatar":"
"}],"short_description":"Get familiarized with the various data preparation tools at your disposal in ArcGIS Data Pipelines.","flexible_content":[{"acf_fc_layout":"content","content":"ArcGIS Data Pipelines<\/a> is a data preparation and integration app in ArcGIS Online and now ArcGIS Enterprise<\/a>. Save time preparing data for visualization and analytics with a no-code, visual data engineering app. Connect to various data sources, apply commonly used data preparation tools, and write results to your content that can be automated to keep data up to date and reliable for decision making.<\/p>\n As the title suggests, this blog will focus on the data preparation tools within ArcGIS Data Pipelines. There are two main goals of this blog. The first goal of this blog is to provide an overview of each of the data processing tools available within the app, with helpful context and descriptions that demonstrate what each tool does. The second goal of this blog is to help you decide which tools to use as you build out your own data pipeline workflows, saving you time or serving as a source of inspiration.<\/p>\n This blog is up to date as of the February 2026 update to ArcGIS Online and ArcGIS Enterprise 12.1 release of ArcGIS Data Pipelines.<\/p>\n To see ArcGIS Data Pipelines in action, refer to the video below.<\/p>\n In the demo video above, we brought in a CSV file representing customer locations from Amazon S3 and applied a series of tools to prepare the data for use in ArcGIS. We used data preparation tools to make the data spatial (Create geometry), filter records to only those within California (Clip), and remove personally identifiable information (Select fields). To automate data updates and ensure the data in ArcGIS Online remained up-to-date, we ran the data pipeline and scheduled it to run on a weekly basis.<\/p>\n Now that we understand what Data Pipelines is and how it fits into the broader ArcGIS ecosystem, let\u2019s briefly explore the input data sources the app supports, before learning about how each tool group empowers you in transforming your data into results that are ready for mapping and analytics.<\/p>\n"},{"acf_fc_layout":"content","content":" Data Pipelines supports a wide variety of input data sources. You can connect to files from your content, from a URL or API, from a file share (in the case of ArcGIS Data Pipelines in ArcGIS Enterprise), or from a cloud storage container such as an Amazon S3 bucket or Microsoft Azure Storage container.<\/p>\n We also support reading in tables from Snowflake, Google BigQuery, and Databricks (only currently available in ArcGIS Online), in addition to feature layers from ArcGIS Online or ArcGIS Enterprise. While this blog is focused more on tools, you can read more about the supported input data sources here:<\/p>\n ArcGIS Data Pipelines includes all the common data preparation tools needed by GIS users to prepare their data for use in visualization and analysis workflows within ArcGIS Online and ArcGIS Enterprise. Together, they allow data to be efficiently transformed into an optimal state for downstream workflows.<\/p>\n Tools are organized into four categorical toolsets: clean, construct, format, and integrate.<\/p>\n"},{"acf_fc_layout":"content","content":" Focuses on improving data quality and removing unnecessary or redundant information.<\/p>\n Used to create new information or structure in your datasets.<\/p>\n Helps standardize structure, schema, and projection for downstream use.<\/p>\n Designed for combining, summarizing, and consolidating datasets.<\/p>\n To learn more about each of the data processing tools, refer to the documentation<\/p>\n ArcGIS Data Pipelines is an easy-to-use data integration app that has powerful tools for data preparation. Each of the tools in Data Pipelines help you transform your data, optimizing its readiness for downstream use in ArcGIS.<\/p>\n This blog provided sample workflows that highlighted some of the tools in action and it helped to define each tool and a common use case for it. If you are new to ArcGIS Data Pipelines, use this blog as a resource to understand how you might design your first data pipeline workflow. If you are an existing user, use this blog to gain inspiration and insight into tools you may not have used before.<\/p>\n"},{"acf_fc_layout":"content","content":" Whether you\u2019re just getting started with ArcGIS Data Pipelines or looking to sharpen your skills, we\u2019ve got you covered. Here are some helpful resources to explore:<\/p>\nSee ArcGIS D<\/span>ata Pipelines in\u00a0<\/span>A<\/span>ction<\/span><\/span>\u00a0<\/span><\/strong><\/h3>\n"},{"acf_fc_layout":"kaltura","video_id":"1_uhniply7","time":false,"start":0},{"acf_fc_layout":"content","content":"
Bring Your Data into ArcGIS Data Pipelines<\/strong><\/h3>\n
\n
Transform and Prepare Your Data<\/strong><\/h3>\n
Clean<\/strong><\/h3>\n
\n
\nExample: Clip building data to a single city\u2019s boundary for a focused analysis.<\/li>\n
\nExample: Filter parcels where LandUse = ‘Residential’.<\/li>\n
\nExample: Limit tree data to a park boundary.<\/li>\n
\nExample: Remove wells that have the same Well_ID to ensure each well is represented once.<\/li>\n
\nExample: Keep Name, Type, and Height_m fields from a larger dataset.<\/li>\n
\nExample: Simplify watershed polygons for display performance.<\/li>\n<\/ul>\n"},{"acf_fc_layout":"content","content":"Construct<\/strong><\/h3>\n
\n
\nExample: Compute area in square kilometers with Area($record.GEOMETRY).<\/li>\n
\nExample: Merge Year, Month, and Day fields into a single SurveyDate field.<\/li>\n
\nExample: Create a point geometry field from a CSV containing Longitude and Latitude coordinate fields.<\/li>\n<\/ul>\n"},{"acf_fc_layout":"content","content":"Format<\/strong><\/h3>\n
\n
\nExample: Map source fields such as \u201cCategory,\u201d \u201cType,\u201d or \u201cEvent_Code\u201d to a standardized \u201cIncident_Type\u201d field in the target.<\/li>\n
\nExample: Transform monthly sales records into a single row per store with columns for each month.<\/li>\n
\nExample: Convert data from WGS84 to NAD83 UTM for accurate distance measurements in downstream analysis.<\/li>\n
\nExample: Split a \u201ctags\u201d array into separate attribute fields.<\/li>\n
\nExample: Standardize County names to ensure consistent capitalization.<\/li>\n<\/ul>\n"},{"acf_fc_layout":"content","content":"Integrate<\/strong><\/h3>\n
\n
\nExample: Merge parcel polygons that share a common ZoningType.<\/li>\n
\nExample: Join demographic tables to census tract polygons using a shared GEOID field.<\/li>\n
\nExample: Merge four quarterly inspection datasets into one annual dataset.<\/li>\n
\nExample: Summarize tree counts by species to identify the most common types citywide.<\/li>\n<\/ul>\n\n
Summary<\/strong><\/h3>\n
Additional Resources<\/strong><\/h3>\n
\n