{"id":2723962,"date":"2025-03-11T11:07:29","date_gmt":"2025-03-11T18:07:29","guid":{"rendered":"https:\/\/www.esri.com\/arcgis-blog\/?post_type=blog&p=2723962"},"modified":"2025-04-17T14:44:57","modified_gmt":"2025-04-17T21:44:57","slug":"2025-developer-technology-summit-arcgis-data-pipelines","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/data-management\/2025-developer-technology-summit-arcgis-data-pipelines","title":{"rendered":"2025 Developer & Technology Summit: ArcGIS Data Pipelines"},"author":263392,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_searchwp_excluded":""},"categories":[23851],"tags":[776052,778742],"industry":[],"product":[36551],"class_list":["post-2723962","blog","type-blog","status-publish","format-standard","hentry","category-data-management","tag-arcgis-data-pipelines","tag-dev-summit-2025","product-arcgis-online"],"acf":{"authors":[{"ID":263392,"user_firstname":"Albert","user_lastname":"Schelin","nickname":"Albert Schelin","user_nicename":"aschelin","display_name":"Albert Schelin","user_email":"ASchelin@esri.com","user_url":"","user_registered":"2021-08-12 21:11:17","user_description":"Albert is a Product Marketing Manager at Esri, promoting ArcGIS Web Editor and ArcGIS Data Pipelines. When he isn't promoting Esri's products, you can find him enjoying a cup of single origin light roast coffee at a local coffee shop.","user_avatar":""},{"ID":7431,"user_firstname":"Bethany","user_lastname":"Scott","nickname":"Bethany Scott","user_nicename":"bethanyscott","display_name":"Bethany Scott","user_email":"BScott@esri.com","user_url":"","user_registered":"2018-03-21 18:21:18","user_description":"Bethany (she\/her) is a Product Engineer on the Data Pipelines team and the GeoAnalytics team. Her background is in biology and GIS with experience in data management and spatial-temporal analysis.","user_avatar":""},{"ID":325202,"user_firstname":"Sarah","user_lastname":"Hanson","nickname":"Sarah Hanson","user_nicename":"shanson","display_name":"Sarah Hanson","user_email":"shanson@esri.com","user_url":"","user_registered":"2023-01-11 15:54:54","user_description":"Sarah is a Product Manager on the ArcGIS Enterprise team, focused on data management. She also manages ArcGIS Data Pipelines, a native data integration capability in ArcGIS. In her free time, Sarah loves to explore new places and spend time outdoors.","user_avatar":""}],"short_description":"ArcGIS Data Pipelines Demo from the 2025 Esri Developer and Technology Summit","flexible_content":[{"acf_fc_layout":"content","content":"

ArcGIS Data Pipelines<\/span><\/a> is a no-code data engineering application in ArcGIS Online that simplifies creating and automating data preparation workflows.\u00a0<\/span>\u00a0<\/span><\/p>\n

At this year\u2019s Developer & Technology Summit, ArcGIS Data Pipelines Product Manager Sarah Hanson offers a great example of how ArcGIS Data Pipelines can be used to efficiently optimize the quality and readiness of data for visualization and analysis.<\/span><\/p>\n"},{"acf_fc_layout":"content","content":"

As a data engineer working at an agricultural research firm that studies crop yield trends, Sarah is tasked with preparing wheat production data that the organization can use to drive their mission-critical work. More specifically, she needs to transform wheat crop production data downloaded from the USDA\u2019s National Agricultural Statistics Service (NASS)<\/a> website into a usable feature layer that has crop yield information stored as columns by county.<\/span><\/p>\n"},{"acf_fc_layout":"content","content":"

See it in Action<\/h2>\n"},{"acf_fc_layout":"kaltura","video_id":"1_ms51rxhi","time":true,"start":"320","stop":"565"},{"acf_fc_layout":"content","content":"

Dive into the Steps<\/h2>\n"},{"acf_fc_layout":"content","content":"

Sarah shows a data pipeline that\u2019s configured to do the following:<\/span>\u00a0<\/span><\/p>\n

    \n
  1. Connect to a wheat production dataset (.csv) stored in an Azure Storage container.<\/span>\u00a0<\/span><\/li>\n
  2. Filter by attribute, to isolate just a single variable, total wheat production.<\/span><\/li>\n
  3. Calculate field, to create a unique ID for each county, by concatenating the state and county codes.<\/span>\u00a0<\/span><\/li>\n
  4. Pivot, to convert the long table into a wide one, where the resulting table has one record for each county and the production values for each year are organized into separate columns. <\/span>\u00a0<\/span><\/li>\n
  5. Join, to geospatially enable the data, using a USA Counties feature layer sourced from the ArcGIS Living Atlas.<\/span>\u00a0<\/span><\/li>\n
  6. Create a feature layer that the organization can leverage for visualization and analysis.<\/span>\u00a0<\/span><\/li>\n<\/ol>\n

     <\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2724882,"id":2724882,"title":"01_PreconfiguredDataPipeline_zoomed","filename":"01_PreconfiguredDataPipeline_zoomed-1.png","filesize":120643,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_PreconfiguredDataPipeline_zoomed-1.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/data-management\/2025-developer-technology-summit-arcgis-data-pipelines\/01_preconfigureddatapipeline_zoomed-2","alt":"The complete data pipeline workflow in the ArcGIS Data Pipelines editor.","author":"7431","description":"","caption":"Data pipeline used to transform wheat production data into a usable feature layer.","name":"01_preconfigureddatapipeline_zoomed-2","status":"inherit","uploaded_to":2723962,"date":"2025-03-10 16:36:16","modified":"2025-03-10 22:37:32","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1922,"height":1082,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_PreconfiguredDataPipeline_zoomed-1-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_PreconfiguredDataPipeline_zoomed-1.png","medium-width":464,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_PreconfiguredDataPipeline_zoomed-1.png","medium_large-width":768,"medium_large-height":432,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_PreconfiguredDataPipeline_zoomed-1.png","large-width":1918,"large-height":1080,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_PreconfiguredDataPipeline_zoomed-1-1536x865.png","1536x1536-width":1536,"1536x1536-height":865,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_PreconfiguredDataPipeline_zoomed-1.png","2048x2048-width":1922,"2048x2048-height":1082,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_PreconfiguredDataPipeline_zoomed-1-826x465.png","card_image-width":826,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_PreconfiguredDataPipeline_zoomed-1-1918x1080.png","wide_image-width":1918,"wide_image-height":1080}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

    The Data<\/h3>\n"},{"acf_fc_layout":"content","content":"

    The wheat crop production data was downloaded from the USDA\u2019s National Agricultural Statistics Service (NASS) website. It includes <\/span>county-level wheat production data from 1997 through 2022, storing multiple variables, including total wheat production. This results in a very long table with multiple records per county. <\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2724482,"id":2724482,"title":"01_1_RawCSV","filename":"01_1_RawCSV.png","filesize":1149149,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_1_RawCSV.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/data-management\/2025-developer-technology-summit-arcgis-data-pipelines\/01_1_rawcsv","alt":"CSV data from the USDA\u2019s National Agricultural Statistics Service (NASS) website.","author":"7431","description":"","caption":"Wheat crop data (CSV) downloaded from the USDA's NASS QuickStats website.","name":"01_1_rawcsv","status":"inherit","uploaded_to":2723962,"date":"2025-03-10 15:16:12","modified":"2025-03-10 22:38:09","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1918,"height":1183,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_1_RawCSV-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_1_RawCSV.png","medium-width":423,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_1_RawCSV.png","medium_large-width":768,"medium_large-height":474,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_1_RawCSV.png","large-width":1751,"large-height":1080,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_1_RawCSV-1536x947.png","1536x1536-width":1536,"1536x1536-height":947,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_1_RawCSV.png","2048x2048-width":1918,"2048x2048-height":1183,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_1_RawCSV-754x465.png","card_image-width":754,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/01_1_RawCSV-1751x1080.png","wide_image-width":1751,"wide_image-height":1080}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

    The Need<\/h3>\n"},{"acf_fc_layout":"content","content":"

    \u201cThe best way to visualize related information in a map is to transform it into a single table, with the geometries, in this example, counties,\u201d Sarah explains. The goal is to transform this CSV so that each county will become a unique record, with the wheat production values for each year stored in separate columns. Additionally, the data will need to be joined to a counties feature layer, so we can visualize it on a map. <\/span>\u00a0<\/span><\/p>\n

    Thanks to Data Pipelines, and its ability to make complex data preparation tasks easy, she shares that this task only took her a few minutes to complete.\u00a0<\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"content","content":"

    The Workflow<\/h3>\n"},{"acf_fc_layout":"content","content":"

    Sarah\u2019s workflow starts with connecting to the CSV dataset which is stored within an Azure Storage container. \u201cData Pipelines allows you to bring in data from many sources<\/a> including public URLs, cloud storage like I am using, databases, and more,\u201d she explains.<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2724982,"id":2724982,"title":"02_Inputs_with_box_border_match_inputs","filename":"02_Inputs_with_box_border_match_inputs.png","filesize":153147,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/02_Inputs_with_box_border_match_inputs.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/data-management\/2025-developer-technology-summit-arcgis-data-pipelines\/02_inputs_with_box_border_match_inputs","alt":"The input data sources available in ArcGIS Data Pipelines.","author":"7431","description":"","caption":"The input data sources available in ArcGIS Data Pipelines.","name":"02_inputs_with_box_border_match_inputs","status":"inherit","uploaded_to":2723962,"date":"2025-03-10 16:52:45","modified":"2025-03-10 22:38:29","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1922,"height":1083,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/02_Inputs_with_box_border_match_inputs-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/02_Inputs_with_box_border_match_inputs.png","medium-width":464,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/02_Inputs_with_box_border_match_inputs.png","medium_large-width":768,"medium_large-height":433,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/02_Inputs_with_box_border_match_inputs.png","large-width":1917,"large-height":1080,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/02_Inputs_with_box_border_match_inputs-1536x865.png","1536x1536-width":1536,"1536x1536-height":865,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/02_Inputs_with_box_border_match_inputs.png","2048x2048-width":1922,"2048x2048-height":1083,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/02_Inputs_with_box_border_match_inputs-826x465.png","card_image-width":826,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/02_Inputs_with_box_border_match_inputs-1917x1080.png","wide_image-width":1917,"wide_image-height":1080}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

    She then points out the data engineering tools<\/a> that Data Pipelines includes, that can be used to clean, construct, format, and integrate datasets before showing the tools used in her workflow.<\/span><\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2724992,"id":2724992,"title":"03_ConfiguredTools_with_tool_list_focus_match_elements","filename":"03_ConfiguredTools_with_tool_list_focus_match_elements.png","filesize":162151,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/03_ConfiguredTools_with_tool_list_focus_match_elements.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/data-management\/2025-developer-technology-summit-arcgis-data-pipelines\/03_configuredtools_with_tool_list_focus_match_elements","alt":"The data engineering tools available in ArcGIS Data Pipelines.","author":"7431","description":"","caption":"The data preparation tools available in ArcGIS Data Pipelines.","name":"03_configuredtools_with_tool_list_focus_match_elements","status":"inherit","uploaded_to":2723962,"date":"2025-03-10 16:53:07","modified":"2025-03-10 22:41:15","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1922,"height":1082,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/03_ConfiguredTools_with_tool_list_focus_match_elements-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/03_ConfiguredTools_with_tool_list_focus_match_elements.png","medium-width":464,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/03_ConfiguredTools_with_tool_list_focus_match_elements.png","medium_large-width":768,"medium_large-height":432,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/03_ConfiguredTools_with_tool_list_focus_match_elements.png","large-width":1918,"large-height":1080,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/03_ConfiguredTools_with_tool_list_focus_match_elements-1536x865.png","1536x1536-width":1536,"1536x1536-height":865,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/03_ConfiguredTools_with_tool_list_focus_match_elements.png","2048x2048-width":1922,"2048x2048-height":1082,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/03_ConfiguredTools_with_tool_list_focus_match_elements-826x465.png","card_image-width":826,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/03_ConfiguredTools_with_tool_list_focus_match_elements-1918x1080.png","wide_image-width":1918,"wide_image-height":1080}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

    Let\u2019s take a deeper dive into each of the tools that Sarah uses in her data pipeline.<\/span><\/p>\n

    To prepare her data, Sarah uses the following tools:<\/span>\u00a0<\/span><\/p>\n