{"id":2530802,"date":"2024-11-12T00:00:57","date_gmt":"2024-11-12T08:00:57","guid":{"rendered":"https:\/\/www.esri.com\/arcgis-blog\/?post_type=blog&p=2530802"},"modified":"2025-02-12T05:11:43","modified_gmt":"2025-02-12T13:11:43","slug":"whats-new-in-arcgis-data-pipelines-november-2024","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/announcements\/whats-new-in-arcgis-data-pipelines-november-2024","title":{"rendered":"What’s New in ArcGIS Data Pipelines (November 2024)"},"author":7431,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_searchwp_excluded":""},"categories":[37101,23851],"tags":[776052,42641,760452,764952,776962],"industry":[],"product":[36551],"class_list":["post-2530802","blog","type-blog","status-publish","format-standard","hentry","category-announcements","category-data-management","tag-arcgis-data-pipelines","tag-arcgis-online","tag-data-engineering","tag-etl","tag-whats-new-november-2024","product-arcgis-online"],"acf":{"authors":[{"ID":7431,"user_firstname":"Bethany","user_lastname":"Scott","nickname":"Bethany Scott","user_nicename":"bethanyscott","display_name":"Bethany Scott","user_email":"BScott@esri.com","user_url":"","user_registered":"2018-03-21 18:21:18","user_description":"Bethany (she\/her) is a Product Engineer on the Data Pipelines team and the GeoAnalytics team. Her background is in biology and GIS with experience in data management and spatial-temporal analysis.","user_avatar":""},{"ID":358272,"user_firstname":"Halle","user_lastname":"Martinucci","nickname":"Halle Martinucci","user_nicename":"hmartinucci","display_name":"Halle Martinucci","user_email":"hmartinucci@esri.com","user_url":"","user_registered":"2024-06-18 18:43:42","user_description":"Halle is an Associate Product Marketing Manager on Esri\u2019s Spatial Analytics & Data Science team.","user_avatar":""}],"short_description":"Document your data pipeline using notes, and prepare your data with new tools, and overwrite layers with the November update of ArcGIS Online.","flexible_content":[{"acf_fc_layout":"content","content":"

In February, we released ArcGIS Data Pipelines<\/span><\/a>, a powerful tool that streamlines data integration in ArcGIS Online. This innovative app offers a fast and efficient way to bring in datasets, geospatially enable them, clean and prepare them, and write the processed data out to a feature layer ready for use in mapping, analysis, and reporting. Available to all Creators and above in ArcGIS Online, Data Pipelines provides you with a low-code, drag-and-drop approach to data engineering, making it easier than ever to design and automate your data integration workflows. In the June update of Data Pipelines<\/span><\/a>, we added support for new input file formats, enhanced the Calculate field tool to support Arcade geometry functions, and made a few enhancements to scheduling including the ability to schedule more frequent updates.<\/p>\n

 <\/p>\n

What\u2019s New in November?<\/b><\/h3>\n

The November 2024 update of Data Pipelines includes new tools, the ability to document your workflow using notes, a new output method to overwrite existing layers, and more. Check out the video below to see some new features in action or keep reading for more information.<\/p>\n

 <\/p>\n"},{"acf_fc_layout":"kaltura","video_id":"1_bp2xtsmy","time":false,"start":0,"stop":""},{"acf_fc_layout":"content","content":"

 <\/p>\n

New data engineering tools<\/b><\/h3>\n

Use two new tools, Pivot and Dissolve, to distill your data into summarized and transformed datasets.<\/p>\n

With the Pivot<\/span><\/a> tool, you can categorize and transform your long, tabular datasets with many records into wide datasets with many fields. This tool can help you explore trends, limit your data to values and attributes of interest, or group your data into summarized categories. Additionally, summarized pivot results can be visualized in dashboards or used for charting. For example, you can create a pivot table using election data to capture voter turnout by geographic region and age group, or any other demographic.<\/p>\n

Dissolve<\/a><\/span> can complete similar aggregations, but instead of acting on attributes alone, it also considers polygon or polyline geometries to identify relationships between records. For example, you can use Dissolve if you want to merge California county polygons into a single state polygon. If each county has population counts or other demographic information, you can choose to calculate summary statistics for them.<\/p>\n

 <\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2561082,"id":2561082,"title":"dissolve_gif_1","filename":"dissolve_gif_1.gif","filesize":1243772,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/11\/dissolve_gif_1.gif","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/announcements\/whats-new-in-arcgis-data-pipelines-november-2024\/dissolve_gif_1","alt":"Dissolve county boundaries into states.","author":"7431","description":"","caption":"Dissolve county boundaries into states.","name":"dissolve_gif_1","status":"inherit","uploaded_to":2530802,"date":"2024-11-05 16:55:30","modified":"2024-11-12 17:28:10","menu_order":0,"mime_type":"image\/gif","type":"image","subtype":"gif","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1196,"height":630,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/11\/dissolve_gif_1-213x200.gif","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/11\/dissolve_gif_1.gif","medium-width":464,"medium-height":244,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/11\/dissolve_gif_1.gif","medium_large-width":768,"medium_large-height":405,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/11\/dissolve_gif_1.gif","large-width":1196,"large-height":630,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/11\/dissolve_gif_1.gif","1536x1536-width":1196,"1536x1536-height":630,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/11\/dissolve_gif_1.gif","2048x2048-width":1196,"2048x2048-height":630,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/11\/dissolve_gif_1-826x435.gif","card_image-width":826,"card_image-height":435,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/11\/dissolve_gif_1.gif","wide_image-width":1196,"wide_image-height":630}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Document your workflow using notes<\/b><\/h3>\n

You asked, and we listened! We\u2019re excited to introduce the ability to add notes to your data pipeline, allowing you to document your process. Use notes to add context to individual workflow elements or provide an overview of the entire data pipeline. You can use notes however you want, but here are some examples of how notes could be used based on feedback from other Data Pipelines users:<\/p>\n