"},{"ID":323652,"user_firstname":"Corinne","user_lastname":"Walker","nickname":"cwalker","user_nicename":"cwalker","display_name":"Corinne Walker","user_email":"cwalker@esri.com","user_url":"","user_registered":"2022-12-06 16:07:24","user_description":"Corinne is a Product Marketing Manager on Esri\u2019s Spatial Analytics & Data Science team. She has a background in marketing and business analytics with experience working in the technology and geospatial industries.","user_avatar":"
"},{"ID":352692,"user_firstname":"Mariia","user_lastname":"Lubinova","nickname":"Mariia Lubinova","user_nicename":"mlubinova","display_name":"Mariia Lubinova","user_email":"mlubinova@esri.com","user_url":"","user_registered":"2024-02-07 23:40:22","user_description":"Mariia is a Product Marketing Manager on Esri's Spatial Analytics & Data Science team, with a background in Marketing and Business Development.","user_avatar":"
"}],"short_description":"ArcGIS Data Pipelines is now available for general use. New tools, features, and enhancements were also added in the February 2024 update.","flexible_content":[{"acf_fc_layout":"content","content":"Effective geospatial analysis relies heavily on accessing, integrating, and preparing data from a variety of sources. This can be a time consuming and tedious task, made even more complex when the source data updates on a regular basis. ArcGIS Data Pipelines<\/a> provides a comprehensive solution to this challenge, and we are excited to announce that it is officially out of beta and available for general use in ArcGIS Online.<\/p>\n"},{"acf_fc_layout":"content","content":" As a native data integration capability, Data Pipelines offers a fast and efficient way to ingest, engineer, and maintain data from a variety of locations, including multiple cloud-based data stores. The drag-and-drop interface minimizes the need for coding skills and simplifies the data integration and preparation process. In addition, the scheduling functionality automates data updates to\u00a0 ensure that your data remains current as the source data evolves.<\/p>\n"},{"acf_fc_layout":"content","content":" In this blog we will explore the recent release enhancements and new tools, and how they streamline data integration workflows. Keep reading to learn how Data Pipelines can unlock the full potential of your geospatial data!<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2261572,"id":2261572,"title":"arcgis-data-pipelines-diagram-n-shadow","filename":"arcgis-data-pipelines-diagram-n-shadow-e1709075235363.png","filesize":551577,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-data-pipelines-diagram-n-shadow-e1709075235363.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/announcements\/arcgis-data-pipelines-is-now-available\/arcgis-data-pipelines-diagram-n-shadow","alt":"ArcGIS Data Pipelines provides a data integration capability in ArcGIS Online.","author":"7431","description":"","caption":"ArcGIS Data Pipelines is an ArcGIS Online app that facilitates data integration and transformation, supporting a wide range of data formats.","name":"arcgis-data-pipelines-diagram-n-shadow","status":"inherit","uploaded_to":2223432,"date":"2024-02-27 21:12:13","modified":"2024-02-27 21:26:31","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":2597,"height":1778,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-data-pipelines-diagram-n-shadow-e1709075235363-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-data-pipelines-diagram-n-shadow-e1709075235363-381x261.png","medium-width":381,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-data-pipelines-diagram-n-shadow-e1709075235363-768x526.png","medium_large-width":768,"medium_large-height":526,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-data-pipelines-diagram-n-shadow-e1709075235363-1577x1080.png","large-width":1577,"large-height":1080,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-data-pipelines-diagram-n-shadow-e1709075235363-1536x1052.png","1536x1536-width":1536,"1536x1536-height":1052,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-data-pipelines-diagram-n-shadow-e1709075235363-2048x1402.png","2048x2048-width":2048,"2048x2048-height":1402,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-data-pipelines-diagram-n-shadow-e1709075235363-679x465.png","card_image-width":679,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-data-pipelines-diagram-n-shadow-e1709075235363-1577x1080.png","wide_image-width":1577,"wide_image-height":1080}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" In the February 2024 update, Data Pipelines has all the features you may know from beta,<\/em> and more! To learn more about the beta releases of Data Pipelines, see the Introducing Data Pipelines in ArcGIS Online (beta release)<\/a> and the What’s New in Data Pipelines (October 2023)<\/a> blog posts.<\/p>\n"},{"acf_fc_layout":"content","content":" New features in this update include support for JSON files, new data engineering tools, plus new experiences and dialogs in the application. Check out the video demo and details below for more information.<\/p>\n"},{"acf_fc_layout":"kaltura","video_id":"1_ouyx82nl","time":false,"start":0,"stop":""},{"acf_fc_layout":"content","content":" Get started with Data Pipelines by taking a quick guided tour of the editor. The tour assists you in learning the key concepts and features of Data Pipelines before you begin building your data integration workflows. You\u2019ll see the tour appear automatically the first time you log in. After that you can access it at any time from the editor toolbar.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2262132,"id":2262132,"title":"DataPipelinesGuidedTour1280720","filename":"DataPipelinesGuidedTour1280720.gif","filesize":801314,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/DataPipelinesGuidedTour1280720.gif","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/announcements\/arcgis-data-pipelines-is-now-available\/datapipelinesguidedtour1280720","alt":"Guided tour of the Data Pipelines editor","author":"7431","description":"","caption":"","name":"datapipelinesguidedtour1280720","status":"inherit","uploaded_to":2223432,"date":"2024-02-27 22:02:15","modified":"2024-02-27 22:02:37","menu_order":0,"mime_type":"image\/gif","type":"image","subtype":"gif","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1280,"height":720,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/DataPipelinesGuidedTour1280720-213x200.gif","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/DataPipelinesGuidedTour1280720.gif","medium-width":464,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/DataPipelinesGuidedTour1280720.gif","medium_large-width":768,"medium_large-height":432,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/DataPipelinesGuidedTour1280720.gif","large-width":1280,"large-height":720,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/DataPipelinesGuidedTour1280720.gif","1536x1536-width":1280,"1536x1536-height":720,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/DataPipelinesGuidedTour1280720.gif","2048x2048-width":1280,"2048x2048-height":720,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/DataPipelinesGuidedTour1280720-826x465.gif","card_image-width":826,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/DataPipelinesGuidedTour1280720.gif","wide_image-width":1280,"wide_image-height":720}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" Data Pipelines now supports JSON files from Amazon S3<\/a>, Microsoft Azure Storage<\/a>, and Public URL<\/a> inputs. Note that JSON files are not yet supported for uploading directly to your content and therefore cannot be used as a File<\/a> input.<\/p>\n"},{"acf_fc_layout":"content","content":" There are also new tools and enhancements for working with the complex field types commonly found in JSON, GeoJSON, and parquet files, such as arrays, maps, and struct fields. These field types contain lists and dictionaries of information that can be difficult to work with without the right tools. To better leverage the information stored in these field types, we\u2019ve added the following new tools and enhancements:<\/p>\n"},{"acf_fc_layout":"content","content":" With the new Map fields<\/a> tool, you can now efficiently update and standardize dataset schemas. For example, Map fields is commonly used to prepare data for use in the Merge<\/a> tool, or for matching fields to the feature layer you want to update using the output Feature layer<\/a> tool. See the screenshot below for an example of how input dataset fields could be mapped to target fields.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2262652,"id":2262652,"title":"MapFields_DataPipelines1","filename":"MapFields_DataPipelines1.png","filesize":46651,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/MapFields_DataPipelines1.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/announcements\/arcgis-data-pipelines-is-now-available\/mapfields_datapipelines1","alt":"Map fields modal in Data Pipelines","author":"7431","description":"","caption":"","name":"mapfields_datapipelines1","status":"inherit","uploaded_to":2223432,"date":"2024-02-27 22:44:15","modified":"2024-02-27 22:44:29","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":964,"height":797,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/MapFields_DataPipelines1-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/MapFields_DataPipelines1.png","medium-width":316,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/MapFields_DataPipelines1.png","medium_large-width":768,"medium_large-height":635,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/MapFields_DataPipelines1.png","large-width":964,"large-height":797,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/MapFields_DataPipelines1.png","1536x1536-width":964,"1536x1536-height":797,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/MapFields_DataPipelines1.png","2048x2048-width":964,"2048x2048-height":797,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/MapFields_DataPipelines1-562x465.png","card_image-width":562,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/MapFields_DataPipelines1.png","wide_image-width":964,"wide_image-height":797}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" Another new feature is the Connection details dialog. Use this dialog to find out when the connection was initiated and how long it has been active. The learn more link<\/a> in the dialog provides more information including how connections work and how credits are consumed while using Data Pipelines.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2260042,"id":2260042,"title":"connection_dialog","filename":"connection_dialog.png","filesize":24997,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/connection_dialog.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/announcements\/arcgis-data-pipelines-is-now-available\/connection_dialog","alt":"Connection details dialog in the Data Pipelines editor","author":"7431","description":"","caption":"","name":"connection_dialog","status":"inherit","uploaded_to":2223432,"date":"2024-02-27 18:09:41","modified":"2024-02-27 18:09:56","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":406,"height":349,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/connection_dialog-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/connection_dialog.png","medium-width":304,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/connection_dialog.png","medium_large-width":406,"medium_large-height":349,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/connection_dialog.png","large-width":406,"large-height":349,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/connection_dialog.png","1536x1536-width":406,"1536x1536-height":349,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/connection_dialog.png","2048x2048-width":406,"2048x2048-height":349,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/connection_dialog.png","card_image-width":406,"card_image-height":349,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/connection_dialog.png","wide_image-width":406,"wide_image-height":349}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" We want to hear from you! There is much more to come for ArcGIS Data Pipelines in the future, and we value your opinion on what new features and enhancements we can add to help you with your data preparation workflows. Share your ideas<\/a> or ask us a question<\/a> in the Data Pipelines community.<\/p>\n"},{"acf_fc_layout":"content","content":" More details on new enhancements and improvements with the February update can be found in the What’s new<\/a> documentation. To learn more about the powerful data integration capabilities of Data Pipelines, check out the Introducing Data Pipelines in ArcGIS Online (beta release)<\/a> blog post, the What’s New in Data Pipelines (October 2023)<\/a> blog post, and refer to the product documentation<\/a>.<\/p>\n"}],"related_articles":"","card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/arcgis-blog-post-arcgis-data-pipelines-ga-release-card.jpg","wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2028\/02\/Banner_image_final.png"},"yoast_head":"\nWhat’s New in ArcGIS Data Pipelines (February 2024)<\/strong><\/h3>\n"},{"acf_fc_layout":"content","content":"
Guided tour of the Data Pipelines editor<\/strong><\/h4>\n"},{"acf_fc_layout":"content","content":"
JSON files and complex field support<\/strong><\/h4>\n"},{"acf_fc_layout":"content","content":"
\n
\n
$record.Services[\"Computer_Access\"]<\/code>.<\/li>\n<\/ul>\n"},{"acf_fc_layout":"content","content":"\n
Public_Wifi_Access<\/code> field by drilling into the Services<\/code> field using the field picker.<\/li>\n<\/ul>\n"},{"acf_fc_layout":"content","content":"Map fields tool<\/strong><\/h4>\n"},{"acf_fc_layout":"content","content":"
Connection details dialog<\/strong><\/h4>\n"},{"acf_fc_layout":"content","content":"
More information<\/strong><\/h4>\n"},{"acf_fc_layout":"content","content":"