"},{"ID":358272,"user_firstname":"Halle","user_lastname":"Martinucci","nickname":"Halle Martinucci","user_nicename":"hmartinucci","display_name":"Halle Martinucci","user_email":"hmartinucci@esri.com","user_url":"","user_registered":"2024-06-18 18:43:42","user_description":"Halle is an Associate Product Marketing Manager on Esri\u2019s Spatial Analytics & Data Science team.","user_avatar":"
"}],"short_description":"This blog showcases the top questions and answers regarding the newly launched ArcGIS Data Pipelines from the 2024 User Conference.","flexible_content":[{"acf_fc_layout":"content","content":"The Esri User Conference consistently attracts GIS enthusiasts from around the world, showcasing major technological breakthroughs, and motivating attendees to contribute to a shared goal of creating a better future through mapping. The 2024 conference was no exception, celebrating a significant milestone as ArcGIS Data Pipelines moved out of beta — an innovation set to revolutionize data integration for ArcGIS Online users. The enthusiasm of the attendees sparked numerous discussions, and we’re thrilled to relay the insights gained from these interactions.<\/p>\n
Dive into this blog for answers to the most common questions about Data Pipelines and start exploring the latest no-code data integration capability from Esri!<\/p>\n
<\/p>\n
What is ArcGIS Data Pipelines?<\/strong><\/p>\n ArcGIS Data Pipelines<\/span><\/a> is a new, native data integration capability in ArcGIS Online. It offers a fast and efficient way to integrate, prepare, and engineer data from a variety of sources, including cloud-based data stores including Amazon S3, Microsoft Azure Storage, Google BigQuery, and Snowflake. It offers a familiar drag-and-drop user interface, allowing organizations to create powerful, visual data integration workflows without the need for a Python script or additional software.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2449812,"id":2449812,"title":"DataPipelines","filename":"001DataPipelines.jpg","filesize":190229,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/001DataPipelines.jpg","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/data-management\/arcgis-data-pipelines-at-the-2024-uc-faq\/001datapipelines","alt":"A screenshot of a data integration workflow in ArcGIS Online.","author":"358272","description":"","caption":"ArcGIS Data Pipelines provides data integration with ArcGIS.","name":"001datapipelines","status":"inherit","uploaded_to":2449802,"date":"2024-08-01 16:05:28","modified":"2024-08-03 00:40:42","menu_order":0,"mime_type":"image\/jpeg","type":"image","subtype":"jpeg","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1911,"height":971,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/001DataPipelines-213x200.jpg","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/001DataPipelines.jpg","medium-width":464,"medium-height":236,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/001DataPipelines.jpg","medium_large-width":768,"medium_large-height":390,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/001DataPipelines.jpg","large-width":1911,"large-height":971,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/001DataPipelines-1536x780.jpg","1536x1536-width":1536,"1536x1536-height":780,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/001DataPipelines.jpg","2048x2048-width":1911,"2048x2048-height":971,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/001DataPipelines-826x420.jpg","card_image-width":826,"card_image-height":420,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/001DataPipelines.jpg","wide_image-width":1911,"wide_image-height":971}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" <\/p>\n Who can use Data Pipelines?<\/strong><\/p>\n Data Pipelines is available to all ArcGIS Online organizations. The people who can access the app include Creator or above user types, with the Publisher or equivalent roles.<\/p>\n Please note, Data Pipelines is not available today for users of ArcGIS Location Platform or Personal Use and Trial subscriptions.<\/p>\n <\/p>\n How does Data Pipelines handle security for data store connections and sharing, especially when dealing with password-protected data sources?<\/strong><\/p>\n All connection information to secured data sources is encrypted and stored within a data store item. This data store item is saved to your content in ArcGIS Online, and can be used in the data pipeline, but cannot be shared with other users.<\/p>\n To learn more about managing connections to supported cloud-based data stores, please refer to this product documentation:\u202fWork with input data<\/span><\/a>.<\/p>\n <\/p>\n Can I connect to an on-premises database (Microsoft SQL Server, Oracle, PostgreSQL)?<\/strong><\/p>\n As of right now, no. Databases that are set up and run on your company’s own system, like Microsoft SQL Server, Oracle, and PostgreSQL, rarely let you connect from outside of their network or the internet. This is what would be needed for Data Pipelines. For this reason, our support for connecting to external databases is generally limited to cloud-based data sources. If your organization copies the data into one of the supported data sources, then you would be able to leverage it within a data pipeline.<\/p>\n <\/p>\n Can I schedule data pipelines?<\/strong><\/p>\n Yes! To automate your workflows, you can schedule data pipelines to run on regular intervals using tasks. Tasks can be scheduled to run from every 15 minutes to every few months.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2450542,"id":2450542,"title":"Data Pipelines Scheduler","filename":"002Scheduler-optimize.gif","filesize":2046410,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/002Scheduler-optimize.gif","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/data-management\/arcgis-data-pipelines-at-the-2024-uc-faq\/002scheduler-optimize","alt":"Demo of using the scheduler feature on ArcGIS Data Pipelines.","author":"358272","description":"","caption":"Use the scheduled tasks page to manage tasks for existing data pipelines.","name":"002scheduler-optimize","status":"inherit","uploaded_to":2449802,"date":"2024-08-01 21:06:04","modified":"2024-08-03 00:39:19","menu_order":0,"mime_type":"image\/gif","type":"image","subtype":"gif","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":909,"height":459,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/002Scheduler-optimize-213x200.gif","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/002Scheduler-optimize.gif","medium-width":464,"medium-height":234,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/002Scheduler-optimize.gif","medium_large-width":768,"medium_large-height":388,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/002Scheduler-optimize.gif","large-width":909,"large-height":459,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/002Scheduler-optimize.gif","1536x1536-width":909,"1536x1536-height":459,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/002Scheduler-optimize.gif","2048x2048-width":909,"2048x2048-height":459,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/002Scheduler-optimize-826x417.gif","card_image-width":826,"card_image-height":417,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/08\/002Scheduler-optimize.gif","wide_image-width":909,"wide_image-height":459}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" Scheduled tasks can be managed either centrally using the scheduled task page shown above or they can also be managed for a single data pipeline from within the editor.<\/p>\n To learn more about scheduling, see the help topic <\/span>Schedule a data pipeline task<\/span><\/a>.<\/span>\u00a0<\/span><\/p>\n <\/p>\n What are the differences between Data Pipelines and Data Interoperability or FME by Safe Software?<\/strong><\/p>\n Data Pipelines is a native data integration capability of ArcGIS Online. It runs within ArcGIS Online and is licensed as a part of (is included with) ArcGIS Online. The focus of Data Pipelines is purely on streamlining data integration for ArcGIS Online, with the resulting data being written to hosted feature layers. ArcGIS Data Interoperability (and FME) supports a larger set of inputs and transformations and is able to write results back to the source.<\/p>\n <\/p>\n How is Data Pipelines different from ArcGIS Velocity?<\/strong><\/p>\n While there are similarities in that both allow you to connect to external data sources and import the data into ArcGIS Online, they serve distinct purposes. ArcGIS Velocity is specifically designed for real-time and big data processing, efficiently handling high-speed data streams from sensors and similar sources. It also is focused on enabling analytics such as device tracking, incident detection, and pattern analysis. Data Pipelines is primarily a data integration application that focuses on data engineering tasks, particularly for non-sensor-based data streams.<\/p>\n If you are interested in learning about the different ways data can be integrated with ArcGIS Online, check out this ArcGIS Blog: Seven Ways to Integrate Data with ArcGIS Online<\/a>.<\/p>\n <\/p>\n Is ArcGIS Data Pipelines like ModelBuilder but for ArcGIS Online?<\/strong><\/p>\n ModelBuilder and Data Pipelines are similar in that they offer a low code, drag and drop user experience for authoring repeatable workflows. However, there are some key differences:<\/p>\n <\/p>\n Is ArcGIS Data Pipelines coming to ArcGIS Enterprise?<\/strong><\/p>\n Yes. ArcGIS Data Pipelines in ArcGIS Enterprise is planned for a future release.<\/p>\n <\/p>\n How many credits does it cost to use Data Pipelines?<\/strong><\/p>\n At the time of the 2024 Esri User Conference, the cost of working in Data Pipelines interactively (in the editor when building a data pipeline) is 50 credits per hour, on a per-minute basis, with a 10-minute minimum. For scheduled tasks, used to run a data pipeline at the interval of your choosing, the cost is 70 credits per hour, also charged on a per minute bases, but with no minimum charge.<\/p>\n For example: if you spend 30 minutes working in the editor building a data pipeline, you will be charged 25 credits. If a scheduled run takes 4 minutes to run, you will be charged 5 credits.<\/p>\n Equate credits to the currency of your choosing and then compare the cost of using Data Pipelines to the cost of hiring a developer to author and maintain a python script, or the time it takes for a GIS analyst to complete the process manually on a repeated basis: the value of Data Pipelines will be quickly realized!<\/p>\n For more information on credits in ArcGIS Online, please review\u202f this documentation<\/span><\/a>.<\/p>\n <\/p>\n I am new to Data Pipelines. How do I get started?<\/strong><\/p>\n A great place to start is in our product documentation. That said, there are a lot of other resources that can be helpful for both new and more experienced users alike. Here are a few to start with:<\/span><\/p>\n <\/p>\n\n
\n
\n
\n