{"id":2280432,"date":"2024-03-11T13:21:14","date_gmt":"2024-03-11T20:21:14","guid":{"rendered":"https:\/\/www.esri.com\/arcgis-blog\/?post_type=blog&p=2280432"},"modified":"2025-02-12T05:18:11","modified_gmt":"2025-02-12T13:18:11","slug":"dev-summit-2024-arcgis-data-pipelines","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines","title":{"rendered":"Dev Summit 2024: ArcGIS Data Pipelines"},"author":207622,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_searchwp_excluded":""},"categories":[23341,23851,37151],"tags":[776052,42641],"industry":[],"product":[36841,36551],"class_list":["post-2280432","blog","type-blog","status-publish","format-standard","hentry","category-analytics","category-data-management","category-design-planning","tag-arcgis-data-pipelines","tag-arcgis-online","product-api-python","product-arcgis-online"],"acf":{"authors":[{"ID":207622,"user_firstname":"Aawaj","user_lastname":"Joshi","nickname":"Aawaj Joshi","user_nicename":"ajoshi","display_name":"Aawaj Joshi","user_email":"ajoshi@esri.com","user_url":"","user_registered":"2021-03-25 20:59:02","user_description":"Aawaj is a Product Engineer on the ArcGIS Enterprise team.","user_avatar":""},{"ID":9832,"user_firstname":"Max","user_lastname":"Payson","nickname":"Max Payson","user_nicename":"mpayson","display_name":"Max Payson","user_email":"MPayson@esri.com","user_url":"https:\/\/github.com\/mpayson\/","user_registered":"2019-08-02 18:26:50","user_description":"","user_avatar":""},{"ID":7431,"user_firstname":"Bethany","user_lastname":"Scott","nickname":"Bethany Scott","user_nicename":"bethanyscott","display_name":"Bethany Scott","user_email":"BScott@esri.com","user_url":"","user_registered":"2018-03-21 18:21:18","user_description":"Bethany (she\/her) is a Product Engineer on the Data Pipelines team and the GeoAnalytics team. Her background is in biology and GIS with experience in data management and spatial-temporal analysis.","user_avatar":""}],"short_description":"ArcGIS Data Pipelines is a new ArcGIS Online application that allows you to access, process, and integrate data from a wide variety of sources.","flexible_content":[{"acf_fc_layout":"content","content":"

ArcGIS Data Pipelines<\/a> is a new ArcGIS Online application that allows you to access, process, and integrate data from a wide variety of sources. It offers an intuitive drag-and-drop interface where you can create reproducible data preparation workflows without writing any code.<\/p>\n"},{"acf_fc_layout":"content","content":"

At this year’s Developer Summit plenary, Max Payson offers a glimpse of what can be accomplished with ArcGIS Data Pipelines.<\/p>\n"},{"acf_fc_layout":"content","content":"

As an analyst at a retail company, Max is tasked with aggregating relevant datasets that can help identify possible locations for new stores in New York City. He has two datasets to work with: the first is a report categorizing different levels of pedestrian demand, and the second is data on open streets, which are streets transformed into public spaces for various activities. The former dataset is stored in his organization’s Amazon S3<\/a> bucket, while the latter is public data accessible via URL<\/a>. Together, these datasets can help visualize foot traffic in streets across the city.<\/p>\n"},{"acf_fc_layout":"content","content":"

Max shows a data pipeline that’s configured to do the following:<\/p>\n

    \n
  1. Connect directly to the pedestrian mobility dataset stored in the Amazon S3 bucket and create a geometry<\/a> field using well-known text (WKT) values. The resulting geometry serves as the target dataset for the join.<\/li>\n
  2. Read the open streets locations data using the public URL and create a new dataset containing only pertinent fields. The resulting dataset serves as the join dataset.<\/li>\n
  3. Join<\/a> the target and join datasets in a one-to-many relationship based on matching field values.<\/li>\n
  4. Create a unified streets feature layer<\/a> based on the join results.<\/li>\n<\/ol>\n"},{"acf_fc_layout":"image","image":{"ID":2281272,"id":2281272,"title":"Max's preconfigured data pipeline to create a unified streets feature layer.","filename":"Pipelines-interface.png","filesize":162465,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-interface.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\/pipelines-interface","alt":"Max's preconfigured data pipeline to create a unified streets feature layer.","author":"207622","description":"Max's preconfigured data pipeline to create a unified streets feature layer.","caption":"Max's preconfigured data pipeline to create a unified streets feature layer.","name":"pipelines-interface","status":"inherit","uploaded_to":2280432,"date":"2024-03-11 18:53:09","modified":"2024-03-11 20:16:41","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1601,"height":697,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-interface-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-interface.png","medium-width":464,"medium-height":202,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-interface.png","medium_large-width":768,"medium_large-height":334,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-interface.png","large-width":1601,"large-height":697,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-interface-1536x669.png","1536x1536-width":1536,"1536x1536-height":669,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-interface.png","2048x2048-width":1601,"2048x2048-height":697,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-interface-826x360.png","card_image-width":826,"card_image-height":360,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-interface.png","wide_image-width":1601,"wide_image-height":697}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

    Max runs the data pipeline and, within seconds, obtains a feature layer with approximately 14000 records.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2281282,"id":2281282,"title":"The data pipeline generates a feature layer with approximately 14000 records.","filename":"Pipelines-output.png","filesize":201376,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-output.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\/pipelines-output","alt":"The data pipeline generates a feature layer with approximately 14000 records.","author":"207622","description":"The data pipeline generates a feature layer with approximately 14000 records.","caption":"The data pipeline generates a feature layer with approximately 14000 records.","name":"pipelines-output","status":"inherit","uploaded_to":2280432,"date":"2024-03-11 18:54:00","modified":"2024-03-11 20:16:36","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1318,"height":663,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-output-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-output.png","medium-width":464,"medium-height":233,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-output.png","medium_large-width":768,"medium_large-height":386,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-output.png","large-width":1318,"large-height":663,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-output.png","1536x1536-width":1318,"1536x1536-height":663,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-output.png","2048x2048-width":1318,"2048x2048-height":663,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-output-826x416.png","card_image-width":826,"card_image-height":416,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-output.png","wide_image-width":1318,"wide_image-height":663}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

    Since the NYC Open Data team updates the dataset of open street locations monthly, Max explores two options to keep his feature layer data up-to-date with minimal intervention. First, he shows the scheduling<\/a> feature built into ArcGIS Data Pipelines, which lets you create scheduled tasks that run once or on a recurring basis.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2281202,"id":2281202,"title":"The scheduling feature built into ArcGIS Data Pipelines.","filename":"Pipelines-schedule.png","filesize":82323,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-schedule.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\/pipelines-schedule","alt":"The scheduling feature built into ArcGIS Data Pipelines.","author":"207622","description":"The scheduling feature built into ArcGIS Data Pipelines.","caption":"The scheduling feature built into ArcGIS Data Pipelines.","name":"pipelines-schedule","status":"inherit","uploaded_to":2280432,"date":"2024-03-11 18:17:43","modified":"2024-03-11 20:16:27","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":877,"height":777,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-schedule-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-schedule.png","medium-width":295,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-schedule.png","medium_large-width":768,"medium_large-height":680,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-schedule.png","large-width":877,"large-height":777,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-schedule.png","1536x1536-width":877,"1536x1536-height":777,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-schedule.png","2048x2048-width":877,"2048x2048-height":777,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-schedule-525x465.png","card_image-width":525,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-schedule.png","wide_image-width":877,"wide_image-height":777}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

    Next, he shows a Jupyter notebook that utilizes ArcGIS API for Python to update his feature layer data programmatically. The notebook sequentially performs the following operations: fetch the data pipeline item using the gis.content.get()<\/strong> method, run the pipeline using the run_data_pipeline()<\/strong> method, get the result using the result()<\/strong> method, obtain the required output from the result by accessing its outputs<\/strong> property, and extract the new feature layer from the output array.<\/p>\n

    Note that Data Pipelines in the Python API is still in the experimental phase and not yet available for use in ArcGIS Notebooks. To use Data Pipelines in the Python API locally, you must install ArcGIS API for Python version 2.3.0 when it becomes available.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2281312,"id":2281312,"title":"Max's Jupyter notebook utilizes the ArcGIS API for Python to update his feature layer data programmatically.","filename":"Pipelines-notebook.png","filesize":275660,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-notebook.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\/pipelines-notebook","alt":"Max's Jupyter notebook utilizes the ArcGIS API for Python to update his feature layer data programmatically.","author":"207622","description":"Max's Jupyter notebook utilizes the ArcGIS API for Python to update his feature layer data programmatically.","caption":"Max's Jupyter notebook utilizes the ArcGIS API for Python to update his feature layer data programmatically.","name":"pipelines-notebook","status":"inherit","uploaded_to":2280432,"date":"2024-03-11 18:55:33","modified":"2024-03-11 20:15:47","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":962,"height":851,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-notebook-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-notebook.png","medium-width":295,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-notebook.png","medium_large-width":768,"medium_large-height":679,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-notebook.png","large-width":962,"large-height":851,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-notebook.png","1536x1536-width":962,"1536x1536-height":851,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-notebook.png","2048x2048-width":962,"2048x2048-height":851,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-notebook-526x465.png","card_image-width":526,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-notebook.png","wide_image-width":962,"wide_image-height":851}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

    With the feature layer already hosted on ArcGIS Online, Max quickly visualizes it to show that streets across the city have been categorized based on pedestrian mobility demand, emphasizing the open streets.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2281322,"id":2281322,"title":"The resultant feature layer includes street data categorized based on pedestrian mobility demand.","filename":"Pipelines-result.png","filesize":823168,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-result.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\/pipelines-result","alt":"The resultant feature layer includes street data categorized based on pedestrian mobility demand.","author":"207622","description":"The resultant feature layer includes street data categorized based on pedestrian mobility demand.","caption":"The resultant feature layer includes street data categorized based on pedestrian mobility demand.","name":"pipelines-result","status":"inherit","uploaded_to":2280432,"date":"2024-03-11 18:56:03","modified":"2024-03-11 20:20:20","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1403,"height":847,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-result-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-result.png","medium-width":432,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-result.png","medium_large-width":768,"medium_large-height":464,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-result.png","large-width":1403,"large-height":847,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-result.png","1536x1536-width":1403,"1536x1536-height":847,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-result.png","2048x2048-width":1403,"2048x2048-height":847,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-result-770x465.png","card_image-width":770,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-result.png","wide_image-width":1403,"wide_image-height":847}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

    Using ArcGIS Data Pipelines, Max seamlessly filtered and enhanced datasets from two disparate sources to produce data that others in his organization can leverage in their site selection workflows. To learn more about Data Pipelines, visit the product page<\/a>\u00a0or check out the ArcGIS Data Pipelines is Now Available<\/a> blog.<\/p>\n"}],"related_articles":"","card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-card.png","wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-banner.png","show_article_image":false},"yoast_head":"\nDev Summit 2024: ArcGIS Data Pipelines<\/title>\n<meta name=\"description\" content=\"At this year\u2019s Developer Summit plenary, Max Payson offers a glimpse of what can be accomplished with ArcGIS Data Pipelines.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Dev Summit 2024: ArcGIS Data Pipelines\" \/>\n<meta property=\"og:description\" content=\"At this year\u2019s Developer Summit plenary, Max Payson offers a glimpse of what can be accomplished with ArcGIS Data Pipelines.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\" \/>\n<meta property=\"og:site_name\" content=\"ArcGIS Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/esrigis\/\" \/>\n<meta property=\"article:modified_time\" content=\"2025-02-12T13:18:11+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@ESRI\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":[\"Article\",\"BlogPosting\"],\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\"},\"author\":{\"name\":\"Aawaj Joshi\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/1a05fe13e7f3a2c7350529406f1ad821\"},\"headline\":\"Dev Summit 2024: ArcGIS Data Pipelines\",\"datePublished\":\"2024-03-11T20:21:14+00:00\",\"dateModified\":\"2025-02-12T13:18:11+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\"},\"wordCount\":5,\"publisher\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#organization\"},\"keywords\":[\"ArcGIS Data Pipelines\",\"ArcGIS Online\"],\"articleSection\":[\"Analytics\",\"Data Management\",\"Design and Planning\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\",\"name\":\"Dev Summit 2024: ArcGIS Data Pipelines\",\"isPartOf\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#website\"},\"datePublished\":\"2024-03-11T20:21:14+00:00\",\"dateModified\":\"2025-02-12T13:18:11+00:00\",\"description\":\"At this year\u2019s Developer Summit plenary, Max Payson offers a glimpse of what can be accomplished with ArcGIS Data Pipelines.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.esri.com\/arcgis-blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Dev Summit 2024: ArcGIS Data Pipelines\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#website\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/\",\"name\":\"ArcGIS Blog\",\"description\":\"Get insider info from Esri product teams\",\"publisher\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.esri.com\/arcgis-blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#organization\",\"name\":\"Esri\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2018\/04\/Esri.png\",\"contentUrl\":\"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2018\/04\/Esri.png\",\"width\":400,\"height\":400,\"caption\":\"Esri\"},\"image\":{\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/esrigis\/\",\"https:\/\/x.com\/ESRI\",\"https:\/\/www.linkedin.com\/company\/5311\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/1a05fe13e7f3a2c7350529406f1ad821\",\"name\":\"Aawaj Joshi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/03\/IMG_6218-1-1-465x465.png\",\"contentUrl\":\"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/03\/IMG_6218-1-1-465x465.png\",\"caption\":\"Aawaj Joshi\"},\"description\":\"Aawaj is a Product Engineer on the ArcGIS Enterprise team.\",\"url\":\"https:\/\/www.esri.com\/arcgis-blog\/author\/ajoshi\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Dev Summit 2024: ArcGIS Data Pipelines","description":"At this year\u2019s Developer Summit plenary, Max Payson offers a glimpse of what can be accomplished with ArcGIS Data Pipelines.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines","og_locale":"en_US","og_type":"article","og_title":"Dev Summit 2024: ArcGIS Data Pipelines","og_description":"At this year\u2019s Developer Summit plenary, Max Payson offers a glimpse of what can be accomplished with ArcGIS Data Pipelines.","og_url":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines","og_site_name":"ArcGIS Blog","article_publisher":"https:\/\/www.facebook.com\/esrigis\/","article_modified_time":"2025-02-12T13:18:11+00:00","twitter_card":"summary_large_image","twitter_site":"@ESRI","twitter_misc":{"Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["Article","BlogPosting"],"@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines#article","isPartOf":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines"},"author":{"name":"Aawaj Joshi","@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/1a05fe13e7f3a2c7350529406f1ad821"},"headline":"Dev Summit 2024: ArcGIS Data Pipelines","datePublished":"2024-03-11T20:21:14+00:00","dateModified":"2025-02-12T13:18:11+00:00","mainEntityOfPage":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines"},"wordCount":5,"publisher":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/#organization"},"keywords":["ArcGIS Data Pipelines","ArcGIS Online"],"articleSection":["Analytics","Data Management","Design and Planning"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines","url":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines","name":"Dev Summit 2024: ArcGIS Data Pipelines","isPartOf":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/#website"},"datePublished":"2024-03-11T20:21:14+00:00","dateModified":"2025-02-12T13:18:11+00:00","description":"At this year\u2019s Developer Summit plenary, Max Payson offers a glimpse of what can be accomplished with ArcGIS Data Pipelines.","breadcrumb":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.esri.com\/arcgis-blog\/"},{"@type":"ListItem","position":2,"name":"Dev Summit 2024: ArcGIS Data Pipelines"}]},{"@type":"WebSite","@id":"https:\/\/www.esri.com\/arcgis-blog\/#website","url":"https:\/\/www.esri.com\/arcgis-blog\/","name":"ArcGIS Blog","description":"Get insider info from Esri product teams","publisher":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.esri.com\/arcgis-blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.esri.com\/arcgis-blog\/#organization","name":"Esri","url":"https:\/\/www.esri.com\/arcgis-blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2018\/04\/Esri.png","contentUrl":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2018\/04\/Esri.png","width":400,"height":400,"caption":"Esri"},"image":{"@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/esrigis\/","https:\/\/x.com\/ESRI","https:\/\/www.linkedin.com\/company\/5311\/"]},{"@type":"Person","@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/1a05fe13e7f3a2c7350529406f1ad821","name":"Aawaj Joshi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.esri.com\/arcgis-blog\/#\/schema\/person\/image\/","url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/03\/IMG_6218-1-1-465x465.png","contentUrl":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2026\/03\/IMG_6218-1-1-465x465.png","caption":"Aawaj Joshi"},"description":"Aawaj is a Product Engineer on the ArcGIS Enterprise team.","url":"https:\/\/www.esri.com\/arcgis-blog\/author\/ajoshi"}]}},"text_date":"March 11, 2024","author_name":"Multiple Authors","author_page":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-online\/analytics\/dev-summit-2024-arcgis-data-pipelines","custom_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/Pipelines-banner.png","primary_product":"ArcGIS Online","tag_data":[{"term_id":776052,"name":"ArcGIS Data Pipelines","slug":"arcgis-data-pipelines","term_group":0,"term_taxonomy_id":776052,"taxonomy":"post_tag","description":"","parent":0,"count":16,"filter":"raw"},{"term_id":42641,"name":"ArcGIS Online","slug":"arcgis-online","term_group":0,"term_taxonomy_id":42641,"taxonomy":"post_tag","description":"","parent":0,"count":420,"filter":"raw"}],"category_data":[{"term_id":23341,"name":"Analytics","slug":"analytics","term_group":0,"term_taxonomy_id":23341,"taxonomy":"category","description":"","parent":0,"count":1334,"filter":"raw"},{"term_id":23851,"name":"Data Management","slug":"data-management","term_group":0,"term_taxonomy_id":23851,"taxonomy":"category","description":"","parent":0,"count":927,"filter":"raw"},{"term_id":37151,"name":"Design and Planning","slug":"design-planning","term_group":0,"term_taxonomy_id":37151,"taxonomy":"category","description":"","parent":0,"count":118,"filter":"raw"}],"product_data":[{"term_id":36841,"name":"ArcGIS API for Python","slug":"api-python","term_group":0,"term_taxonomy_id":36841,"taxonomy":"product","description":"","parent":36601,"count":151,"filter":"raw"},{"term_id":36551,"name":"ArcGIS Online","slug":"arcgis-online","term_group":0,"term_taxonomy_id":36551,"taxonomy":"product","description":"","parent":0,"count":2435,"filter":"raw"}],"primary_product_link":"https:\/\/www.esri.com\/arcgis-blog\/?s=#&products=arcgis-online","_links":{"self":[{"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/blog\/2280432","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/blog"}],"about":[{"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/types\/blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/users\/207622"}],"replies":[{"embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/comments?post=2280432"}],"version-history":[{"count":0,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/blog\/2280432\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/media?parent=2280432"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/categories?post=2280432"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/tags?post=2280432"},{"taxonomy":"industry","embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/industry?post=2280432"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/www.esri.com\/arcgis-blog\/wp-json\/wp\/v2\/product?post=2280432"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}