{"id":1870552,"date":"2023-03-08T19:06:35","date_gmt":"2023-03-09T03:06:35","guid":{"rendered":"https:\/\/www.esri.com\/arcgis-blog\/?post_type=blog&p=1870552"},"modified":"2023-03-10T12:49:29","modified_gmt":"2023-03-10T20:49:29","slug":"dev-summit-2023-automate-image-data-management-workflows","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/data-management\/dev-summit-2023-automate-image-data-management-workflows","title":{"rendered":"Dev Summit 2023: Automate image data management workflows"},"author":15561,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_searchwp_excluded":""},"categories":[37501,23851,738191],"tags":[387782,42301,555752,31181,86442],"industry":[],"product":[36571],"class_list":["post-1870552","blog","type-blog","status-publish","format-standard","hentry","category-administration","category-data-management","category-developers","tag-arcgis-api-for-python","tag-arcgis-enterprise","tag-arcgis-notebooks","tag-arcpy","tag-data-management","product-arcgis-enterprise"],"acf":{"authors":[{"ID":15561,"user_firstname":"Tori","user_lastname":"O hara","nickname":"Tori O'Hara","user_nicename":"tohara","display_name":"Tori O'Hara","user_email":"TOhara@esri.com","user_url":"","user_registered":"2020-03-09 20:04:56","user_description":"Tori is a technical writer on the ArcGIS Enterprise team.","user_avatar":""}],"short_description":"ArcGIS Enterprise has a variety of tools that can help organization automate their image data management workflows.","flexible_content":[{"acf_fc_layout":"content","content":"

The ArcGIS system is a platform <\/span>built <\/span>for automation. <\/span>GIS <\/span>organizations using ArcGIS products, such as ArcGIS Enterprise, have access to various tools and systems that can automate preexisting workflows<\/span>. In his demonstration, Jie shows how an ArcGIS Enterprise organization with ArcGIS Image Server can leverage the functionalities of<\/span> ArcPy<\/span>, <\/span>ArcGIS API for Python<\/span>,<\/span> and ArcGIS Notebooks to automate<\/span> image data management.<\/span><\/span><\/p>\n"},{"acf_fc_layout":"kaltura","video_id":"1_kk12l8t8","time":true,"start":"839","stop":"1130"},{"acf_fc_layout":"content","content":"

Gathering data for the image service<\/span><\/span><\/h2>\n

For this demonstration, Jie created an image service published from <\/span>mosaic datasets. The mosaic datasets reference live, cloud-based weather forecast data for the continental United State<\/span>s<\/span>. Specifically, the mosaic datasets reference wind forecast data<\/span>, which<\/span> contains information such as wind direction and speed values<\/span>. These values<\/span> can be interpolated <\/span>by ArcGIS Enterprise <\/span>and then visualized using <\/span>vector symbology.<\/span><\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1870562,"id":1870562,"title":"ArcGIS Pro wind symbology","filename":"pro-wind-symbology.png","filesize":1259884,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/pro-wind-symbology.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/data-management\/dev-summit-2023-automate-image-data-management-workflows\/pro-wind-symbology","alt":"","author":"15561","description":"","caption":"Figure 1: An example of how vector symbology is used to visualize wind speed and direction data in ArcGIS Pro.","name":"pro-wind-symbology","status":"inherit","uploaded_to":1870552,"date":"2023-03-09 02:26:46","modified":"2023-03-09 02:29:02","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1500,"height":816,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/pro-wind-symbology-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/pro-wind-symbology.png","medium-width":464,"medium-height":252,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/pro-wind-symbology.png","medium_large-width":768,"medium_large-height":418,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/pro-wind-symbology.png","large-width":1500,"large-height":816,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/pro-wind-symbology.png","1536x1536-width":1500,"1536x1536-height":816,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/pro-wind-symbology.png","2048x2048-width":1500,"2048x2048-height":816,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/pro-wind-symbology-826x449.png","card_image-width":826,"card_image-height":449,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/pro-wind-symbology.png","wide_image-width":1500,"wide_image-height":816}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Jie is referencing forecast data from the <\/span><\/span>NOAA National Digital Forecast Database (NDFD)<\/span><\/span><\/a>. The NOAA NDFD <\/span>uses<\/span> public AWS S3 cloud storage that contains data files of up to 7 days of weather forecast data<\/span>. <\/span>These data files are<\/span> constantly updating, which requires Jie\u2019s image service to <\/span>update<\/span> its <\/span>reference<\/span>s to include <\/span>the newest data<\/span>. To accomplish this, Jie created two scripts<\/span> in ArcGIS Notebooks<\/span>: one to create an image service that references these data files, and another to update <\/span>those<\/span> references whenever new data is available.\u00a0<\/span><\/span><\/p>\n

Creating the image service<\/h2>\n

Jie\u2019s first script <\/span>has three main steps to create his image service. First, his script creates the cloud <\/span>storge<\/span> connection file that points to <\/span>NOAA <\/span>NDFD\u2019s public S3 bucket.<\/span><\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1870572,"id":1870572,"title":"ArcGIS Notebooks script for configuration file","filename":"script-one_confguration-file.png","filesize":400887,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_confguration-file.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/data-management\/dev-summit-2023-automate-image-data-management-workflows\/script-one_confguration-file","alt":"","author":"15561","description":"","caption":"Figure 2: An script in ArcGIS Notebooks. This specific step in the script creates a cloud storage configuration file.","name":"script-one_confguration-file","status":"inherit","uploaded_to":1870552,"date":"2023-03-09 02:30:21","modified":"2023-03-09 02:34:48","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1488,"height":760,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_confguration-file-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_confguration-file.png","medium-width":464,"medium-height":237,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_confguration-file.png","medium_large-width":768,"medium_large-height":392,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_confguration-file.png","large-width":1488,"large-height":760,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_confguration-file.png","1536x1536-width":1488,"1536x1536-height":760,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_confguration-file.png","2048x2048-width":1488,"2048x2048-height":760,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_confguration-file-826x422.png","card_image-width":826,"card_image-height":422,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_confguration-file.png","wide_image-width":1488,"wide_image-height":760}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Once a connection <\/span>has <\/span>been established, his script creates the mosaic dataset that references the wind forecast data by calling the <\/span><\/span>Create Mosaic Dataset<\/span><\/span><\/a> and <\/span><\/span>Add Raster to Mosaic Dataset<\/span><\/span><\/a> tools<\/span>. Because Jie\u2019s script created a cloud storage connection file, this step will only reference the data, rather than copying <\/span>it<\/span> to ArcGIS Image Server.<\/span><\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1870582,"id":1870582,"title":"ArcGIS Notebooks script that creates a mosaic dataset","filename":"script-one_mosaic.png","filesize":554080,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_mosaic.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/data-management\/dev-summit-2023-automate-image-data-management-workflows\/script-one_mosaic","alt":"","author":"15561","description":"","caption":"Figure 3: A script in ArcGIS Notebooks. This step of the script creates a mosaic dataset that references public weather forecast data.","name":"script-one_mosaic","status":"inherit","uploaded_to":1870552,"date":"2023-03-09 02:32:33","modified":"2023-03-09 02:35:05","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1498,"height":764,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_mosaic-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_mosaic.png","medium-width":464,"medium-height":237,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_mosaic.png","medium_large-width":768,"medium_large-height":392,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_mosaic.png","large-width":1498,"large-height":764,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_mosaic.png","1536x1536-width":1498,"1536x1536-height":764,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_mosaic.png","2048x2048-width":1498,"2048x2048-height":764,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_mosaic-826x421.png","card_image-width":826,"card_image-height":421,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_mosaic.png","wide_image-width":1498,"wide_image-height":764}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Finally, the script creates the image service using the ArcGIS API for Python <\/span><\/span>Create Service<\/span><\/span><\/a> metho<\/span>d. Since his service is referencing live data, the service <\/span>has<\/span> the <\/span>live data<\/span> flag enabled<\/span>.<\/span><\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1870592,"id":1870592,"title":"ArcGIS Notebook script that creates an image service.","filename":"script-one_create.png","filesize":424873,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_create.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/data-management\/dev-summit-2023-automate-image-data-management-workflows\/script-one_create","alt":"","author":"15561","description":"","caption":"Figure 4: A script in ArcGIS Notebooks. This step of the script that creates an image service that has live data.","name":"script-one_create","status":"inherit","uploaded_to":1870552,"date":"2023-03-09 02:35:44","modified":"2023-03-09 02:37:05","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1490,"height":762,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_create-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_create.png","medium-width":464,"medium-height":237,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_create.png","medium_large-width":768,"medium_large-height":393,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_create.png","large-width":1490,"large-height":762,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_create.png","1536x1536-width":1490,"1536x1536-height":762,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_create.png","2048x2048-width":1490,"2048x2048-height":762,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_create-826x422.png","card_image-width":826,"card_image-height":422,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/script-one_create.png","wide_image-width":1490,"wide_image-height":762}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Jie shows us his result imagery layer, which has the most recent 7-day wind forecast data. Now that Jie\u2019s created his image service, he needs to keep it<\/span>s data references<\/span> up to date<\/span>.<\/span><\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1870602,"id":1870602,"title":"Image layer showing 7 day wind forecast data","filename":"wind-result.png","filesize":1211111,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/wind-result.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/data-management\/dev-summit-2023-automate-image-data-management-workflows\/wind-result","alt":"","author":"15561","description":"","caption":"Figure 5: A imagery layer showing the most-recent, 7-day wind forecast data.","name":"wind-result","status":"inherit","uploaded_to":1870552,"date":"2023-03-09 02:37:50","modified":"2023-03-09 02:39:15","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1496,"height":766,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/wind-result-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/wind-result.png","medium-width":464,"medium-height":238,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/wind-result.png","medium_large-width":768,"medium_large-height":393,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/wind-result.png","large-width":1496,"large-height":766,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/wind-result.png","1536x1536-width":1496,"1536x1536-height":766,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/wind-result.png","2048x2048-width":1496,"2048x2048-height":766,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/wind-result-826x423.png","card_image-width":826,"card_image-height":423,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2023\/03\/wind-result.png","wide_image-width":1496,"wide_image-height":766}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Updating the image service<\/span><\/span><\/h2>\n

Jie\u2019s second script updates his image <\/span>service<\/span> by performing two functions:<\/span><\/span><\/p>\n