Acronyms referenced in this post:<\/p>\n
In this post, we will cover how to take Esri\u2019s pretrained deep learning models (available through ArcGIS Living Atlas of the World) and refine them with your own training data. This fine-tuning process can increase accuracy in detecting objects or making classifications, by tailoring the model to fit your geography and your imagery source characteristics, including the resolution, bit depth, and number of bands. If you are unfamiliar with the process of setting up a deep learning environment, sizing the relevant and right GPU for your workflow, and running inferencing with ArcGIS Pro, it is recommended that you catch up with the previous blog posts on deep learning before diving into this guide: Deep Learning with ArcGIS Pro Tips & Tricks: <\/span>Part 1<\/span><\/a>, <\/span>Part 2<\/span><\/a>, and <\/span>Part 3<\/span><\/a>.<\/span>\u00a0<\/span><\/p>\n Interest in deep learning is growing across many sectors that run the gamut of agriculture, natural resources, defense, and more. Esri continues to invest resources to make deep learning accessible to all users, expanding deep learning capabilities across the ArcGIS system.\u00a0<\/span>\u00a0<\/span><\/p>\n There are three main workflows for using deep learning within ArcGIS:\u00a0<\/span>\u00a0<\/span><\/p>\n For a detailed guide on the first workflow, using the pretrained models, see <\/span>Deep Learning with ArcGIS Pro Tips & Tricks Part 2<\/span><\/a>.\u00a0<\/span>\u00a0<\/span><\/p>\n In this blog post, we will cover how to fine-tune Esri\u2019s existing pretrained deep learning models to fit to your local geography, imagery<\/a>, or features of interest. This process will take less data, compute resources, and training time than training a new model from scratch.<\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"content","content":" Deep learning models are constrained by the data used to train them. A model trained on Landsat imagery over New York City cannot be expected to return quality results when run on high-resolution imagery over rural Brazil. To get the best outcome from inferencing, the initial training data should be as consistent as possible with the area of interest (AOI). Factors to consider include the landscape, climate, construction patterns (if identifying buildings, are the roofs made of similar materials?), season, and imagery quality and resolution. The pretrained models in ArcGIS Living Atlas are designed to be more broadly applicable than a model you would train yourself, but you may find that none of the models in ArcGIS Living Atlas directly align with your AOI. In this scenario, your options are to create a model from scratch, or fine-tune the existing model with new training data to improve the result. If the initial model couldn\u2019t identify any of the features you were looking for or you had a completely blank output, you likely need to train a new model. But if the model is identifying a few of the correct features, even if it is doing a poor job, it is worth fine-tuning the existing model, as this is the cheaper and faster option. You can also create training data to fine-tune the model and repurpose that training data to create a model from scratch if you are still not achieving the desired results.\u00a0 Below, we will walk through an example in which fine-tuning an existing model was effective for getting an accurate result.\u00a0<\/span>\u00a0<\/span><\/p>\n Below is a map of Cox\u2019s Bazar, Bangladesh, showing drone imagery provided by the United Nations International Organization for Migration (<\/span>IOM<\/span><\/a>). The goal of this project was to extract the locations of Rohingya refugee tents so that aid workers could better track the number of people living in the refugee camp and how they were moving over time. To extract the tent footprints, we first ran Esri\u2019s pretrained, out-of-the-box <\/span>Building Footprint Extraction – Africa<\/span><\/a> dlpk using the <\/span>Detect Objects Using Deep Learning<\/span><\/a> geoprocessing tool. The initial result is shown below in blue.\u00a0\u00a0<\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1465742,"id":1465742,"title":"Pre-Trained-Model-Cox's Bazar","filename":"Pre-Trained-Africa-Coxs-Bazar.png","filesize":1240884,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/Pre-Trained-Africa-Coxs-Bazar.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/imagery\/fine-tune-a-pretrained-deep-learning-model\/pre-trained-africa-coxs-bazar","alt":"Pre-Trained Refugee Tents","author":"19291","description":"","caption":"Figure 1: Out-of-the-box dlpk results.","name":"pre-trained-africa-coxs-bazar","status":"inherit","uploaded_to":1463912,"date":"2022-01-24 15:46:24","modified":"2022-01-24 16:22:29","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":998,"height":668,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/Pre-Trained-Africa-Coxs-Bazar-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/Pre-Trained-Africa-Coxs-Bazar.png","medium-width":390,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/Pre-Trained-Africa-Coxs-Bazar.png","medium_large-width":768,"medium_large-height":514,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/Pre-Trained-Africa-Coxs-Bazar.png","large-width":998,"large-height":668,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/Pre-Trained-Africa-Coxs-Bazar.png","1536x1536-width":998,"1536x1536-height":668,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/Pre-Trained-Africa-Coxs-Bazar.png","2048x2048-width":998,"2048x2048-height":668,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/Pre-Trained-Africa-Coxs-Bazar-695x465.png","card_image-width":695,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/Pre-Trained-Africa-Coxs-Bazar.png","wide_image-width":998,"wide_image-height":668}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" The model is returning some of the tent footprints, but many have been missed, especially in the lower left of the image. Some extra work is needed to improve the output of the deep learning model, so we will use the Building Footprints Extraction \u2013 Africa model as a base and fine-tune it to identify the tents in our drone imagery.<\/p>\n <\/p>\n First, verify that you have the necessary hardware, software, and libraries.<\/p>\n Verify that the machine you are working with has an appropriate GPU. Training a deep learning model is more hardware demanding than inferencing (running the model). The bare minimum you will need for fine-tuning a deep learning model is a NVIDIA RTX<\/a> or NVIDIA Quadro<\/a> with 8 GB of RAM, or something equivalent. This will allow you to train a model for a small-scale proof of concept, however, if you are preparing for a production workflow, you should look for a higher-performing GPU, such as a NVIDIA V100<\/a> GPU with a minimum of 16 GB of RAM. For more information about GPU-powered production environments, check the NVIDIA Data Center GPUs<\/a>.<\/strong><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1465802,"id":1465802,"title":"fig2","filename":"fig2.jpg","filesize":26303,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig2.jpg","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/imagery\/fine-tune-a-pretrained-deep-learning-model\/fig2-8","alt":"graph of NVIDIA GPUs with Tensor Cores.","author":"137201","description":"","caption":"Figure 2: NVIDIA GPUs with Tensor Cores.","name":"fig2-8","status":"inherit","uploaded_to":1463912,"date":"2022-01-24 16:23:38","modified":"2022-01-24 16:24:02","menu_order":0,"mime_type":"image\/jpeg","type":"image","subtype":"jpeg","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":630,"height":354,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig2-213x200.jpg","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig2.jpg","medium-width":464,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig2.jpg","medium_large-width":630,"medium_large-height":354,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig2.jpg","large-width":630,"large-height":354,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig2.jpg","1536x1536-width":630,"1536x1536-height":354,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig2.jpg","2048x2048-width":630,"2048x2048-height":354,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig2.jpg","card_image-width":630,"card_image-height":354,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig2.jpg","wide_image-width":630,"wide_image-height":354}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" CUDA toolkit is another essential component to training your deep learning model. CUDA\u00ae is a parallel computing platform and programming model developed by NVIDIA for general computing on GPUs. With CUDA, developers can speed up computing applications with the power of GPUs<\/span>.<\/span>\u00a0To install CUDA, visit the <\/span>CUDA Toolkit download page<\/span><\/a>.<\/span>\u00a0<\/span><\/p>\n To train or fine-tune a deep learning model within ArcGIS, you will need ArcGIS Pro and an ArcGIS <\/span>Image Analyst license<\/span><\/a>. For more details about this topic, visit the <\/span>Deep learning in ArcGIS Pro<\/span><\/a> page.\u00a0<\/span>\u00a0<\/span><\/p>\n Finally, you will need to install all the dependent libraries for a deep learning workflow. Esri has packaged these in the <\/span>deep learning frameworks for ArcGIS Pro<\/span><\/a>. Note that each version of ArcGIS Pro requires specific versions of deep learning libraries. When you upgrade ArcGIS Pro, you need to install the deep learning libraries that correspond to that version of ArcGIS Pro. For the list of libraries required at each version along with other information, see <\/span>Deep learning frequently asked questions<\/span><\/a>.\u00a0<\/span>\u00a0<\/span><\/p>\n For more detailed instructions on the prerequisites for deep learning in ArcGIS Pro, refer to the blog post <\/span>Deep Learning with ArcGIS Pro Tips & Tricks: Part 1<\/span><\/a>,<\/span> Deep learning prerequisites section.<\/span>\u00a0<\/span><\/p>\n You are now ready to fine-tune an ArcGIS deep learning model (dlpk). For the example workflow, we will be using the <\/span>Building Footprint Extraction – Africa<\/span><\/a> dlpk, but the same methodology can be applied to any other dlpk.<\/span>\u00a0<\/span><\/p>\n To add new training data to a model, you will need to understand how and with what the original model was trained. Most of the time, this information is included in the dlpk\u2019s metadata or description. If you cannot find the information in the model description on ArcGIS Online or want more information, follow the steps below to access the full dlpk metadata.\u00a0<\/span>\u00a0<\/span><\/p>\n Now that we have determined the parameters to use for our <\/span>f<\/span>ine-<\/span>tuning<\/span>, we are ready to create and export the <\/span>training data for deep learning.\u00a0<\/span><\/span>\u00a0<\/span><\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1465812,"id":1465812,"title":"fig3","filename":"fig3-2.png","filesize":25896,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig3-2.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/imagery\/fine-tune-a-pretrained-deep-learning-model\/fig3-7","alt":"contents of the emd file","author":"137201","description":"","caption":"Figure 3: Contents of the .emd file. ","name":"fig3-7","status":"inherit","uploaded_to":1463912,"date":"2022-01-24 16:26:07","modified":"2022-01-24 16:26:26","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":491,"height":727,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig3-2-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig3-2.png","medium-width":176,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig3-2.png","medium_large-width":491,"medium_large-height":727,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig3-2.png","large-width":491,"large-height":727,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig3-2.png","1536x1536-width":491,"1536x1536-height":727,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig3-2.png","2048x2048-width":491,"2048x2048-height":727,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig3-2-314x465.png","card_image-width":314,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/01\/fig3-2.png","wide_image-width":491,"wide_image-height":727}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"\n
When to fine-tune a model<\/span>\u00a0<\/span><\/h2>\n
Prerequisites for fine-tuning deep learning models<\/h1>\n
GPU availability<\/h2>\n
CUDA<\/span>\u00a0<\/span><\/h2>\n
ArcGIS Pro\u00a0<\/span>\u00a0<\/span><\/h2>\n
Deep learning frameworks for ArcGIS<\/span>\u00a0<\/span><\/h2>\n
Understand the pretrained model<\/span>\u00a0<\/span><\/h1>\n
The inputs<\/span>\u00a0<\/span><\/h2>\n
\n
\n
\n
Create training data<\/span>