{"id":2275312,"date":"2024-03-13T10:03:07","date_gmt":"2024-03-13T17:03:07","guid":{"rendered":"https:\/\/www.esri.com\/arcgis-blog\/?post_type=blog&p=2275312"},"modified":"2024-03-14T12:49:00","modified_gmt":"2024-03-14T19:49:00","slug":"dev-summit-2024-extending-the-segment-anything-model-sam","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis\/geoai\/dev-summit-2024-extending-the-segment-anything-model-sam","title":{"rendered":"Dev Summit 2024: Extending the Segment Anything Model (SAM)"},"author":303212,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_searchwp_excluded":""},"categories":[770712],"tags":[27481,773972,26411],"industry":[],"product":[421922],"class_list":["post-2275312","blog","type-blog","status-publish","format-standard","hentry","category-geoai","tag-automation","tag-segment-anything-model","tag-segmentation","product-arcgis"],"acf":{"short_description":"Discover how you can extend the Segment Anything Model (SAM) to improve your workflows.","flexible_content":[{"acf_fc_layout":"content","content":"

Segmentation models help to identify objects such as trees, water bodies, and crop fields in satellite or aerial imagery and extract them as features. While automating this process greatly enhances geospatial workflows, training the models can be a very time consuming task. This is where Meta AI\u2019s Segment Anything Model (SAM)<\/a> comes in!<\/p>\n

SAM is a foundational model developed to segment anything in an image without the need for additional training.<\/p>\n

SAM<\/a> has been added to our library of pretrained models<\/a>, allowing you to use it seamlessly with ArcGIS. Since it was released, SAM has quickly become one of our most popular models.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2276932,"id":2276932,"title":"samairplanes","filename":"samairplanes.png","filesize":891141,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samairplanes.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis\/geoai\/dev-summit-2024-extending-the-segment-anything-model-sam\/samairplanes","alt":"","author":"303212","description":"","caption":"SAM is used to extract airplanes as features.","name":"samairplanes","status":"inherit","uploaded_to":2275312,"date":"2024-03-06 19:20:53","modified":"2024-03-06 21:15:23","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":930,"height":591,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samairplanes-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samairplanes.png","medium-width":411,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samairplanes.png","medium_large-width":768,"medium_large-height":488,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samairplanes.png","large-width":930,"large-height":591,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samairplanes.png","1536x1536-width":930,"1536x1536-height":591,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samairplanes.png","2048x2048-width":930,"2048x2048-height":591,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samairplanes-732x465.png","card_image-width":732,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samairplanes.png","wide_image-width":930,"wide_image-height":591}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

While SAM works great for many workflows, you may find that SAM alone doesn\u2019t always suit your specific needs.<\/p>\n

Have you ever wished you could indicate which type of object to extract as features, or wanted to fine-tune the model on your data? In her plenary demo, Priyanka will share how you can do just that.<\/p>\n"},{"acf_fc_layout":"kaltura","video_id":"1_xjkmg1he","time":true,"start":"146"},{"acf_fc_layout":"content","content":"

<\/h1>\n

Text SAM<\/h1>\n"},{"acf_fc_layout":"content","content":"

In a scene with many types of objects such as buildings, roads, and trees, SAM will segment everything regardless of the object type. Depending on your workflow, you may wish for the model to only extract buildings from the imagery.<\/p>\n

This is why we have released the Text SAM<\/a> model as a deep learning package. Text SAM combines SAM with another model, Grounding DINO<\/a>, which detects objects using text prompts. The Text SAM model allows you to input any free-form text that describes your object of interest. The model will then extract features based on the text you provide.<\/p>\n

In her demo, Priyanka will run Text SAM as a geoprocessing tool from ArcGIS Pro.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2275962,"id":2275962,"title":"textsam1","filename":"textsam1-1.png","filesize":1693556,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsam1-1.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis\/geoai\/dev-summit-2024-extending-the-segment-anything-model-sam\/textsam1-2","alt":"Run the Text SAM model","author":"303212","description":"","caption":"Priyanka is running Text SAM as a geoprocessing tool.","name":"textsam1-2","status":"inherit","uploaded_to":2275312,"date":"2024-03-05 22:10:02","modified":"2024-03-06 21:19:43","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1655,"height":868,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsam1-1-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsam1-1.png","medium-width":464,"medium-height":243,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsam1-1.png","medium_large-width":768,"medium_large-height":403,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsam1-1.png","large-width":1655,"large-height":868,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsam1-1-1536x806.png","1536x1536-width":1536,"1536x1536-height":806,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsam1-1.png","2048x2048-width":1655,"2048x2048-height":868,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsam1-1-826x433.png","card_image-width":826,"card_image-height":433,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsam1-1.png","wide_image-width":1655,"wide_image-height":868}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Priyanka has entered trailer homes as the object of interest she wants the model to identify in her input imagery. Let\u2019s see what happens when she runs the model.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2276762,"id":2276762,"title":"textsamresults","filename":"textsamresults.png","filesize":988014,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsamresults.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis\/geoai\/dev-summit-2024-extending-the-segment-anything-model-sam\/textsamresults","alt":"Text SAM model results","author":"303212","description":"","caption":"Results after running Text SAM as a geoprocessing tool.","name":"textsamresults","status":"inherit","uploaded_to":2275312,"date":"2024-03-06 18:26:13","modified":"2024-03-06 21:20:19","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":862,"height":446,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsamresults-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsamresults.png","medium-width":464,"medium-height":240,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsamresults.png","medium_large-width":768,"medium_large-height":397,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsamresults.png","large-width":862,"large-height":446,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsamresults.png","1536x1536-width":862,"1536x1536-height":446,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsamresults.png","2048x2048-width":862,"2048x2048-height":446,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsamresults-826x427.png","card_image-width":826,"card_image-height":427,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/textsamresults.png","wide_image-width":862,"wide_image-height":446}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

The model has successfully identified the trailers. It has even included a trailer outside of its presumed parking spot, differentiating it from other vehicles on the road!<\/p>\n"},{"acf_fc_layout":"content","content":"

SAMLoRA model<\/h1>\n"},{"acf_fc_layout":"content","content":"

SAM is designed to work with 3-band, natural color RGB imagery. To finetune it on other data types that normally require massive computing resources such as synthetic-aperture radar (SAR) data, elevation data, and multispectral imagery, we now have the SAMLoRA model.<\/p>\n

The SAMLoRA model, available in the arcgis.learn module<\/a>, applies the low-rank-adaptation (LoRA) technique to the SAM model and finetunes it for geospatial imagery segmentation. LoRA is a training technique that enables few-shot learning, significantly reducing the number of trainable parameters.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2275882,"id":2275882,"title":"loramodel","filename":"loramodel.png","filesize":183160,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/loramodel.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis\/geoai\/dev-summit-2024-extending-the-segment-anything-model-sam\/loramodel","alt":"Segmenting geospatial imagery using low-rank-adaptation of SAM model","author":"303212","description":"","caption":"Segmenting geospatial imagery using low-rank-adaptation of SAM model.","name":"loramodel","status":"inherit","uploaded_to":2275312,"date":"2024-03-05 21:55:03","modified":"2024-03-06 21:21:47","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1142,"height":399,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/loramodel-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/loramodel.png","medium-width":464,"medium-height":162,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/loramodel.png","medium_large-width":768,"medium_large-height":268,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/loramodel.png","large-width":1142,"large-height":399,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/loramodel.png","1536x1536-width":1142,"1536x1536-height":399,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/loramodel.png","2048x2048-width":1142,"2048x2048-height":399,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/loramodel-826x289.png","card_image-width":826,"card_image-height":289,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/loramodel.png","wide_image-width":1142,"wide_image-height":399}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

In her demo, Priyanka extracts building footprints by using the SAMLoRA model finetuned on elevation data with just a handful of images.<\/p>\n

Instead of needing a GPU cluster and several days of training, she\u2019s able to do this in under 30 minutes on her laptop GPU.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2276772,"id":2276772,"title":"samloraresults","filename":"samloraresults.png","filesize":349216,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samloraresults.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis\/geoai\/dev-summit-2024-extending-the-segment-anything-model-sam\/samloraresults","alt":"Results for the SAMLoRA model","author":"303212","description":"","caption":"Results identifying building footprints.","name":"samloraresults","status":"inherit","uploaded_to":2275312,"date":"2024-03-06 18:27:46","modified":"2024-03-06 21:39:18","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1156,"height":532,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samloraresults-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samloraresults.png","medium-width":464,"medium-height":214,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samloraresults.png","medium_large-width":768,"medium_large-height":353,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samloraresults.png","large-width":1156,"large-height":532,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samloraresults.png","1536x1536-width":1156,"1536x1536-height":532,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samloraresults.png","2048x2048-width":1156,"2048x2048-height":532,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samloraresults-826x380.png","card_image-width":826,"card_image-height":380,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/samloraresults.png","wide_image-width":1156,"wide_image-height":532}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

The model has successfully identified the building footprints.<\/p>\n"},{"acf_fc_layout":"content","content":"

Additional resources<\/h1>\n"},{"acf_fc_layout":"content","content":"

In her demo, Priyanka has highlighted how SAM as a deep learning package can be used in your workflows, how you can prompt SAM using text inputs to segment specific features and imagery, and how you can fine-tune SAM on your data.<\/p>\n

We are excited to see how you leverage SAM in your workflows! For more information on how to get started, explore the following:<\/p>\n