{"id":2311202,"date":"2024-05-14T19:40:06","date_gmt":"2024-05-15T02:40:06","guid":{"rendered":"https:\/\/www.esri.com\/arcgis-blog\/?post_type=blog&p=2311202"},"modified":"2024-05-16T06:44:44","modified_gmt":"2024-05-16T13:44:44","slug":"text-sam-extracting-gis-features-using-text-prompts","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/geoai\/text-sam-extracting-gis-features-using-text-prompts","title":{"rendered":"Text SAM: Extracting GIS Features Using Text Prompts"},"author":348302,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_searchwp_excluded":""},"categories":[770712,22931],"tags":[186132,774402,774332,774392,774382],"industry":[],"product":[36561],"class_list":["post-2311202","blog","type-blog","status-publish","format-standard","hentry","category-geoai","category-imagery","tag-deep-learning","tag-free-form-text","tag-sam","tag-text-prompt","tag-text-sam","product-arcgis-pro"],"acf":{"authors":[{"ID":348302,"user_firstname":"Priyanka","user_lastname":"Tuteja","nickname":"Priyanka Tuteja","user_nicename":"ptuteja","display_name":"Priyanka Tuteja","user_email":"ptuteja@esri.com","user_url":"","user_registered":"2023-11-02 15:56:33","user_description":"Principal Product Engineer in the GeoAI team at Esri R&D Center, New Delhi, India.","user_avatar":""},{"ID":6911,"user_firstname":"Rohit","user_lastname":"Singh","nickname":"Rohit Singh","user_nicename":"rsinghesri-com","display_name":"Rohit Singh","user_email":"rsingh@esri.com","user_url":"","user_registered":"2018-03-02 00:19:00","user_description":"Rohit Singh is Director of Esri\u2019s R&D Center in New Delhi, leading the design and development of Geospatial AI capabilities across the ArcGIS platform. He has played a key role in the development of ArcGIS API for Python, ArcGIS Java Engine API, and the Linux enablement of ArcGIS. An alumnus of IIT Kharagpur, Rohit holds an MS in Computer Science with specialization in AI from Georgia Tech.","user_avatar":""},{"ID":356312,"user_firstname":"Shivani","user_lastname":"Pathak","nickname":"Shivani Pathak","user_nicename":"spathak","display_name":"Shivani Pathak","user_email":"spathak@esri.com","user_url":"","user_registered":"2024-04-15 13:32:55","user_description":"","user_avatar":""}],"short_description":"Prompt Segment Anything Model (SAM) with free form text to extract features in your imagery","flexible_content":[{"acf_fc_layout":"content","content":"

Meta’s Segment Anything Model (SAM) is making waves in the realm of image segmentation. It can precisely segment objects in images and can be used to extract GIS features such as center-pivot farms, ships or yachts that distinctly stand out from their background. However, while SAM excels at <\/span><\/span><\/span>segmenting objects, it does <\/span><\/span><\/span>not <\/span><\/span><\/span>categorize them <\/span><\/span><\/span>by the type of object<\/span><\/span><\/span>, and this poses a challenge.<\/span><\/span><\/span><\/p>\n

In a complex scene with buildings, roads and trees, SAM segments each object, generating distinct masks for each instance, but it doesn’t provide any knowledge of the type of object. Consequently, extracting GIS features representing trees or buildings- becomes impractical as all features belong to an unknown ‘object’ class. This limitation renders SAM incompatible with GIS workflows, as the lack of informative data impedes feature extraction.<\/p>\n

Enter Grounding Dino<\/a>, an open-source vision-language model that excels at detecting objects in images given their textual description. By integrating SAM with Grounding Dino, it becomes possible not only to detect features of a specific kind in imagery but also to precisely segment them<\/span><\/span>. The synergy between Grounding Dino and SAM allows users to provide free-form text prompts, describing the object of interest they want to extract. This innovative approach enables SAM to extract features based on textual descriptions, vastly expanding its utility in GIS applications.<\/p>\n

We have released Text SAM<\/a> as an open-source\u00a0sample model<\/a> that can be prompted using free-form text prompts to extract features of various kinds.\u00a0 The source code of this model is available here<\/a> for those who want to learn how they can integrate such models with ArcGIS.<\/p>\n

 <\/p>\n

 <\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2319282,"id":2319282,"title":"text_sam_inf","filename":"text_sam_inf-1.gif","filesize":3491138,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/04\/text_sam_inf-1.gif","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/geoai\/text-sam-extracting-gis-features-using-text-prompts\/text_sam_inf-2","alt":"","author":"348302","description":"","caption":"","name":"text_sam_inf-2","status":"inherit","uploaded_to":2311202,"date":"2024-04-26 13:06:24","modified":"2024-04-26 13:06:24","menu_order":0,"mime_type":"image\/gif","type":"image","subtype":"gif","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1916,"height":993,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/04\/text_sam_inf-1-213x200.gif","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/04\/text_sam_inf-1.gif","medium-width":464,"medium-height":240,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/04\/text_sam_inf-1.gif","medium_large-width":768,"medium_large-height":398,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/04\/text_sam_inf-1.gif","large-width":1916,"large-height":993,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/04\/text_sam_inf-1-1536x796.gif","1536x1536-width":1536,"1536x1536-height":796,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/04\/text_sam_inf-1.gif","2048x2048-width":1916,"2048x2048-height":993,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/04\/text_sam_inf-1-826x428.gif","card_image-width":826,"card_image-height":428,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/04\/text_sam_inf-1.gif","wide_image-width":1916,"wide_image-height":993}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Let\u2019s look at some of the results that we obtained using text prompts such as words or phrases.<\/p>\n"},{"acf_fc_layout":"storymap","title":"","description":"","static":false,"storymap_url":"https:\/\/arcg.is\/0r045r0<\/a>"},{"acf_fc_layout":"content","content":"

Note: Avoid acronyms and short forms when describing your object to get accurate detections.<\/p>\n

While Text SAM excels at segmenting diverse objects, the best results can be obtained by keeping the following in mind:<\/span>\u00a0<\/span><\/p>\n

    \n
  1. \u201cThings\u201d, not \u201cstuff\u201d\u2014<\/strong>Text SAM is suitable for extracting objects with clear boundaries and distinct shapes, such as cars, trees, buildings, etc. It is not suitable at extracting amorphous or indistinct elements like grass, water, forests, etc., which might cover large areas and may not have well defined boundaries or shapes. Pixel classification models, such as <\/span>High Resolution Land Cover Classification<\/span><\/b> are better suited for that.<\/span>\u00a0<\/span><\/li>\n
  2. Pick the right cell-size\u2014<\/strong>The cell-size is often the most important model argument for any deep learning based inferencing task. It plays a critical role as it directly affects how much of the object and it\u2019s background the model \u201csees\u201d during prediction. <\/span>Other model arguments <\/span>like the<\/span> text threshold, box threshold, and batch size <\/span>may enhance performance. <\/span><\/span><\/span>Here\u2019s<\/span> a quick rundown of these parameters:<\/span><\/span>\u00a0<\/span>\u00a0<\/span>\n
      \n
    1. \n