"}],"short_description":"Learn about the Text Analysis tools for performing Natural Language Processing (NLP) tasks in the new GeoAI toolbox in ArcGIS Pro 3.0","flexible_content":[{"acf_fc_layout":"content","content":"At ArcGIS Pro 3.0, we introduced the GeoAI toolbox<\/a>, which brings new AI and machine learning algorithms to ArcGIS Pro.\u00a0 This toolbox contains eight geoprocessing tools that help to solve Natural Language Processing problems (Text Analysis Toolset \u2013 6 tools), and classification and regression problems (Feature and Tabular Analysis Toolset \u2013 2 tools).<\/p>\n In this blog article, we\u2019ll focus on the six tools within the Text Analysis toolset<\/a>, which are designed to work with unstructured text data using Natural Language Processing (NLP) techniques.\u00a0 We’ll first get a brief introduction to unstructured text data and its intersection with GIS, we’ll then define NLP and explore some common NLP tasks, then wrap up by walking through some workflows that use the new Text Analysis tools to solve real-world NLP problems.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1694742,"id":1694742,"title":"toolbox_v3","filename":"toolbox_v3.png","filesize":30073,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/08\/toolbox_v3.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/text-analysis-in-arcgis-pro-3-0\/toolbox_v3","alt":"","author":"154341","description":"","caption":"","name":"toolbox_v3","status":"inherit","uploaded_to":1694622,"date":"2022-08-24 14:53:54","modified":"2022-08-24 14:53:54","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":416,"height":431,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/08\/toolbox_v3-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/08\/toolbox_v3.png","medium-width":252,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/08\/toolbox_v3.png","medium_large-width":416,"medium_large-height":431,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/08\/toolbox_v3.png","large-width":416,"large-height":431,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/08\/toolbox_v3.png","1536x1536-width":416,"1536x1536-height":431,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/08\/toolbox_v3.png","2048x2048-width":416,"2048x2048-height":431,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/08\/toolbox_v3.png","card_image-width":416,"card_image-height":431,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/08\/toolbox_v3.png","wide_image-width":416,"wide_image-height":431}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" As GIS professionals, we are very accustomed to working with three main data types: vector data, tabular data, and raster data. Generally, we refer to vector and tabular data as “structured”, meaning that it is highly organized into rows and columns, has a predefined data model, and is easily searchable.\u00a0 Raster data is commonly referred to as “unstructured”, in that it does not have a predefined data model or schema, and is difficult to search.\u00a0 The table below shows some common vector, tabular, and raster data formats used in ArcGIS Pro.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1703332,"id":1703332,"title":"data_formats_v3","filename":"data_formats_v3.png","filesize":24995,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/data_formats_v3.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/text-analysis-in-arcgis-pro-3-0\/data_formats_v3","alt":"","author":"154341","description":"","caption":"","name":"data_formats_v3","status":"inherit","uploaded_to":1694622,"date":"2022-09-01 14:44:30","modified":"2022-09-01 14:44:30","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1647,"height":320,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/data_formats_v3-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/data_formats_v3.png","medium-width":464,"medium-height":90,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/data_formats_v3.png","medium_large-width":768,"medium_large-height":149,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/data_formats_v3.png","large-width":1647,"large-height":320,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/data_formats_v3-1536x298.png","1536x1536-width":1536,"1536x1536-height":298,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/data_formats_v3.png","2048x2048-width":1647,"2048x2048-height":320,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/data_formats_v3-826x160.png","card_image-width":826,"card_image-height":160,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/data_formats_v3.png","wide_image-width":1647,"wide_image-height":320}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" Another example of unstructured data is text.\u00a0 This is free-flowing language that lacks the row\/column structure of vector and tabular data.\u00a0 Text is found almost everywhere–in books and journals, news articles, metadata, text messages, Word documents\/PowerPoint slides, PDFs, medical records, customer surveys, invoices, and of course all over the internet on websites, social media posts, instant messages, and emails.\u00a0 What’s really interesting for us as GIS professionals is that many forms of unstructured text actually contain location information such as coordinates, addresses, place names, geographies, etc.\u00a0 Take for example the event below scraped from the City of Philadelphia website<\/a>:<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1703352,"id":1703352,"title":"nlp_example","filename":"nlp_example.png","filesize":26139,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/nlp_example.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/text-analysis-in-arcgis-pro-3-0\/nlp_example","alt":"","author":"154341","description":"","caption":"","name":"nlp_example","status":"inherit","uploaded_to":1694622,"date":"2022-09-01 14:57:29","modified":"2022-09-01 14:57:29","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":590,"height":478,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/nlp_example-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/nlp_example.png","medium-width":322,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/nlp_example.png","medium_large-width":590,"medium_large-height":478,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/nlp_example.png","large-width":590,"large-height":478,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/nlp_example.png","1536x1536-width":590,"1536x1536-height":478,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/nlp_example.png","2048x2048-width":590,"2048x2048-height":478,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/nlp_example-574x465.png","card_image-width":574,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/nlp_example.png","wide_image-width":590,"wide_image-height":478}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" Is there meaningful locational or geographical information in this unstructured text example?\u00a0 Is there any additional information that we can extract from this text?\u00a0 Is it possible to pull this unstructured information and have it logically organized into a structured table with rows and columns?<\/p>\n The answer is yes, and one of the state-of-the-art ways to do it is using Natural Language Processing (NLP).\u00a0 NLP is a field of computer science concerned with the processing and analysis of natural language text.\u00a0 It’s about giving computers the capacity to understand and learn from written text and spoken word in a similar fashion to humans.\u00a0 From its beginnings in the 1950s, the study of NLP has evolved from linguistic theory and small scale technological prototypes to handwritten rule-based systems in the 1980s, and eventually to statistical and machine learning methods as computer technology improved into the 1990s and 2000s.\u00a0 Since the early 2010s, NLP has been primarily accomplished using deep learning.\u00a0 Deep neural networks solve complex problems by breaking them down into multiple, simple levels of representation (e.g. layers) and sequentially learning more about the problem at each level.\u00a0 This type of architecture is particularly well suited for working with text data, which is itself a sequence of words or phrases that transpire over time.\u00a0 Some common NLP tasks include:<\/p>\n In the following sections, we’ll have a look the new Text Analysis tools in terms of the overall task they complete and then see some specific use cases and applications.<\/p>\n <\/p>\n"},{"acf_fc_layout":"sidebar","content":" Note:<\/strong> All tools in the GeoAI toolbox require the installation of the required deep learning frameworks libraries. For instructions on installing deep learning packages, see\u00a0Deep Learning Libraries Installers for ArcGIS<\/a>.<\/p>\n","image_reference":false,"layout":"standard","image_reference_figure":"","snippet":"","spotlight_name":"","section_title":"","position":"Center","spotlight_image":false},{"acf_fc_layout":"content","content":" Task: <\/strong><\/p>\n Tools:<\/strong><\/p>\n Use cases and applications:<\/strong><\/p>\n Example use case:<\/strong><\/p>\n The example below shows a folder full of text files containing crime incident reports from the city of Madison, Wisconsin, USA.\u00a0 Each report may contain information related to the location of the crime, the crime type, the date and time of the crime, if there was a weapon used, and who the reporting officer was.\u00a0 The new named-entity recognition tools can be used to train a NLP model to extract several entities from each report into a structured table, and then geocode the locations of the crimes to produce a map.<\/p>\n <\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1703162,"id":1703162,"title":"named_entity_workflow","filename":"named_entity_workflow.png","filesize":198989,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/named_entity_workflow.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/text-analysis-in-arcgis-pro-3-0\/named_entity_workflow","alt":"","author":"154341","description":"","caption":"Named-entity recognition workflow.","name":"named_entity_workflow","status":"inherit","uploaded_to":1694622,"date":"2022-09-01 12:00:45","modified":"2022-09-01 15:18:17","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1859,"height":1048,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/named_entity_workflow-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/named_entity_workflow.png","medium-width":464,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/named_entity_workflow.png","medium_large-width":768,"medium_large-height":433,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/named_entity_workflow.png","large-width":1859,"large-height":1048,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/named_entity_workflow-1536x866.png","1536x1536-width":1536,"1536x1536-height":866,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/named_entity_workflow.png","2048x2048-width":1859,"2048x2048-height":1048,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/named_entity_workflow-826x465.png","card_image-width":826,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/named_entity_workflow.png","wide_image-width":1859,"wide_image-height":1048}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" Task: <\/strong><\/p>\n Tools:<\/strong><\/p>\n Use cases and applications:<\/strong><\/p>\n Task: <\/strong><\/p>\n Tools:<\/strong><\/p>\n Use cases and applications:<\/strong><\/p>\n Example use case:<\/strong><\/p>\n The example below highlights the use of text classification to predict the country associated with an address.\u00a0 The “Address” field contains information for street, number, and town, but is incomplete because it is missing the country.\u00a0 As a result, it will be difficult to geocode these locations.\u00a0 This task is further complicated by the fact that the addresses occur in several different languages.\u00a0 The new text classification tools can be used to train a NLP model to identify the country associated with each incomplete address based on a sample of correctly assigned addresses provided by the user, then append it as a new column to the table.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":1703312,"id":1703312,"title":"text_classification_table_v2","filename":"text_classification_table_v2.png","filesize":51215,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/text_classification_table_v2.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/text-analysis-in-arcgis-pro-3-0\/text_classification_table_v2","alt":"","author":"154341","description":"","caption":"Text classification: The \"Prediction\" field shows the predicted country appended to each address.","name":"text_classification_table_v2","status":"inherit","uploaded_to":1694622,"date":"2022-09-01 14:43:29","modified":"2022-09-01 14:43:34","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1674,"height":915,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/text_classification_table_v2-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/text_classification_table_v2.png","medium-width":464,"medium-height":254,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/text_classification_table_v2.png","medium_large-width":768,"medium_large-height":420,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/text_classification_table_v2.png","large-width":1674,"large-height":915,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/text_classification_table_v2-1536x840.png","1536x1536-width":1536,"1536x1536-height":840,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/text_classification_table_v2.png","2048x2048-width":1674,"2048x2048-height":915,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/text_classification_table_v2-826x451.png","card_image-width":826,"card_image-height":451,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2022\/09\/text_classification_table_v2.png","wide_image-width":1674,"wide_image-height":915}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":" So that is a brief background on unstructured text data, its use cases in GIS, and some common Natural Language Processing (NLP) tasks that are available in the Text Analysis toolset of the new GeoAI toolbox.<\/p>\n In addition to these tools, we have also released pretrained deep learning models<\/a> for text analysis\/NLP tasks.\u00a0 These pretrained models aim to eliminate some of the most challenging and time consuming manual aspects of deep learning, such as collecting training data, processing time, and compute resources, etc.\u00a0 Currently, the three pretrained NLP models available on the ArcGIS Living Atlas of the World<\/a> perform address standardization<\/a>, country classification<\/a>, and named-entity recognition<\/a> workflows, and can be used directly within the Transform Text Using Deep Learning<\/a>,\u00a0 Classify Text Using Deep Learning<\/a>, and Extract Entities Using Deep Learning<\/a> tools, respectively.<\/p>\n In future blog articles, we’ll go through a more detailed, step-by-step walkthrough of some specific workflows for named-entity recognition, text translation, and text classification.\u00a0 Stay tuned!<\/p>\n <\/p>\n"},{"acf_fc_layout":"sidebar","content":"Unstructured text data and GIS<\/strong><\/h1>\n
Natural Language Processing (NLP)<\/strong><\/h1>\n
\n
Named-entity Recognition<\/strong><\/h2>\n
\n
\n
\n
Machine Translation<\/strong><\/h2>\n
\n
\n
\n
Text Classification<\/strong><\/h2>\n
\n
\n
\n
Conclusions and resources<\/strong><\/h2>\n