{"id":2712852,"date":"2025-03-10T06:00:43","date_gmt":"2025-03-10T13:00:43","guid":{"rendered":"https:\/\/www.esri.com\/arcgis-blog\/?post_type=blog&p=2712852"},"modified":"2025-03-09T12:06:07","modified_gmt":"2025-03-09T19:06:07","slug":"spatial-data-science-using-arcgis-notebooks-blog-2-data-fusion-wrangling-and-preparation","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/spatial-data-science-using-arcgis-notebooks-blog-2-data-fusion-wrangling-and-preparation","title":{"rendered":"Spatial data science using ArcGIS Notebooks Blog 2: Data fusion, wrangling, and preparation"},"author":154341,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_searchwp_excluded":""},"categories":[23341],"tags":[387782,555752,760452,24341,759592],"industry":[],"product":[36841,36561],"class_list":["post-2712852","blog","type-blog","status-publish","format-standard","hentry","category-analytics","tag-arcgis-api-for-python","tag-arcgis-notebooks","tag-data-engineering","tag-python","tag-spatial-data-science","product-api-python","product-arcgis-pro"],"acf":{"authors":[{"ID":154341,"user_firstname":"Nicholas","user_lastname":"Giner","nickname":"Nick Giner","user_nicename":"nginer","display_name":"Nicholas Giner","user_email":"NGiner@esri.com","user_url":"","user_registered":"2021-01-07 14:31:25","user_description":"Nick Giner is a Product Manager for Spatial Analysis and Data Science. Prior to joining Esri in 2014, he completed Bachelor\u2019s and PhD degrees in Geography from Penn State University and Clark University, respectively. In his spare time, he likes to play guitar, golf, cook, cut the grass, and read\/watch shows about history.","user_avatar":""},{"ID":358272,"user_firstname":"Halle","user_lastname":"Martinucci","nickname":"Halle Martinucci","user_nicename":"hmartinucci","display_name":"Halle Martinucci","user_email":"hmartinucci@esri.com","user_url":"","user_registered":"2024-06-18 18:43:42","user_description":"Halle is an Associate Product Marketing Manager on Esri\u2019s Spatial Analytics & Data Science team.","user_avatar":""}],"short_description":"Use an ArcGIS Pro Notebook for a complete spatial data science workflow to map gentrification in US cities","flexible_content":[{"acf_fc_layout":"content","content":"

Introduction<\/h2>\n

In this second blog article, we\u2019ll discuss the input data and the steps used to bring it all together and calculate our five gentrification indicator variables.<\/p>\n

Data sources<\/h2>\n

The input data came from three main sources:<\/p>\n

1) National Historical Geographic Information System (NHGIS)<\/a><\/strong> – provided freely downloadable US census tract boundary GIS files and CSVs representing hundreds of data variables related to socio-economic and demographic characteristics, education level, median family income, and median gross rent. For the purpose of this study, I downloaded data for these variables for four time steps: 1990, 2000, 2010, and 2020.<\/p>\n

2) ArcGIS Living Atlas of the World<\/strong><\/a> \u2013 because certain variables were not available from NHGIS for 2020, we approximated these with the most recent data from the ArcGIS Living Atlas. Detailed steps for accessing this data are described in the following sections.<\/p>\n

3) NYC OpenData<\/a> <\/strong>– was used to help us identify and remove areas where gentrification cannot take place (e.g. parks, airports, cemeteries, etc.). In particular, we used the 2010 Neighborhood Tabulation Areas.<\/p>\n"},{"acf_fc_layout":"content","content":"

Data preparation steps<\/h2>\n

Prepare the US census tract GIS data in ArcGIS Pro<\/em><\/strong><\/p>\n

The base GIS dataset representing US census tract boundaries for the year 2010* contains several important fields.\u00a0 For example, the GISJOIN field contains a unique identifier on which all subsequent joins will take place. \u00a0The STATEFP10 and COUNTYFP10 fields contain unique state and county FIPS<\/strong><\/a> identifiers for each census tract, respectively.\u00a0 To make it easier to query census tracts for specific areas such as New York City (which contains census tracts across five counties) we used the Calculate Field<\/strong><\/a> tool to concatenate the STATEFP10 and COUNTYFP10 fields into a new text field.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2713222,"id":2713222,"title":"tract_concat","filename":"tract_concat.jpg","filesize":96710,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/tract_concat.jpg","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/spatial-data-science-using-arcgis-notebooks-blog-2-data-fusion-wrangling-and-preparation\/tract_concat","alt":"","author":"154341","description":"","caption":"","name":"tract_concat","status":"inherit","uploaded_to":2712852,"date":"2025-03-03 21:15:28","modified":"2025-03-03 21:15:28","menu_order":0,"mime_type":"image\/jpeg","type":"image","subtype":"jpeg","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":806,"height":375,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/tract_concat-213x200.jpg","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/tract_concat.jpg","medium-width":464,"medium-height":216,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/tract_concat.jpg","medium_large-width":768,"medium_large-height":357,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/tract_concat.jpg","large-width":806,"large-height":375,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/tract_concat.jpg","1536x1536-width":806,"1536x1536-height":375,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/tract_concat.jpg","2048x2048-width":806,"2048x2048-height":375,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/tract_concat.jpg","card_image-width":806,"card_image-height":375,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/tract_concat.jpg","wide_image-width":806,"wide_image-height":375}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"sidebar","content":"

Note:<\/strong> *The NHGIS provides time series tables which contain data from multiple times (e.g. 1990, 2000, 2010, 2020) for the geographic boundaries from a single census.\u00a0 In our case, the time series tables are standardized to the 2010 census tract boundaries.\u00a0 You can read more about the methodology for geographic standardization here<\/a><\/strong>.<\/p>\n","image_reference":false,"layout":"standard","image_reference_figure":"","snippet":"","spotlight_name":"","section_title":"","position":"Center","spotlight_image":false},{"acf_fc_layout":"content","content":"

From here, we can select census tracts by their true county FIPS code.\u00a0 In the case of New York City, a simple attribute query of five county FIPS codes (36005-Bronx, 36047-Kings, 36061-New York, 36081-Queens, 36085-Richmond) gave us the 2,164 census tracts that make up New York City.\u00a0 After exporting these selected census tracts as a new feature class with the appropriate spatial reference (in this case, NAD 1983 State Plane New York Long Island<\/strong><\/a>), the final step was to use the Spatial Join<\/strong><\/a> tool to append the neighborhood name to any census tract whose center falls within it.\u00a0 This will allow us to identify census tracts in parks, airports, cemeteries, etc. that will be removed from the analysis in subsequent steps.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2713312,"id":2713312,"title":"spatial_join","filename":"spatial_join.jpg","filesize":39581,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/spatial_join.jpg","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/spatial-data-science-using-arcgis-notebooks-blog-2-data-fusion-wrangling-and-preparation\/spatial_join-2","alt":"","author":"154341","description":"","caption":"","name":"spatial_join-2","status":"inherit","uploaded_to":2712852,"date":"2025-03-03 21:26:33","modified":"2025-03-03 21:26:33","menu_order":0,"mime_type":"image\/jpeg","type":"image","subtype":"jpeg","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":392,"height":448,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/spatial_join-213x200.jpg","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/spatial_join.jpg","medium-width":228,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/spatial_join.jpg","medium_large-width":392,"medium_large-height":448,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/spatial_join.jpg","large-width":392,"large-height":448,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/spatial_join.jpg","1536x1536-width":392,"1536x1536-height":448,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/spatial_join.jpg","2048x2048-width":392,"2048x2048-height":448,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/spatial_join.jpg","card_image-width":392,"card_image-height":448,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/spatial_join.jpg","wide_image-width":392,"wide_image-height":448}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Using ArcGIS Pro\u2019s\u00a0Data Engineering<\/strong><\/a>\u00a0view, we can easily see and explore the summary statistics and distributions of each variable in the feature class.\u00a0 We also now have the \u201cntaname\u201d attribute, which represents the neighborhood name.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2713322,"id":2713322,"title":"DE_view","filename":"DE_view.jpg","filesize":113834,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/DE_view.jpg","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/spatial-data-science-using-arcgis-notebooks-blog-2-data-fusion-wrangling-and-preparation\/de_view-4","alt":"","author":"154341","description":"","caption":"","name":"de_view-4","status":"inherit","uploaded_to":2712852,"date":"2025-03-03 21:31:03","modified":"2025-03-03 21:31:03","menu_order":0,"mime_type":"image\/jpeg","type":"image","subtype":"jpeg","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1238,"height":438,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/DE_view-213x200.jpg","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/DE_view.jpg","medium-width":464,"medium-height":164,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/DE_view.jpg","medium_large-width":768,"medium_large-height":272,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/DE_view.jpg","large-width":1238,"large-height":438,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/DE_view.jpg","1536x1536-width":1238,"1536x1536-height":438,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/DE_view.jpg","2048x2048-width":1238,"2048x2048-height":438,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/DE_view-826x292.jpg","card_image-width":826,"card_image-height":292,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/DE_view.jpg","wide_image-width":1238,"wide_image-height":438}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"image","image":{"ID":2713332,"id":2713332,"title":"nyc_census_tracts_parks","filename":"nyc_census_tracts_parks.jpg","filesize":164977,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/nyc_census_tracts_parks.jpg","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/spatial-data-science-using-arcgis-notebooks-blog-2-data-fusion-wrangling-and-preparation\/nyc_census_tracts_parks","alt":"","author":"154341","description":"","caption":"Map showing the census tract boundaries (yellow) and neighborhoods. Parks, airports, and cemeteries are green.","name":"nyc_census_tracts_parks","status":"inherit","uploaded_to":2712852,"date":"2025-03-03 21:33:43","modified":"2025-03-07 17:03:39","menu_order":0,"mime_type":"image\/jpeg","type":"image","subtype":"jpeg","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":800,"height":689,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/nyc_census_tracts_parks-213x200.jpg","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/nyc_census_tracts_parks.jpg","medium-width":303,"medium-height":261,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/nyc_census_tracts_parks.jpg","medium_large-width":768,"medium_large-height":661,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/nyc_census_tracts_parks.jpg","large-width":800,"large-height":689,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/nyc_census_tracts_parks.jpg","1536x1536-width":800,"1536x1536-height":689,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/nyc_census_tracts_parks.jpg","2048x2048-width":800,"2048x2048-height":689,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/nyc_census_tracts_parks-540x465.jpg","card_image-width":540,"card_image-height":465,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/nyc_census_tracts_parks.jpg","wide_image-width":800,"wide_image-height":689}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Read and wrangle tabular CSV files of historic census data (year 2000)<\/em><\/strong><\/p>\n

Now that I have a clean, prepared GIS dataset, I\u2019ll move into an ArcGIS Pro Notebook to continue cleaning, wrangling, integrating, and otherwise preparing my datasets for analysis.<\/p>\n

Like most data science notebooks, I\u2019ll start off with a cell where I import the necessary Python libraries.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2713362,"id":2713362,"title":"python_libs","filename":"python_libs.jpg","filesize":14693,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/python_libs.jpg","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-pro\/analytics\/spatial-data-science-using-arcgis-notebooks-blog-2-data-fusion-wrangling-and-preparation\/python_libs-3","alt":"","author":"154341","description":"","caption":"","name":"python_libs-3","status":"inherit","uploaded_to":2712852,"date":"2025-03-03 21:41:23","modified":"2025-03-03 21:41:23","menu_order":0,"mime_type":"image\/jpeg","type":"image","subtype":"jpeg","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":244,"height":170,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/python_libs-213x170.jpg","thumbnail-width":213,"thumbnail-height":170,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/python_libs.jpg","medium-width":244,"medium-height":170,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/python_libs.jpg","medium_large-width":244,"medium_large-height":170,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/python_libs.jpg","large-width":244,"large-height":170,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/python_libs.jpg","1536x1536-width":244,"1536x1536-height":170,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/python_libs.jpg","2048x2048-width":244,"2048x2048-height":170,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/python_libs.jpg","card_image-width":244,"card_image-height":170,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2025\/03\/python_libs.jpg","wide_image-width":244,"wide_image-height":170}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"