Bay Area Utility Improves GIS Data Management with Quality Assurance Initiative

From drinking to cooking and personal hygiene, water is a vital resource for the health of individuals and their communities. State and local agencies across the United States are working hard to maintain the safety of the nation's water supply by treating water to remove contaminants and distributing it via millions of miles of pipelines.

One of the missions of the East Bay Municipal Utility District, or EBMUD (District), is to provide reliable, high-quality water and wastewater services to residents within its service area. EBMUD's water system serves 1.4 million customers in the eastern region of the San Francisco Bay Area, and its award-winning wastewater treatment protects San Francisco Bay and serves 685,000 customers.

EBMUD's infrastructure includes 4,300 miles of pipelines with fixtures like valves, fittings, and pipe junctions that are mapped and modeled within their geographic information system (GIS). The mapping group creates and maintains the water network data in an enterprise geodatabase that utilizes a geometric network. GIS data is widely used by many stakeholders within the district, such as the water distribution planning group for an enterprise hydraulic model to estimate the pressure and flow at any point in the water distribution system, to conduct hydraulic studies to size facilities, to recommend facility outages, and to identify distribution system improvements and critical locations. The operations and maintenance group also uses GIS data for dispatching, responding to leak investigation, capturing valve exercising data, and creating outage plans for emergency repairs.

The success of hydraulic modeling and operational exercises relies on the quality of the water distribution system data, which had significant quality assurance and accuracy challenges due to the volume of information and artifacts from conversion from legacy systems. To address this, EBMUD implemented a new set of tools that has enabled an efficient data review process, helping the district ensure that data is of the highest quality.

Accurate data is important in many aspects of EBMUD's work. Hydraulic modeling results depend heavily on the quality of GIS data used in the analysis. Accurate network connectivity between the miles of pipelines and junctions and geometric coincidence in pipes and valves are critical to enterprise hydraulic modeling. Also, identifying pipes that have a higher probability of failure is crucial for pipeline risk modeling so that replacements can be prioritized. Preparing outage plans to alert impacted customers also requires accurate data. 

The editors in the mapping group export the feature classes into a file geodatabase (FGDB) to perform hydraulic/pipeline modeling and operational/maintenance exercises. They correct any errors found in their local copy of the FGDB; yet in the past, these fixes were not copied back into the default version of the geodatabase. As a result, editors ended up having to fix the same errors every time they download the latest data locally.

Although the geometric network provides basic quality control methods, the quality of EBMUD's GIS data was less than ideal, which ultimately required manual corrections on the user end to obtain proper results for planning as well as operations and maintenance. The QA/QC method was also time-consuming as editors manually sorted through data to find errors.

As such, EBMUD began looking for a more comprehensive way to improve quality assurance of its GIS data to proactively identify errors, clean up the database, and provide higher-quality data. 

"From exercises on patching leaks to identifying the pipes most likely to fail in the future, GIS data is used for a lot of critical operations," says Rachel Wong, GISP, GIS software engineer II for EBMUD. "If the GIS data quality is not good enough for a particular job, it can impact staff's analysis and work." 

When members of EBMUD attended the 2013 Esri User Conference, they participated in a data health check activity that allows users to review data on-site with an Esri industry expert. The group met with Gurunathan Ganesarethinam, a consultant and project manager from geodata engineering in Esri's Professional Services.

Using the ArcGIS Data Reviewer extension, automated checks were run against a copy of EBMUD's water network data, giving staff a look at the product's capabilities and the types of errors that can be found. The activity helped EBMUD gain an overall assessment of the quality of its data, and agency officials decided to implement the solution.

ArcGIS Data Reviewer is an extension to ArcGIS Desktop and ArcGIS Enterprise that provides an out-of-the-box framework to perform QA/QC directly within ArcGIS, a complete mapping and analytics platform.

According to Wong, the group chose this extension because it provided an extensive set of data validation checks to validate feature integrity and find attribute and relationship errors. If not for these out-of-the-box checks, staff would have to build complicated models using ArcGIS Desktop geoprocessing tools or writing custom Python scripts.

ArcGIS Data Reviewer also offered EBMUD a solution to store and manage data errors in a centralized location as a Reviewer Table within the geodatabase. The editors in the mapping group use this feature to view data errors, navigate to their location, group errors by category, and understand the error severities to prioritize the data cleanup work.

Ganesarethinam conducted an on-site workshop at EBMUD to train the engineering and mapping teams how to use the ArcGIS Data Reviewer extension. After discussing the different types of checks they needed, he also helped them configure those checks. EBMUD editors then internally expanded the configurations using the knowledge gained during the workshop, crafting it to suit their needs.

For the implementation of ArcGIS Data Reviewer, the team began with a list of high-priority data errors to check in the geodatabase. These include disconnected features (overshoot/undershoot at pipe intersections and orphan pipes), pipe ends without a proper device, coincident pipes, and the pipes' geometry with kickbacks.

Wong validated features in the geodatabase using the configured QC checks and stored the resultant errors for editors to use when correcting features. She also trained the editors to use the new configurations to find errors on their own. 

Using ArcGIS Data Reviewer has helped improve data quality that impacts day-to-day operations involving GIS data. Edits are done more efficiently with the new QC configurations. Editors in the mapping group can now validate their work daily when they make map edits, ensuring a complete QC check before a job is done. Also, rather than manually sorting through data to find/fix issues, the correction process has been streamlined due to its ability to manage the error life cycle within ArcGIS Data Reviewer.

According to Wong, the ability to find errors in an automated way enabled the team to focus its efforts on fixing them. Editors were able to correct a substantial 80 percent of the errors within six months of implementing the tool. This streamlined approach helped speed up the error cleanup. Upon rerunning the same checks in June 2019, error reductions included pipes undershoot/overshoot—from 2,400 to 400; pipes with errors at crossings—from 490 to 6; and pipes that don't split at the tee—from 2,100 to 30.

"We did not have a comprehensive way to identify errors. We could use geoprocessing tools, but the tools could not cover everything," says Wong. "But ArcGIS Data Reviewer has really helped us identify errors more efficiently. We can look for the exact type of geometric or an attribute error and then go right to the error location in our data."

When data is in a geometric network, there are no notifications when a rule is violated, making it difficult to pinpoint errors. "Without Data Reviewer, we would have had to adopt a time-consuming process to build checks using geoprocessing tools and export the results as individual layers. Editors would sort through those output layers to identify the features that need to be corrected," explains Wong. "This process has been improved with the Reviewer Table feature in ArcGIS Data Reviewer. "

This feature also simplifies the task of finding the location of errors, making the job of editing more efficient. Editors can now proactively identify errors and fix them before data is sent for hydraulic modeling. The table also indicates who has reviewed the work and what the status is, improving collaboration in the group.

Wong says, "Because QA/QC results are saved in the geodatabase, there's a certain degree of transparency and error tracking. We can easily go into the Reviewer Table to see how editors have verified data and view their progress."

Other features Wong and the editors enjoy include the ability to run multiple data checks at once, and the QC grid functionality in the Reviewer Table. Using a QC grid, errors can be linked to the pressure zone polygons to help editors assign and prioritize tasks based on pressure zones. Focusing on specific pressure zones when cleaning up the data reduces the risk of conflicts and ultimately increases productivity.

"With ArcGIS Data Reviewer, we've gained efficiency and a cleaner workflow," says Wong.