Arizona Department of Environmental Quality Reduces Outages and Improves System Performance with ArcGIS Monitor

The Arizona Department of Environmental Quality (ADEQ) has three environmental programs—air quality, water quality, and waste—whose staff are dedicated to administering the state’s environmental laws and federal programs to prevent air, water, and land pollution. Each of these programs relies heavily on accurate data, efficient workflows, and interactive toolsets to ensure regulatory compliance and inform decision-making to protect public health and the environment. 

At the heart of this effort is the geospatial intelligence solutions (GIS) team, who have created tools tailored to each program’s specific workflows and needs, improving their ability to visualize and analyze their data for better decision-making. The team’s expansive infrastructure system supports these outcomes, ensuring ADEQ has the insights needed to address Arizona’s most pressing environmental challenges. 

Due to an increased demand for geographic information system (GIS) services, the GIS team has increased the number of servers in recent years, which has led to challenges such as performance issues and outages. The group needed a new solution to monitor the health of their growing enterprise GIS system and deployed ArcGIS Monitor, an enterprise-grade monitoring solution. 

Using ArcGIS Monitor has given the GIS team a more complete view of their system and improved its efficiency, helping deliver the reliability ADEQ staff need to protect the environment.

The ADEQ enterprise GIS includes an internal system comprised of one production environment and one development environment, which are both three-machine deployments of ArcGIS Enterprise. Additionally, there are five support servers, three extension servers, and one outward-facing machine that hosts custom applications and anything shared with the public.

However, this growth presented new challenges. The team began facing scalability issues, including system downtimes, performance problems, and outages, particularly following software patches, which are meant to maintain system health. With no detailed metrics available, Terry says troubleshooting these issues became time-consuming, and he and his team were largely reactive in addressing problems.

“We knew that we needed to monitor these systems better so we could be more reliable and efficient in delivering our content. We really needed some way to do what we’re doing better,” says Terry.

Prior to implementing ArcGIS Monitor, the team used custom Python scripts to check health endpoints from the ArcGIS Enterprise portal and GIS server admin APIs. They also depended on the ArcGIS Server and ArcGIS Enterprise portal logs to monitor system status. While this provided some visibility into software health, the team still lacked the detailed system metrics needed for a deeper understanding of their deployment. 

Often, they had to rely on their IT department’s broader monitoring tools that could identify system issues but provided only general recommendations like adding more RAM or disk space without pinpointing the underlying causes.

It became clear that the team needed more visibility into their system’s health to proactively manage performance and optimize resources. As a government organization, efficiency and accountability to taxpayers were additional priorities, and Terry’s team sought a solution that could help them achieve these performance goals while minimizing costs.

As current ArcGIS users, Terry says they looked for an Esri solution first because it would work within their current environment and support a seamless transition. He began using ArcGIS Monitor to monitor the health of their system and liked the fact that Monitor is designed around GIS services. ArcGIS Monitor is an enterprise-grade monitoring solution that works with ArcGIS Enterprise by providing information about system health, usage, and performance.

However, when the team first installed version 10.8.1 of Monitor, they were not particularly knowledgeable about system monitoring or performance metrics. That quickly began to change once they installed Monitor and followed the documentation to begin collecting basic metrics on system health. From there, they configured alerts to notify them of potential issues and began learning about the metrics that were going out of the recommended boundaries most often.

Over time, the team engaged more deeply with the tool, and their expertise grew exponentially. By the release of version 2023.0, they had developed a strong understanding of their system’s intricacies and could use Monitor to solve real problems effectively. This growth was driven by a combination of hands-on experience and the valuable insights that Monitor provided, empowering them to take proactive measures in managing system health.

The introduction of analysis view capabilities in the latest version proved to be a game changer. This feature allowed the team to configure custom reports tailored to their specific system architecture, enabling them to observe, identify, and troubleshoot issues more efficiently than ever before. Now, Terry checks Monitor about 5–10 times per day, utilizing it as a central hub for monitoring system performance. “We rely on those more than notifications now because we do have our baseline established and we recognize the patterns in the resource use,” he notes.

The team primarily monitors two analysis views. The first focuses on critical system metrics such as RAM and CPU usage, storage, pagefiles, health endpoints for the ArcGIS Enterprise portal, and ArcGIS Server. The second focuses on GIS service metrics, such as the RAM and CPU consumed by each active ArcSOC process, as well as data on service requests, mean request times, instance use, and saturation points.

Through their ongoing engagement with Monitor, the ADEQ GIS team has significantly enhanced their operational capabilities and taken full control of their system health.

The use of Monitor has enhanced visibility and provided the GIS team at ADEQ with a holistic view of their system. The primary goal of the GIS team was to have an efficient and reliable system, and Terry says Monitor has helped them achieve it. They now have minimal system downtime that is meeting their goal of keeping their production system running between 60 percent and 80 percent RAM usage during business hours.

“We know that we’re using that system as efficiently as possible. It’s not over or under-allocated. That metric shows that the systems are operating at an efficient and reliable level,” says Terry.

The detailed metrics Monitor provides has meant less time troubleshooting for the GIS team, freeing up time for other tasks. Terry explains that because they’ve reached a point where the system is now optimized, only about 20 percent of his time is spent monitoring the system. He now has more time to develop new products and build applications, an important aspect of his role.

“Even when we do have issues that are unexpected, because of these tools and these reports, we can diagnose what the issue is and respond to things faster,” says Terry.

In a recent instance, Terry noticed in Monitor that the CPU on their production machine was very uncharacteristic. He checked the process page, saw a sync tool error, and investigated the issue by checking all syncs, eventually determining that the backup process had run off schedule. He says this is a good example of how he is using metrics to keep the system performing efficiently.

The detailed data also helps Terry and the team prove the need for additional server resources. He says any requests, like an increase in RAM, are now data-backed. Previously, the team relied on basic monitoring to determine if systems were operational, which made justifying resource needs challenging. Now, requests for resources are based on clear metrics, which fosters a productive dialogue with IT teams.

“Before [ArcGIS Monitor], our monitoring was mostly just really based on whether the system was up or down,” says Terry. “And because [now we can see] patterns, we can tell when something’s wrong…and be proactive in our solution, which has been our goal all along.”

ArcGIS Monitor analysis views are frequently used, and notifications are configured to alert the team to important system issues and understand patterns in their system. This helps them avoid future errors. Terry created two analysis views, one with a system overview and metrics on RAM, CPU, storage, and pagefiles, and a second that is more GIS service-focused and has allowed the GIS team to better tune system performance.

Terry explains, “This [second analysis view] has been…a big money saver because it’s allowed us to tune our services in a way that we can keep our CPU and RAM in check and is preventing us from going to a higher AWS tier and just having to pay higher server costs.”

Terry has shown Monitor to senior leadership and received a positive response, including from ADEQ’s chief data officer, who was impressed by the detailed metrics available to help inform key decisions. As a civil servant, Terry emphasizes the importance of responsibly managing resources and ensuring the system runs as efficiently as possible.

“Arizonans treasure the unique environment of our state and its essential role in sustaining well-being and economic vitality today and for future generations,” says Terry.