Natural disasters, infrastructure issues, and power outages may cause systems to unexpectedly go offline, become unresponsive, or crash at any time. Because of this, the best time to plan for that worst-case scenario and recovery options is at the time of system design and creation; the second-best time is now.
Sound backup and recovery strategies are an often-overlooked component of enterprise systems and require software administrators to balance minimal data loss against performance hits and storage space when building a recovery strategy. However, the ability to take a save point of your organization’s work and reliably return to it in case of an emergency is essential to building trust with your end users as well as your colleagues, and is worth the effort it takes to plan and set up.
This blog provides some best practices for taking backups of ArcGIS Enterprise. Through reading this you will learn the basics of machine, software, and referenced data level backups, and how to begin to orchestrate your backup strategy to best protect your data and access in emergencies.
Backup Anatomy
What does it mean when we say, “Are backups available for this deployment?” This question can refer to any of the following backup types:
- Application-level backups: ArcGIS Enterprise components and managed content, configurations, and groups
- VM or system-level backups: The operating system and machine on which ArcGIS Enterprise runs
- Data-level backups: Content which is managed by other software / sources like: RDBMS, Cloud Data Warehouses, Data Lakes
A comprehensive backup strategy addresses each of these domains. No one tool can address all of these at once, so ensuring each of these backups runs successfully is important. In the segments below, we will talk about examples of how each of these domains can be addressed through a different tool or method.
Note: Esri recently published the ArcGIS Architecture Center which contains the ArcGIS Well-Architected Framework resource. This is an excellent source of information about installing ArcGIS Enterprise to fulfil your organization’s requirements.
Application-Level Backups
ArcGIS Enterprise on Windows and Linux includes a utility called the Web GIS Disaster Recovery tool (usually referred to by the short name webgisdr) that exports a backup file of an entire ArcGIS Enterprise deployment. The file contains information on services published to ArcGIS Enterprise, along with some security settings, utility services, and federated ArcGIS Server information. The webgisdr tool will not backup the machine’s file system, create backups of referenced data sources, or hardware specific settings (e.g. machine settings, network settings).
The webgisdr tool works by launching backup commands on ArcGIS Enterprise’s ArcGIS Server, ArcGIS Data Store, and Portal for ArcGIS components. The outputs of these files are bundled and compressed into one export file, which can be used to restore the content information of ArcGIS Enterprise to that point in time.
The webgisdr workflow is resource intensive for each of the machines participating in the ArcGIS Enterprise deployment, so it is important for administrators to time this process to execute during off-peak hours, ideally when no changes are being made to the organization. To alleviate these resource concerns, as well as to provide more frequent backups, webgisdr supports two running modes: ‘full’ and ‘incremental’.
Full backups take a complete snapshot of ArcGIS Enterprise and each of its components. It contains information on all the services published to ArcGIS Enterprise, hosted feature layer data, group and member information and much more. Full backups take more time and space to run but should be considered a full standalone export of the deployment.
Incremental backups are created based on an existing full backup to include updates that may have occurred between the two backups. The advantage of an incremental backup is that they generally take less time and resources to create than full backups and are smaller in file size. Even though they are more convenient, administrators should consider that they will essentially need to run two restore operations in the event of an incident, which may take more time depending on the length of time between the full backup and the incremental backup.
The webgisdr tool is how ArcGIS Enterprise in Windows and Linux protects its managed content through a single tool, however what if you are using ArcGIS Enterprise on Kubernetes?
An ArcGIS Enterprise deployment on Kubernetes includes the ArcGIS Enterprise Manager and its API which allow you to create a backup of the system configuration, hosted geospatial data, and organizational items and content. In addition to using block storage devices or file storage, ArcGIS Enterprise on Kubernetes lets you use cloud-native services such as Amazon S3, Azure Blog, or Google Cloud Storage. Through ArcGIS Enterprise Manager, the administrator of the organization can:
- Schedule backups
- Set backups to recur at a specific time interval
- Define how long backups will be retained and when they will be deleted
- Encrypt the backups
More information on how to set up and maintain backups on ArcGIS Enterprise on Kubernetes can be found here: Data loss and downtime minimization.
Backing up ArcGIS Enterprise on Windows, Linux and Kubernetes through their respective methods is fully supported by Esri Technical Support, and issues with creating or importing these backups can be discussed directly with Esri Technical Support. However, VM or system-level and Data-level backups are not supported by Esri Technical Support.
Now that we’ve discussed how we can safeguard ArcGIS Enterprise at its built in application level, let’s address what steps you should take to safeguard your machines and reference content.
VM or System-Level Backups
It is important to consider the underlying architecture and configurations of a system, and to have a strategy to recover these if needed. These strategies will vary depending on what type of underlying architecture is running ArcGIS Enterprise, so we will focus on a common example.
If working with virtual machines through a cloud provider, a backup may take the form of a machine snapshot. This can be a suitable way to “roll back” a machine to a point in the past where the software and hardware was working properly. However, there are some important considerations to think about when creating these backups.
- Machine state: When snapshots of virtual machines are taken, Esri software should not be running. This prevents changes within the organization, which may cause ArcGIS Enterprise components to not start upon rolling back to the image.
- Content consistency: Machine snapshots can be paired with a webgisdr backup to ensure that content available in ArcGIS Enterprise matches what is present in the underlying file structure of the host machine.
Coordinating the machine snapshot and webgisdr creates an application-consistent backup that can be used to recover an ArcGIS Enterprise deployment and its underlying infrastructure.
The sum of these considerations can be used to create a backup that covers the content and configuration of ArcGIS Enterprise as well as the hardware. The last domain we should consider are outside sources of data through referenced data stores.
Data-Level Backups
ArcGIS Enterprise supports creating referenced services from a variety of data sources. Content can be published referencing a variety of data sources, including but not limited to: Oracle databases, Microsoft’s SQL Server, PostgreSQL and others. Each of these database providers contains utilities and methods to back up the tables contained within.
Geodatabases are not the only type of referenced data supported by ArcGIS Enterprise. Image services can work with rasters that are contained within a file share; knowledge graphs can work off graph stores that are independent of the database provider. It is important to fully understand the footprint of your referenced content and have plans available to back up each of these.
Although this blog will not go into depth about specific considerations for different types of referenced content, consistency with the other two domains is crucial in developing a sound strategy. Administrators should aim to fully backup their referenced datastores during times where no edits are being conducted, ideally at the same time as the machine and software level backups are created. This ensures that there is data consistency with the machine and software level backups.
Bring It All Together
So far, we’ve identified three levels of backups: software, machine, and referenced data stores. The common thread that ties these together is time: backups should be taken at roughly the same time when new edits are not happening to ensure that data is being preserved in the process. This can involve down time, but at the very least should be accomplished when the system is not being used to edit, create, or delete information.
However, these actions can be fairly time consuming if they are being done manually. Each backup option we’ve discussed so far supports some level of automation. The webgisdr tool can be configured to run off a predetermined configuration file, thereby enabling users to set up scripts that automatically kick off that process. Similarly, most other backup options often rely on scheduling and scripting to be consistently enabled. Studying your organization’s usage habits and utilizing automation is a good way to ensure that backups are taken regularly.
Lastly, we need to discuss validating your backup strategy. Validating a backup should involve restoring and proving the acceptability of the restored environment. Validation should occur before a backup is needed on a non-production version of your system. This serves to ensure that your strategy will work as intended and that nothing has been missed. Validation also includes regularly practicing recovery, especially when the scale of ArcGIS Enterprise increases to include new data sources, products, and procedures.
Parting Thoughts
This is just an overview of how to create a holistic backup strategy but will be a good starting place for ArcGIS Enterprise Administrators looking to make sure that their Enterprise deployment is backed up at all levels. Through careful planning, process automation, and validation you can ensure that your system is protected against data loss during unexpected system failures.
Article Discussion: