Administrators of ArcGIS Server are always looking for better tools to monitor, secure, optimize, and troubleshoot their sites.
This month’s release of ArcGIS Enterprise 10.7 brings several new features server administrators can use to better understand what’s going on under the hood, and to keep server sites running their best.
All these features and properties apply both to ArcGIS Server sites that are federated with an ArcGIS Enterprise portal, and to sites in a stand-alone configuration.
Work with geoprocessing jobs in ArcGIS Server Manager
At 10.6.1, we introduced a suite of operations and resources in the ArcGIS Server Administrator Directory that allow you to monitor and intervene in your site’s current geoprocessing jobs. At 10.7, those tools are now accessible in ArcGIS Server Manager.
The new Jobs page, under the Site tab, offers a full query function, with several categories of filters. You can view all the jobs running on a specific service; those currently carrying a specific status (such as WAITING or EXECUTING); or those assigned to a specific machine.
When jobs appear in your query, you can interact and intervene with them. If a job hasn’t finished yet, you can pause or cancel it.
This comes in handy when a job is running slowly or isn’t completing – perhaps due to machine downtime or inadequate computing resources – or when you need to clear the queue of jobs waiting to run, in order to prioritize a specific job.
You can also delete a job, canceling the job if it is running and deleting all files and folders related to the job from the system. Use this to clear your disk space when you know a large number of jobs will be submitted soon.
Conserve memory using the shared instance pool
Beginning at 10.7, ArcGIS Server includes a new feature called the shared instance pool. As opposed to the existing model, where each service running on a site has its own dedicated pool of service instances, services that use the shared pool use its instances to handle their requests – and consume no additional memory when they aren’t in use.
Shared instances provide a better solution to the problem of balancing site performance with available machine resources.
For compatible services that don’t constantly receive traffic, using the shared pool allows you to provide appropriate resources to handle their traffic, while drastically reducing their memory footprints on your machines.
For a complete explanation of service instances and the shared instance pool, check out this week’s blog – replete with gym sock GIFs. (We’ll explain.)
Filter logs by Request ID
A new property is attached to all service requests in ArcGIS Server that uniquely identifies each individual request.
This Request ID value is recorded in each log message that is written for the request during its life cycle in the ArcGIS Server site.
This enables you to find just the log messages that you are looking for in a much simpler way than before, where you might have had to sort through many other interleaved log messages from other requests and events happening simultaneously with the specific request you are trying to troubleshoot.
How the Request ID helps troubleshooting
Let’s say my ArcGIS Server site has been regularly failing a certain job. I go to the Logs page in ArcGIS Server Manager, use the Columns button to add Request ID to the displayed columns, and run a first pass to query all the logs from the past hour. Scrolling through, I find a log message regarding my failed job.
That message carries a Request ID value, which I can copy. Navigating to the logs query page in the ArcGIS Server Administrator Directory, I’ll run a new query for all logs carrying this specific Request ID.
In the results, I can examine all logs dealing with this specific job. I find one message indicating that the job failed because the data set was too large. Hmm. If only there were a way for me to raise the limit for memory allocated to any one service…
Set a Java Heap Size value for individual services
The Java heap size associated with a service affects the amount of data that the service can process at once. This advanced setting commonly comes in to play when large files need to be uploaded to a service (such as with some geoprocessing services), or when large responses are generated and must be sent to the client (also usually for geoprocessing services, or some particularly large queries to feature services).
You can already change the default heap size limit for all services running in your site using heap size properties in the ArcGIS Server Administrator Directory. But if you raise the limit for all services, that could strain the memory resources for your machines.
Now at 10.7, you can change the heap size for any individual service published to your site using the javaHeapSize property. This property is available through the Edit Service operation.
Being able to set it at a per-service level means you don’t have to worry about this change taxing your resources, unlike the properties that apply to all services. After all, when you raise the Java heap size, you’re usually doing so in order to resolve issues with a specific service.
Returning to my hypothetical troubleshooting scenario: I’ve determined that my job was failing because its input was a file that was too large to be processed. Now I have a solution!
First, I’ll use the new Jobs page in ArcGIS Server Manager to cancel the job that’s currently waiting to be processed. (See? It all fits together.)
Then, I’ll log in to the ArcGIS Server Administrator Directory, navigate to the appropriate service, and edit its service properties to specify a javaHeapSize framework property of 128 MB.
When I click Save Edits, the service will be restarted with its new memory settings, and I can retry running my job.
Improve efficiency in multiple-machine sites
Prior to 10.7, if one or more machines in a multiple-machine site went down or was unavailable, the performance of administrative and publishing operations would be sluggish until a site administrator could resolve the problem or unregister the machine.
This is especially pertinent in large cloud deployments – for example, if AWS instance is removed from a deployment due to auto-scaling logic, or an abrupt shutdown due to a failure.
In 10.7, we’ve introduced a way to automate the monitoring of machine activity and step in when a machine has been unresponsive for a while.
Now, machines that are part of an ArcGIS Server site regularly report their active status or “heartbeat” to the configuration store. This heartbeat, or the lack of it, allows us to set new properties that monitor and intervene when a machine has been inactive for a certain duration.
The first property will suspend a machine automatically after it’s been inactive for a certain period. Suspension means that the machine will not receive administrative or publishing requests from the ArcGIS Server site.
By default, a property called machineSuspendThreshold is set to 60 minutes. This means that if a machine has not updated its “heartbeat” in the past 60 minutes, the site will suspend it from receiving administrative and publishing requests.
The site won’t have to wait for the machine to respond, thereby avoiding performance costs when the machine goes down. If the machine comes back online and updates its “heartbeat”, it will be reinstated to receive requests.
You can customize the machineSuspendThreshold by specifying a new value for this server property in the ArcGIS Server Administrator Directory.
A second property, called suspendedMachineUnregisterThreshold, goes a step further and determines when to unregister the machine from the site completely.
This is an irreversible operation; accordingly, it’s disabled by default (set to -1). If you update this value, it should be higher than that of the machineSuspendThreshold.
This property is likely to be most useful for administrators of large ArcGIS Server sites running in the cloud; if a machine fails or is shut down by cloud auto-scaling mechanisms, it can be removed from the site automatically.
Default security improvements
We are always strengthening the default security posture for ArcGIS Server sites, as well as the optional stringent measures administrators can take to make their sites even more secure. At 10.7, there are several security changes that server administrators should be aware of.
By default, at 10.7, ArcGIS Server only allows communication over HTTPS and port 6443. Previously, the default setting was to allow both HTTP and HTTPS communication. It’s highly recommended to keep this setting at “HTTPS Only” to ensure the confidentiality of your data during use.
The Transport Layer Security (TLS) protocol secures the interactions between ArcGIS Server and client apps that make requests to it. At 10.7, the default setting now only supports communication over TLS version 1.2. You can modify this setting using the ArcGIS Server Administrator Directory to also allow communication over TLS versions 1.0 and/or 1.1, but it’s a security best practice to allow only the most recent version of the protocol.
Note: These changes aren’t just taking place for ArcGIS Server. At 10.7, the ArcGIS Enterprise portal also only supports HTTPS and TLS 1.2 by default. And in April, ArcGIS Online will switch to only allow TLS 1.2. To learn more and prepare for the change, see https://support.esri.com/en/tls.
While HTTPS and the latest version of TLS help defend against man-in-the-middle attacks, another common security threat is vulnerability to cross-site scripting (XSS) attacks. One such type of attack exploits a web browser’s act of MIME-sniffing – analyzing an asset and then changing its content-type – to disguise malicious content as something else.
To protect against this class of security vulnerabilities, ArcGIS Server 10.7 now sends a content-type header, nosniff, that prevents users’ web browsers from MIME-sniffing. This header is sent by default.
One last note. At 10.7, we’ve significantly updated the ArcGIS Server documentation for administrators. You can now find content related to site deployment – both planning a site and configuring it once you’ve installed the software – in the Deploy section. The Administer section has been reorganized for clarity and ease of use. And guides to developing solutions and extensions for ArcGIS Server can be found in the Develop section.
We hope you’ll find it a welcome change!