{"id":2282342,"date":"2024-03-13T10:00:46","date_gmt":"2024-03-13T17:00:46","guid":{"rendered":"https:\/\/www.esri.com\/arcgis-blog\/?post_type=blog&p=2282342"},"modified":"2024-03-15T08:54:40","modified_gmt":"2024-03-15T15:54:40","slug":"dev-summit-2024-observability-and-reliability-in-arcgis-enterprise-on-kubernetes","status":"publish","type":"blog","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/administration\/dev-summit-2024-observability-and-reliability-in-arcgis-enterprise-on-kubernetes","title":{"rendered":"Dev Summit 2024: Observability and reliability in ArcGIS Enterprise on Kubernetes"},"author":15561,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_searchwp_excluded":""},"categories":[37501],"tags":[759712,771782,376702,770992],"industry":[],"product":[36571],"class_list":["post-2282342","blog","type-blog","status-publish","format-standard","hentry","category-administration","tag-arcgis-enterprise-on-kubernetes","tag-business-operations","tag-high-availability","tag-performance-monitoring","product-arcgis-enterprise"],"acf":{"authors":[{"ID":15561,"user_firstname":"Tori","user_lastname":"O hara","nickname":"Tori O'Hara","user_nicename":"tohara","display_name":"Tori O'Hara","user_email":"TOhara@esri.com","user_url":"","user_registered":"2020-03-09 20:04:56","user_description":"Tori is a technical writer on the ArcGIS Enterprise team.","user_avatar":""},{"ID":7561,"user_firstname":"Bill","user_lastname":"Major","nickname":"bmajor","user_nicename":"bmajor","display_name":"Bill Major","user_email":"bmajor@esri.com","user_url":"","user_registered":"2018-03-22 19:23:54","user_description":"Bill is a lead software development engineer on the ArcGIS Enterprise team, focused on ArcGIS Notebooks, security, Kubernetes, and framework development.","user_avatar":""}],"short_description":"ArcGIS Enterprise on Kubernetes has a number of features that can keep production environments reliable, resilient, and operational.","flexible_content":[{"acf_fc_layout":"content","content":"

When the unexpected happens in your production environment, you hope that it is resilient enough to remain operational and recover as quickly as possible. With an ArcGIS Enterprise on Kubernetes deployment, organizations have access to multiple features that can provide this level of resiliency. In their plenary demonstration, Chris and Bill highlight some of these features, including ArcGIS Enterprise on Kubernetes\u2019s support for the multiple availability zone deployments.<\/p>\n"},{"acf_fc_layout":"kaltura","video_id":"1_312rzjmx","time":true,"start":"387","stop":"752"},{"acf_fc_layout":"content","content":"

Configuring a highly available deployment<\/strong><\/h2>\n

For their demonstration, Chris and Bill have set up a highly available deployment that will be undergoing a chaos test. While an ArcGIS Enterprise on Kubernetes deployment has built-in high availability<\/a>, Chris and Bill are also leveraging a number of other features to reduce the amount of downtime, data loss, and improve resiliency.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2282412,"id":2282412,"title":"grafana dashboard","filename":"grafana-dashboard.png","filesize":259222,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/grafana-dashboard.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/administration\/dev-summit-2024-observability-and-reliability-in-arcgis-enterprise-on-kubernetes\/grafana-dashboard","alt":"","author":"15561","description":"","caption":"The current Grafana dashboard view of Chris and Bill's Kubernetes cluster ahead of the chaos test.","name":"grafana-dashboard","status":"inherit","uploaded_to":2282342,"date":"2024-03-12 16:46:59","modified":"2024-03-12 16:57:33","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1429,"height":574,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/grafana-dashboard-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/grafana-dashboard.png","medium-width":464,"medium-height":186,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/grafana-dashboard.png","medium_large-width":768,"medium_large-height":308,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/grafana-dashboard.png","large-width":1429,"large-height":574,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/grafana-dashboard.png","1536x1536-width":1429,"1536x1536-height":574,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/grafana-dashboard.png","2048x2048-width":1429,"2048x2048-height":574,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/grafana-dashboard-826x332.png","card_image-width":826,"card_image-height":332,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/grafana-dashboard.png","wide_image-width":1429,"wide_image-height":574}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Cloud native object stores<\/em><\/h3>\n

Starting at the 11.2 release, ArcGIS Enterprise on Kubernetes organizations can use cloud native services to integrate cloud object stores.<\/a> Cloud object stores act as the organization\u2019s object store or backup store location. This aids in increasing reliability and resiliency while also reducing the demand on in-cluster resources.<\/p>\n

Multi-AZ deployments<\/em><\/h3>\n

Also starting at 11.2, ArcGIS Enterprise on Kubernetes deployments can deploy their Kubernetes cluster across multiple availability zones<\/a>. By using topology spread constraints<\/a>, administrators can control how scheduling occurs across the cluster. For Chris and Bill\u2019s deployment, each availability zone becomes the separation boundary between replicas of each workload.<\/p>\n

Enhanced availability architecture profile<\/em><\/h3>\n

The enhanced availability architecture profile<\/a> is designed for use in business or mission-critical production environments and provides the highest level of availability, as it includes increased and expanded redundancy across critical pods. If an organization is configured to use multiple availability zones, the enhanced availability profile is the only profile that guarantees adequate coverage for all stateful workloads in the case of an availability zone failure.<\/p>\n

 <\/p>\n

Testing reliability when faced with failure<\/strong><\/h2>\n

Chris begins his demonstration by showing the current state of his organization. Due to the interplay between the multiple availability zone deployment, enhanced availability profile, and cloud native object store, the organization is set up to be able to withstand a significant outage.<\/p>\n

To put this to the test, Chris stops one of the availability zones, disrupting the cluster. As a result, machines have terminated, and numerous pods are shifting from a running to pending state.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2282402,"id":2282402,"title":"pods after failure","filename":"pods-after-failure.png","filesize":197376,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-after-failure.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/administration\/dev-summit-2024-observability-and-reliability-in-arcgis-enterprise-on-kubernetes\/pods-after-failure","alt":"","author":"15561","description":"","caption":"A view of the pods within Chris and Bill's Kubernetes cluster, immediately after the availability zone has been stopped.","name":"pods-after-failure","status":"inherit","uploaded_to":2282342,"date":"2024-03-12 16:46:56","modified":"2024-03-12 16:57:44","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":779,"height":433,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-after-failure-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-after-failure.png","medium-width":464,"medium-height":258,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-after-failure.png","medium_large-width":768,"medium_large-height":427,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-after-failure.png","large-width":779,"large-height":433,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-after-failure.png","1536x1536-width":779,"1536x1536-height":433,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-after-failure.png","2048x2048-width":779,"2048x2048-height":433,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-after-failure.png","card_image-width":779,"card_image-height":433,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-after-failure.png","wide_image-width":779,"wide_image-height":433}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Observability and monitoring after failure<\/h2>\n

With Chris\u2019s chaos test implemented, Bill now needs to evaluate the impact of the zone\u2019s outage. From the number of pods in a pending state, we can see that Kubernetes is attempting to rebalance after the loss in capacity.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2282392,"id":2282392,"title":"pods rebuilding","filename":"pods-rebuilding.png","filesize":177830,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-rebuilding.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/administration\/dev-summit-2024-observability-and-reliability-in-arcgis-enterprise-on-kubernetes\/pods-rebuilding","alt":"","author":"15561","description":"","caption":"A view of the pods in Chris and Bill's Kubernetes cluster, showing that Kubernetes is attempting to restart, schedule, and rebalance after the outage.","name":"pods-rebuilding","status":"inherit","uploaded_to":2282342,"date":"2024-03-12 16:46:51","modified":"2024-03-12 16:57:52","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":778,"height":423,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-rebuilding-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-rebuilding.png","medium-width":464,"medium-height":252,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-rebuilding.png","medium_large-width":768,"medium_large-height":418,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-rebuilding.png","large-width":778,"large-height":423,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-rebuilding.png","1536x1536-width":778,"1536x1536-height":423,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-rebuilding.png","2048x2048-width":778,"2048x2048-height":423,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-rebuilding.png","card_image-width":778,"card_image-height":423,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/pods-rebuilding.png","wide_image-width":778,"wide_image-height":423}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Next, Bill moves to check ArcGIS Enterprise Manager to check if there are any critical logs. The lack of critical logs shows that the enhanced availability architecture profile is providing the expected resiliency.<\/p>\n

Bill then moves on to check perform a basic health check on the core framework services, which shows that the relational store health check has failed.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2282472,"id":2282472,"title":"basic health check","filename":"basic-health-check.png","filesize":184935,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/basic-health-check.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/administration\/dev-summit-2024-observability-and-reliability-in-arcgis-enterprise-on-kubernetes\/basic-health-check","alt":"","author":"15561","description":"","caption":"The results of the basic health check, showing that the relational store health check has failed.","name":"basic-health-check","status":"inherit","uploaded_to":2282342,"date":"2024-03-12 16:56:54","modified":"2024-03-12 16:58:01","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1331,"height":599,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/basic-health-check-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/basic-health-check.png","medium-width":464,"medium-height":209,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/basic-health-check.png","medium_large-width":768,"medium_large-height":346,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/basic-health-check.png","large-width":1331,"large-height":599,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/basic-health-check.png","1536x1536-width":1331,"1536x1536-height":599,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/basic-health-check.png","2048x2048-width":1331,"2048x2048-height":599,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/basic-health-check-826x372.png","card_image-width":826,"card_image-height":372,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/basic-health-check.png","wide_image-width":1331,"wide_image-height":599}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

When Bill validates the relational data store, it returns a warning status. This means that the primary relational store is healthy, but the standby is not.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2282382,"id":2282382,"title":"data store warnings","filename":"data-store-warnings.png","filesize":133689,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/data-store-warnings.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/administration\/dev-summit-2024-observability-and-reliability-in-arcgis-enterprise-on-kubernetes\/data-store-warnings","alt":"","author":"15561","description":"","caption":"The results after validating the relational store in ArcGIS Enterprise Manager.","name":"data-store-warnings","status":"inherit","uploaded_to":2282342,"date":"2024-03-12 16:46:44","modified":"2024-03-12 16:58:12","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1429,"height":390,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/data-store-warnings-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/data-store-warnings.png","medium-width":464,"medium-height":127,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/data-store-warnings.png","medium_large-width":768,"medium_large-height":210,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/data-store-warnings.png","large-width":1429,"large-height":390,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/data-store-warnings.png","1536x1536-width":1429,"1536x1536-height":390,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/data-store-warnings.png","2048x2048-width":1429,"2048x2048-height":390,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/data-store-warnings-826x225.png","card_image-width":826,"card_image-height":225,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/data-store-warnings.png","wide_image-width":1429,"wide_image-height":390}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Though not highly available, the organization remains available in this degraded state\u00a0. Bill moves on to monitoring the organization to track current performance through his Grafana dashboard.<\/p>\n

The first chart in the dashboard shows the successful HTTP requests over time. This chart shows that while there is an impact to throughput while Kubernetes is rebalancing, all services are continuing to respond and are returning to their expected performance.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2282432,"id":2282432,"title":"requests over time","filename":"requests-over-time.png","filesize":482508,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/requests-over-time.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/administration\/dev-summit-2024-observability-and-reliability-in-arcgis-enterprise-on-kubernetes\/requests-over-time","alt":"","author":"15561","description":"","caption":"A chart monitoring the HTTP requests occurring over time.","name":"requests-over-time","status":"inherit","uploaded_to":2282342,"date":"2024-03-12 16:48:27","modified":"2024-03-12 16:58:19","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1429,"height":453,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/requests-over-time-213x200.png","thumbnail-width":213,"thumbnail-height":200,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/requests-over-time.png","medium-width":464,"medium-height":147,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/requests-over-time.png","medium_large-width":768,"medium_large-height":243,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/requests-over-time.png","large-width":1429,"large-height":453,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/requests-over-time.png","1536x1536-width":1429,"1536x1536-height":453,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/requests-over-time.png","2048x2048-width":1429,"2048x2048-height":453,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/requests-over-time-826x262.png","card_image-width":826,"card_image-height":262,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/requests-over-time.png","wide_image-width":1429,"wide_image-height":453}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

The second chart shows request failures. While some errors have occurred, this chart also shows that critical services remain \u00a0operational even while Kubernetes is working to rebalance after the outage.<\/p>\n"},{"acf_fc_layout":"image","image":{"ID":2282372,"id":2282372,"title":"failed requests over time","filename":"failed-requests-over-time.png","filesize":89778,"url":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/failed-requests-over-time.png","link":"https:\/\/www.esri.com\/arcgis-blog\/products\/arcgis-enterprise\/administration\/dev-summit-2024-observability-and-reliability-in-arcgis-enterprise-on-kubernetes\/failed-requests-over-time","alt":"","author":"15561","description":"","caption":"A chart showing the error count of all requests over time.","name":"failed-requests-over-time","status":"inherit","uploaded_to":2282342,"date":"2024-03-12 16:46:14","modified":"2024-03-12 16:58:27","menu_order":0,"mime_type":"image\/png","type":"image","subtype":"png","icon":"https:\/\/www.esri.com\/arcgis-blog\/wp-includes\/images\/media\/default.png","width":1429,"height":182,"sizes":{"thumbnail":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/failed-requests-over-time-213x182.png","thumbnail-width":213,"thumbnail-height":182,"medium":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/failed-requests-over-time.png","medium-width":464,"medium-height":59,"medium_large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/failed-requests-over-time.png","medium_large-width":768,"medium_large-height":98,"large":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/failed-requests-over-time.png","large-width":1429,"large-height":182,"1536x1536":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/failed-requests-over-time.png","1536x1536-width":1429,"1536x1536-height":182,"2048x2048":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/failed-requests-over-time.png","2048x2048-width":1429,"2048x2048-height":182,"card_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/failed-requests-over-time-826x105.png","card_image-width":826,"card_image-height":105,"wide_image":"https:\/\/www.esri.com\/arcgis-blog\/app\/uploads\/2024\/03\/failed-requests-over-time.png","wide_image-width":1429,"wide_image-height":182}},"image_position":"center","orientation":"horizontal","hyperlink":""},{"acf_fc_layout":"content","content":"

Conclusion<\/strong><\/h2>\n

During their presentation, Chris and Bill showed how ArcGIS Enterprise on Kubernetes remains reliable, resilient, and operational, even when faced with critical failures.<\/p>\n

For more information on ArcGIS Enterprise on Kubernetes, and to see if this deployment solution is the right one for your organization, use the links below to reference our documentation and other blogs:<\/p>\n