Contributors to this article include our presenters at the User Conference 2021 technical workshops on Kubernetes: Nidhi Arora, Subrat Bora, Laurence Clinton, Moginraj Mohandas, Eva Mui, Jon Quinn, Sara Sanchez, Shreyas Shinde, Vaibhav Singh, Trevor Seaton, Jeff Smith, Garima Tiwari, Gayathri Vijay, Markus Walker, and Caroline Wright.
Thank you to everyone who attended our User Conference in 2021! Once again it was a virtual conference, which means we could not interact with you in person, but it did provide an excellent opportunity to hear from and share with our customers, partners, and GIS colleagues from around the world. This article is a follow-up to the technical sessions that focused on ArcGIS Enterprise on Kubernetes, with responses to questions submitted during the sessions.
At the conference, we were able to share with you details about how the launch of ArcGIS Enterprise 10.9 included the ability for eligible customers to deploy onto your own managed Kubernetes environment. This deployment option continues to provide the scalability and resilience you expect from ArcGIS Enterprise, with the added benefits of a cloud native architecture bringing manageability and elasticity.
If you attended the conference, you still have access to recorded sessions for a short time within the conference platform. We have published a blog with highlights from the product team Q&A session at the conference. The plenary presentations are already published to YouTube, and soon we will add recordings from the technical sessions as well. Meanwhile, our resource page for ArcGIS Enterprise on Kubernetes has plenty of additional resources, including links to technology demonstrations. And don’t miss our product documentation!
Without further ado, here are questions you submitted in your own words, along with our responses. Reach out to your Esri representative if you have any questions about ArcGIS Enterprise on Kubernetes. We hope to see you in person at next year’s conference, if not sooner!
Scope, Eligibility, and Licensing
Q: By ArcGIS Enterprise on Kubernetes, do you mean it is based on Kubernetes PaaS offerings such as AKS from Azure? Or can we use open-source Kubernetes stack?
A: We support specific Kubernetes environments provided by Amazon AWS (EKS), Microsoft Azure (AKS), and Red Hat OpenShift. Soon we will also support Google Kubernetes Engine (GKE). We do not support customers assembling their own deployment or custom Kubernetes cluster using selective container images, as the software is intended to be deployed as a single holistic, interdependent solution. We’d like to know more about your specific requirements, so please talk with your Esri representative. As far as functionality, at 10.9 we had to defer some capabilities until future releases. We are working toward functional parity with Windows and Linux as a top priority. You can read more here about what is included with ArcGIS Enterprise 10.9 on Kubernetes.
Q; Will the release of ArcGIS Enterprise on Kubernetes be only available to certain customers or be available for anyone to purchase?
A: Currently, access is subject to eligibility requirements including an active Enterprise Agreement. More detail is described here, but please also talk with your Esri representative. We are evaluating potential adjustments to eligibility in the future and will share details as soon as we can.
Q: If no one at our organization uses Kubernetes, are the Windows and Linux deployment options still viable for us?
A: Yes. Kubernetes is a new option to deploy ArcGIS Enterprise, but it is still one product. The Windows & Linux deployment options remain fully viable, supported, and will probably remain appropriate for many of our customers. You are not missing out on unique GIS user functionality if you decide not to deploy onto Kubernetes. If your team has a modern cloud strategy calling for a distributed architecture based on microservices, with software delivered via containerization, or if you need your large or complex ArcGIS Enterprise deployments to become manageable, we think you should be looking into Kubernetes.
Q: How will licensing change with Kubernetes?
A: When we launched 10.9, we intentionally kept things simple and consistent, requiring an Enterprise Agreement as part of eligibility, including requiring that you use 10.9 licenses for ArcGIS Server and portal user authorization – the JSON and ECP or PRVC files you are used to. In the future, we do expect there will be changes to how access to Kubernetes is licensed, that may differ from how you license the Windows or Linux deployment options. We are in detailed planning now. We know you want – and deserve – more details; please talk with your Esri representative with your specific input or concerns.
Q: Can Esri provide some sample pricing calculators for the various Kubernetes architecture profiles on supported providers?
A: Thank you for this question and idea. We provide System Requirements in our product documentation, and you could use this as input into the cloud providers’ service cost calculators such as this one available on Azure; the AWS console provides something similar. However, Esri does not maintain our own tool, in part due to changing components and pricing for hardware and services cost, which are highly dynamic.
Q: For a deployment of ArcGIS Enterprise on Kubernetes would a Kubernetes deployment cost more or less than a traditional deployment with similar performance specs in a similar environment such as Microsoft Azure?
A: This will depend on many variables such as level of resilience desired, up-time requirements, performance requirements, acceptable levels of latency and computing power needed and much more. We encourage you to talk with your Esri representative to analyze how the infrastructure costs may be impacted in your situation. However, most customers report that they expect to achieve a higher level of utilization, without having to maintain unused system resources designed to handle unexpected peak usage. This is due to the improved elasticity of the Kubernetes architecture along with the ability to quickly scale services.
Q: Please explain how licensing works for ArcGIS Enterprise on Kubernetes at 10.9.
A: With 10.9, once an eligible customer activates Kubernetes they provision an ArcGIS Server license file (ECP or PRVC) and portal license file (JSON) and upload them during the process to create an organization. Obtaining authorization files and uploading the files while creating an organization is described in our product documentation.
Q: Do you have documentation on how to set up ArcGIS Enterprise on Kubernetes in AWS/Azure or on Prem?
A: Thanks for the question! We have extensive product documentation available. Please see System Requirements and planning your deployment as examples.
Q: Are there plans to introduce training/webinars/seminars on the Esri Training Catalog for ArcGIS Enterprise on Kubernetes environments to help build customer confidence…?
A: We encourage you first to check out resources, such as curricula or learning paths already provided by Kubernetes providers such as Amazon AWS, Microsoft Azure, or Red Hat OpenShift. The Kubernetes.io site also has excellent documentation and resources. Moreover, we are working with our Partners team and Esri partner network to enable a specialization that will help identify partners able to assist with Kubernetes deployment and management services and consulting. As far as specific Esri training on Kubernetes, we do not yet have specific plans to create content available to customers. However, it’s been an active topic for discussion, so please stay tuned.
Architecture
Q: Am I right with this analogy that ArcGIS Enterprise on Kubernetes is like ArcGIS Online but with increased control over it; and with less cloud infrastructure or maintenance efforts compared to traditional ArcGIS Enterprise?
A: ArcGIS Enterprise on Kubernetes gives you the ability to deploy ArcGIS Enterprise on the Kubernetes platform. You can almost think of it as support on a new deployment platform in addition to Windows and Linux. It provides all the same great capabilities with simplified deployment and upgrades, and ability to scale massively and in an isolated manner.
Q: I’m new to Kubernetes. The training material describes that there must be an existing Kubernetes system. Is there a SaaS where we can use Kubernetes?
A: The Kubernetes deployment option is meant for organizations wanting to run ArcGIS Enterprise on their own infrastructure (on-premises or on the cloud). It is not architected as a Software as a Service (SaaS) offering. Esri’s SaaS offering in that pattern is ArcGIS Online.
Q: How does a Kubernetes deployment compare with clustered-site approach, where multiple servers support a single site?
A: ArcGIS Enterprise on Kubernetes is a deployment of ArcGIS Enterprise onto a Kubernetes environment, which is organized into pods to allocate resources and provide resources, rather than discrete servers or virtual machines. A Kubernetes deployment also has ability to federate additional ArcGIS Server sites to it that are running in Windows or Linux environments. From a scaling standpoint, on Windows and Linux, you will need to scale out to multiple ArcGIS Server machines in your site to improve response times and availability for your services, or alternatively increase instances of your services. In Enterprise on Kubernetes, you can scale each service individually by adding more pods of that service. This means isolated scaling and ability to scale massively, without affecting other services or having to duplicate other services.
Q: Can you explain the architectural differences between shared and dedicated instances in Kubernetes? Are they different than they are in Windows/Linux?
A: In ArcGIS Enterprise on Kubernetes, there are shared pods, that may or may not represent services in your deployment, that host shared instance services. For example, it’s possible that one pod may host all your shared instance map services. Similarly, there are one or more pods of a shared hosted feature server that will host all your hosted feature services. You can scale out the pod replicas for these as needed. Dedicated services run on dedicated pods. So, you could have one or more pods of a certain map service run in your cluster that is only servicing that service. All the system and utility services that are deployed as part of your organization/deployment are also running as dedicated pods that can be scaled out individually.
Q: Is my assumption right that at the low-level docker images and containers are what makes scaling possible? If so, are low level changes permitted on configs?
A: Each deployment’s components and services can be scaled independently due to the new architecture in Kubernetes. By containerizing the ArcGIS Enterprise components, we are now able to manage and scale them independently and administrators can do so in the new ArcGIS Enterprise Manager.
Q: Does Kubernetes automatically scale up the number of instances, memory and CPU devoted to services that are getting very high usage?
A: In 10.9, ArcGIS Enterprise on Kubernetes allows you to scale out the pod replicas of your services or tune the memory and CPU resources for the pods running for a shared or dedicated service, via ArcGIS Enterprise Manager or the Administrator API. We are actively working on being able to automatically scale pod replicas based on some resource criteria, in a future release.
Q: How is autoscaling likely to affect licensing?
A: Currently, we do not support horizontal autoscaling. Most pods can be configured by the administrator by increasing available pods to support the system responsiveness. Please watch the operate and scale demonstration available here or the product documentation available here for more information about scaling.
Q: For updates, what access to the internet is required by the server and which servers require this access?
A: For updates and release upgrades, access to the internet is required for two areas. 1) Access to a container registry host (for example, docker.io) where Esri manages container images, and where the new container images are retrieved. 2) Access to a well-known public URL where a version manifest document, managed by Esri, helps your deployment determine whether an update or a new release is available.
Q: How is a service change pushed to workers? Is it automatic or manual?
A: The service pods (workers in your statement) run a reconciliation loop periodically to test for service changes. If the service that they are observing is changed, they apply the change.
Q: For those of us who are uninitiated, what is the roadmap to lifting the enterprise platform into Microsoft Azure (AKS)? What are the high-level requirements?
A: The System Requirements are documented here. Please reach out to your Esri representative for additional help and details about your specific situation.
Q: Do you already have plans to support other runtimes than Docker (CRI)?
A: This question may be referring to the announcement last year that Docker as a container runtime is deprecated after v1.20 for Kubernetes. This will not impact the containers we have built for ArcGIS Enterprise on Kubernetes. Our container runtimes are based on the Container Runtime Interface and not dependent on the Docker shim which is being deprecated.
Deployment
Q: When will ArcGIS Enterprise on Kubernetes be available for download through MyEsri.com?
A: The software launched as part of the 10.9 release in May 2021. If you are eligible for access to Kubernetes, you will download the deploy script and supporting files from My Esri. The remainder of the software will be pulled from a container image registry managed by Esri. Please talk with your Esri representative about access to the Kubernetes deployment option.
Q: Do you provide install media and updates for private self-hosted container repositories as well?
A: We do not provide media for initial deployment nor updates for ArcGIS Enterprise on Kubernetes. All our containers are hosted in a private organization on Docker Hub. You can pull the containers to your local registry if you need to. To support updates, besides the containers, you will need to download the version manifest file and host it in your local web server. For more information, please contact your Esri representative. We are investigating providing a user-friendly experience to support this workflow.
Q: Will ArcGIS Enterprise on Kubernetes 10.9.1 support Tanzu (VMWare) or Rancher?
A: Not at 10.9.1. In the future, we will be evaluating additional environments such as VMware Tanzu and Rancher. It helps us to learn about your unique requirements for these environments, the timing and urgency you have for them, so that the priorities we set for future releases continue to reflect those of our customers.
Q: Are you planning to support Helm charts?
A: Yes, we are actively looking at further automation and supporting Helm charts. At 10.9 it is not currently supported. We believe that with the Kubernetes deployment option’s deployment script with silent installation options, the ArcGIS Enterprise Manager, the ability to use the Administrator API, and simplified upgrades and updates you can have a streamlined administrative experience for activities such as deploying the software, managing your environment, and keeping it up to date.
Q: Can ArcGIS Enterprise Manager be used to create and manage Enterprise Manager deployments?
A: To deploy ArcGIS Enterprise on Kubernetes, you will use the deploy script, and you can use the same deployment script to deploy to multiple environments. Currently, you can only deploy one organization within a cluster, and ArcGIS Enterprise Manager will only service one organization. If you have specific requirements that prompted this question, please contact your Esri representative so we can learn more about your needs
Q: What is the persistent volume data storage strategy for this implementation? What back-ends are supported – relational databases or file-based sources or enterprise geodatabases?
A: ArcGIS Enterprise on Kubernetes runs hosted data stores. They require persistent volumes configured by the provider of Kubernetes. You can use dynamic provisioning for those cases. You can also register cloud, relational or folder-based data stores containing your data. They can be registered via the Enterprise portal. There are recommendations and best practices for the type of storage used for certain components. For example, for the spatiotemporal and logs store as well as the relational data store, block storage should be used like an EBS volume or Azure disk, rather than network storage like Azure Files or EFS.
Q: When working with Azure AKS, do the enterprise configuration files need to live in Azure Files?
A: The ArcGIS Enterprise configuration files are stored within a hosted relational database that is powered by persistent volumes configured by your Kubernetes administrator. Typically, on Azure you can use Azure Disks or Azure Files as your volume provider. Having said that, Azure Files are Network File Shares (NFS) while Azure Disks are locally mounted disk volumes. So it is recommended that when using persistent volumes for hosted data stores, that you take performance into consideration and prefer Azure Disks instead of Azure Files.
Q: We found ArcGIS Enterprise on Kubernetes to not scale when the Cluster is spawned across multiple availability zones (AZ) because of volume claims. Do you plan to support multiple availability zones for Kubernetes clusters to scale, for example Feature Services?
A: Feature services will scale across multiple AZ. It is a stateless app. All stateless apps will scale across multiple AZ, unlike the stateful sets. We have built redundancy for the stateful sets with architecture profiles which can be deployed in a multiple-AZ cluster.
Q: How straightforward is it to migrate between architectural profiles?
A: There’s no way to migrate between architecture profiles after you have completed the activities of deployment and creating an organization. However, most pods can be scaled independently to support more availability or throughput. System-managed storage , such as relational data store, spatiotemporal and index stores, cannot be scaled.
Administration
Q: Is ArcGIS Enterprise Manager just for Kubernetes?
A: That is correct. ArcGIS Enterprise Manager currently supports ArcGIS Enterprise on Kubernetes only. You will continue to use ArcGIS Server Manager to manage ArcGIS Server on Windows and Linux.
Q: Will the ArcGIS Enterprise on Kubernetes deployment be easier to upgrade than the current ArcGIS Enterprise deployment?
A: One of our core values behind this effort is to streamline procedures such as deployments, upgrades, and updates. Yes, the experience of upgrading in a Kubernetes environment will involve fewer actions required by the administrator, with the upgrade begun from a single click in the ArcGIS Enterprise Manager interface or a call to the ArcGIS Enterprise Administrator API. All the updated software will then be pulled into your environment by the upgrade process.
Q: Is the system available during updates? Any downtime in the update process?
A: Yes, the system is available in read only mode during the update process. In this mode, your organization members can view content, users and groups, and settings but cannot modify them.
Q: Will we need to use the ArcGIS Enterprise Admin API to complete the rollback process for upgrades, or can we simply remove it using the remove option in ArcGIS Enterprise Manager, like that with updates?
A: We can apply the rollback from ArcGIS Enterprise Manager. Once the updates are installed, we can see them listed in the installed tab where it also allows user to manually remove any specific installed update. Note that only updates can be rollbacked. New releases which are available to upgrade the system to a new version of the software with new features, improved functionalities, and sometimes a different look and feel cannot be rolled back. For example, a release upgrade moves the software from version 10.9.0 to version 10.9.1 cannot be rolled back. Therefore, it is always advisable to take backup before starting a new release upgrade.
Q: In Kubernetes are there still walarchive backups every hour or no?
A: No, since incremental backups are not supported in ArcGIS Enterprise on Kubernetes, walarchive logs are not generated. Once they are supported, an administrator will need to opt-in as an explicit decision to generate them, instead of an implicit action as part of creating a backup, as with ArcGIS Enterprise on Windows and Linux at 10.8.1 and earlier.
Security
Q: Does ArcGIS Kubernetes support Azure AD and SAML as identity provider?
A: Yes, ArcGIS Enterprise on Kubernetes fully supports Azure AD as an identity provider using either SAML or OpenID Connect. SAML and OpenID authentication is fully supported with the current release.
Q: Will ArcGIS Enterprise on Kubernetes support OKTA Authentication?
A: Yes, just like with Windows or Linux, ArcGIS Enterprise on Kubernetes supports Okta as an identity provider using either SAML or OpenID Connect.
Q: Can you speak to deploying this behind an organizations firewall? Is that possible or is it only available in a cloud based environment?
A: In addition to cloud based Kubernetes environments like Microsoft Azure (AKS), Amazon AWS (EKS) and – coming soon – Google Cloud (GKE), we also support Kubernetes on the Red Hat OpenShift environment which can be setup within your organization’s firewall.
Q: Is there different functionality that is available in AKS compared to EKS?
A: No, from the standpoint of GIS functionality or ArcGIS administrator functionality, it is the same whether you are deploying onto EKS, AKS, or Red Hat OpenShift. Of course, the underlying implementation of Kubernetes services will be managed differently depending on your chosen environment.
Q: Are there plans to support Nutanix Karbon?
A: Nutanix Karbon is not officially supported and there are no immediate plans to do so. We will continue to evaluate new Kubernetes offerings. We will be expanding to support other environments in the future, for example in the upcoming release we plan to support Google Kubernetes Engine (GKE).
Q: Will there still be a limit of on identity provider to be configured with ArcGIS Enterprise on Kubernetes like the traditional deployments?
A: The same limitations that exist for Enterprise on Windows or Linux also apply for ArcGIS Enterprise on Kubernetes. Only one SAML identity provider can be configured unless you have a federation of identity providers.
Q: If we enable sever or debug logs, can we run them for long time and use them as live log monitoring? As the logs keep growing, will it affect the storage size, or it is treated differently in Kubernetes?
A: Service usage statistics are kept in a special store called the metric store or Prometheus. Your logs are stored in Elasticsearch. They are all connected to persistent volumes. Logs will grow over time, but you can prune and clean them via APIs or the user interface. Export the logs that you want to save over the long term outside your environment.
Q: Can I federate ArcGIS Servers with my ArcGIS Enterprise on Kubernetes deployment?
A: Yes, you can federate an ArcGIS Server of version 10.5 or higher. Supported server licensed roles include ArcGIS Server (Advanced and Standard editions) and ArcGIS Image Server (excluding raster analytics).
Q: Can we deploy the federated server itself in a Kubernetes environment?
A: Yes, users can federate a standalone ArcGIS Server through the Portal UI Settings Page by providing the server URL along with administrator credentials. To clarify, the federated Server can be anywhere and if ArcGIS Enterprise on Kubernetes can communicate with it, federation is possible. The ArcGIS Server can be version 10.5 or higher and must had a licensed role of either ArcGIS GIS Server or ArcGIS Image Server. We do not support installing a standalone containerized ArcGIS Server. ArcGIS Enterprise on Kubernetes is designed to deploy and orchestrate all containers together, to provide the equivalent of a base deployment.
Q: Are you planning on supporting 3rd party logging tools such as Splunk so that we can start to analyze the enterprise server logs on Arc GIS Enterprise Kubernetes or are the logs still in a propriety format. Currently I can monitor Kubernetes health using Splunk.
A: This may be available in future release. The log schema is not in a proprietary/xml format in Enterprise on Kubernetes. It is native UNIX (pipe separated) format. The logs are generated locally in each pod and harvested and written to our Elasticsearch-based log service. We are looking into also writing the logs to standard console so that you could harvest them and store them in Splunk. You can also fetch the logs using our log query Admin API and store them into your stores.
Q; Does ArcGIS Enterprise on Kubernetes support distributed collaboration?
A: Yes, ArcGIS Enterprise on Kubernetes supports distributed collaboration. At 10.9 it is possible for ArcGIS Enterprise on Kubernetes to collaborate with ArcGIS Online. At 10.9.1 it will be possible to also collaborate with ArcGIS Enterprise on Windows/Linux.
Migration and Environments
Q: Can we migrate ArcGIS Enterprise from Windows or Linux directly into Kubernetes? For instance, performing a migration of content between two parallel running ArcGIS Enterprise deployments. Is this possible?
A: We are exploring options to migrate from an existing ArcGIS Enterprise to Enterprise on Kubernetes. However, the migration workflow from ArcGIS Enterprise on Windows or Linux to Kubernetes are not yet supported and unlikely in the next release. You will need to deploy and create an organization on Kubernetes and republish your GIS services from ArcGIS Pro or your Enterprise portal. Stay tuned for updates from our product team as we know this is an area of great interest to customers considering Kubernetes.
Q: if we have a Windows deployment of ArcGIS Enterprise, can we use the WebGISDR tool to export the configuration and import into a new Kubernetes deployment , or are we basically starting from scratch and needing to migrate our data and applications manually?
A: We do not have a migration workflow yet to ArcGIS Enterprise on Kubernetes, which is something we are investigate for future releases. Currently, you won’t be able to export the configurations of ArcGIS Enterprise from Windows or Linux using the WebGISDR tool and import it to the ArcGIS Enterprise on Kubernetes.
Q: If we configure a staging environment on Kubernetes, and publish all the map/feature services using referenced data, is there a way to clone this whole staging environment into a new production environment so that the services will be ready to use? The URL/SSL cert will be different between environments.
A: There are a few ways to migrate content between environments such as export/import functionality or the Python API to clone items. For services that use referenced data, that would generally involve republishing the services. You’ll need to consider how the data should be accessed, for example if you have a production database vs a staging database.
Content and Services
Q: Will image service functionality be equivalent to ArcGIS Image Dedicated and will it simply require an Image Server license?
A: In 10.9, you can federate a standalone ArcGIS Image Server site to your Enterprise on Kubernetes deployment to support image services. Note that raster analytics is not yet supported in ArcGIS Enterprise on Kubernetes.
Q: What databases are supported as of right now, and can be used to publish a service?
A: We currently support PostgreSQL and Microsoft SQL Server databases as registered enterprise databases. You can publish services with source data in these two databases. You can register connection files for these two supported databases and publish your services. We do not support Oracle as a registered enterprise geodatabase.
Q: When are you planning on supporting Oracle as a registered enterprise database?
A: You can federate a standalone ArcGIS Server to your Enterprise on Kubernetes portal. The federated server will support registering Oracle enterprise databases and publishing services with data. However, our challenge with supporting Oracle data sources directly within the Kubernetes environment is that our container images are Ubuntu-based and Oracle drivers are not officially supported on Ubuntu. Please reach out to your Oracle account manager and raise this with them while we continue to consider options from our side.
Q: Can vector tiles in ArcGIS on Kubernetes be configured in ArcGIS Pro to use existing caches? If so, and the requested vector tile does not exist in the cache, will the service fetch and generate it?
A: The option to bind to an existing cache is not available for vector tile layers. This feature is only available for cached map image layers. See this documentation. You can use the option to publish vector tile layers to an existing cache using ArcGIS API for Python. See this article for more information. Vector caches cannot be created on demand by ArcGIS Enterprise. You can rebuild vector caches if they are built using ArcGIS Pro 2.8, ArcGIS Enterprise 10.9. See more information here.
Q: Are we able to switch service cache folder when the new cache is created from a different server site, and copied to the production server cache folder with a staging folder name? On Windows, we simply swap the folder name for a service, to update the cache files.
A: At ArcGIS Enterprise 10.9 on Kubernetes, you cannot edit the cache directory used for a cached map image layer. We are working on providing an experience like the one you have when using ArcGIS Server Manager. However, if you are using a cache data store in cloud, you can simply swap the folder name for a service, to update the cache files.
Q: Will cache files created from a GIS Server on Windows work for cached service hosted in Kubernetes? I’m assuming we are using a cloud cache store.
A: That is correct. The cache created via ArcGIS Enterprise on Windows, Linux and Kubernetes can be used in either environment if using cache stores in the cloud. More information is available here related to best practices for tile caches.
Article Discussion: