ArcGIS Hub is a cloud-based engagement platform that helps organizations communicate more effectively with their communities. Hub is widely used by customers of all sizes and across various industries, from local governments using Hub as a community event planner to school districts publishing demographic information on student enrollment. Thousands of Hub sites are built and shared by our customers worldwide, as you can find in ArcGIS Hub Gallery.
Hub is committed to providing strong security guarantees to its customers. As a service provider supporting US federal agencies, Hub participates in the Federal Risk and Authorization Management Program (FedRAMP), a program to provide standardized approaches on security assessment and authorization for cloud services. From development through operation, FedRAMP enforces a series of security controls across the full lifecycle of Hub products and requires continuous monitoring of security metrics on key artifacts.
As essential artifacts in modern software delivery, Docker images are a key focus of FedRAMP security controls. In preparation for the FedRAMP audit, the Hub DevOps team reviewed the security risks in Hub’s earlier Docker image management practices and implemented systematic approaches to mitigate them. In this post, we share behind-the-scenes insights from Hub’s journey toward FedRAMP authorization and offer practical guideline for teams looking to strengthen their Docker image management processes.
Docker Image Security
FedRAMP specifies multiple security controls for Docker images, including the following requirements:
- Images must be hardened to remove unnecessary files and permissions.
- Images must be free of critical Common Vulnerabilities and Exposures (CVEs).
- Images must be continuously scanned, and security findings should be addressed within a fixed timeframe.
- Images must be compliant with Federal Information Processing Standards (FIPS) for cryptographic safety.
These requirements imposed significant challenges on our original Docker image management model.
The Hub backend system consists of multiple API services that are developed using a micro-service architecture and are maintained by different product teams. These services are packaged as Docker images, pushed to AWS ECR, and deployed to AWS EKS clusters to serve customer requests.
Previously, Docker images were maintained by individual product teams that owned each service. The fully distributed ownership created a few problems under FedRAMP:
- Although Hub backend services were universally developed in Node.js or Python, teams selected different base images when creating their Dockerfiles and rarely updated them. This disparity resulted in multiple versions of runtimes and different operating systems running in production.
- There was no consistent build process across the organization. The lack of scan-in-build made it hard to establish early detection for vulnerabilities.
- Product teams were not security specialists, and necessary hardening was usually missed in the initial docker image design.
- It may increase response time for patching critical vulnerabilities across all services as the Hub DevOps team is the only team dedicated to infrastructure security.
These problems became a major concern during our first FedRAMP audit and pushed us to reshape the development of docker images in Hub.
Approaching FedRAMP Readiness
FedRAMP-ready Base Image
Hub API services are developed with Node.js. A common containerization practice is to build services on top of official community base images from DockerHub. While this approach is convenient, the security posture of community base images is not guaranteed:
- Community images may include vulnerabilities and are not suitable to be used directly in a FedRAMP-regulated environment.
- When a vulnerability is discovered, the maintainer is not obligated to address it within the FedRAMP-required timeframe.
- Unlike vulnerabilities, other compliance requirements, such as FIPS, are generally not a concern for image maintainers.
Rather than addressing these issues in each individual service image, the Hub DevOps team centralized the docker image build process and provided FedRAMP-ready base images on top of community base images with extra features:
- Build with lightweight operating systems, such as Alpine Linux and slim Debian Linux.
- Install FIPS-compliant cryptographic modules.
- Patch system dependencies nightly to automatically incorporate latest updates.
The new base images are more secure than their community counterparts and remain fully within Hubʼs FedRAMP authorization boundary. As a comparison, Hub node.js base images generally carry no vulnerabilities while the upstream community base images are still problematic.
After migrating all services to use shared base images, the entire Hub backend now runs on a consistent environment with reduced security risk and quicker vulnerability response time.
Build with Best Practices
When collaborating with product teams to update service Dockerfiles with new base image, the Hub DevOps team introduced specific build-time security practices to improve image quality and security. These best practices include:
- Use multistage builds to isolate build environments and include only runtime dependencies in the final image.
- Install dependencies only from trusted official repositories.
- Consolidate RUN steps to reduce image layers.
- Remove unnecessary temporary files and build caches.
- Remove unnecessary system users and minimize the use of root user.
The size of hardened service images was reduced from hundreds of megabytes to sub-hundred megabytes. This simplification reduces the number of components in each service image, resulting in a clearer security posture and faster startup time. These practices are codified in a template Dockerfile for new services, allowing future services to start from a secure, minimal baseline.
Consistent Scanning
Even when backend services run on the same base image, they do not share an identical security posture. Runtime dependencies and service code cause each image to differ from the others. In order to provide a consistent security evaluation of each image, the Hub DevOps team implemented a series of scanning at different stages of the image lifecycle:
- Development
- Detect and patch vulnerable dependencies using GitHub Dependabot, with automated test validation and controlled auto-merge.
- Pre-Build (CI)
- Lint Dockerfiles with hadolint to identify insecure instructions, anti-patterns, and syntax issues before image builds.
- Post-Build (CI)
- Scan image layers for malware using ClamAV with virus signature database updated daily.
- Post-Push (Registry)
- Scan operating system vulnerabilities using AWS ECR on-push image scanning.
- Perform deep dependency analysis using Esriʼs internal Black Duck platform
These scans are encapsulated in Hubʼs shared GitHub Actions workflows and are reused by the build process of all Hub services.
For pre-push scans executed in CI, detected findings cause the build workflow to fail with a notification sent to the change committer. Since the image has not yet entered the release process, Hub engineers retain flexibility to remediate or mitigate the issue before deployment.
For post-push scans performed on registry, findings are shared with the Esri Product Security team for further evaluation. At this stage, images may already be running in production, requiring Hub engineers to take immediate remediation action in accordance with FedRAMP-required timelines.
Timely Vulnerability Resolution
Logs from online scans in AWS ECR and BlackDuck Platform are streamed to the analytics systems of Esri Product Security team. A finding will raise an internal alert and security professionals will follow up to assess its severity level: low, medium, high, and critical. Severity levels are assessed using the Common Vulnerability Scoring System (CVSS) with consideration of vulnerability exposure and ease of exploitation.
Hub is obligated to provide timely resolution for vulnerability based on its severity. The more severe a vulnerability is, the sooner Hub engineers are required to address it:
- Critical: immediate
- High: 15 days
- Medium: 30 days
- Low: 90 days
With the use of universal base images, vulnerabilities in the base operating system and its packages can be addressed in the base image. And patches can be distributed through the standard service development and release process. For vulnerabilities at runtime dependencies, we work closely with the product development team to discover the root cause and address it swiftly.
Managed Image Rotation
Every year, Hub DevOps team runs an annual review of all Docker images running in production to identify deprecated or to-be-deprecated dependencies. This review is particularly important for Hubʼs Python and Node.js services, as their runtimes follow an annual deprecation process and stop providing security support for unsupported versions. If a service is using deprecating dependencies, we will replace its legacy base image with a new image using an active version. This practice ensures Hub services are always using a dependency version with continuous security patching.
Conclusion
Since the initial FedRAMP audit in 2023, the Hub DevOps team has implemented multiple strategies to transform Docker image management for elevated security requirements. We evolved from managing Docker images individually across product teams to a centralized approach featuring FedRAMP-ready base images and security-inspired automation. This systematic approach helped mitigate dozens of security findings early in the audit and has kept Hub services resilient against new vulnerabilities in subsequent years. It ensures service images are developed with security controls applied from the start and ensures we can quickly respond to any emerging threats.
Security is a top priority for Hub engineering. Achieving FedRAMP authorization marks not an endpoint, but a new phase of continued investment in strengthening Hubʼs security implementation and raising the security baseline across the infrastructure. If this work sounds interesting to you, please check out our open positions at Esri Careers.
About the author
