Today, organizations rely on ArcGIS systems to deliver critical capabilities across their organizations. Therefore, designing for performance, reliability, and responsive user experience in a cost-effective manner is critical. Thankfully, the ArcGIS Architecture Center has a wealth of actionable information to help you achieve these goals.
Within the Architecture Library, you’ll find a growing category called test studies. You can use these resources to design, refine, and optimize your ArcGIS systems to meet your organization’s needs so you can deploy with greater confidence.
Types of test studies
A test study is a structured evaluation of a system’s performance and user experience, determining whether it satisfies requirements and performs well under real-world conditions. Test studies are created to help organizations build well-architected systems by offering practical examples of common tools used, methods employed, and lessons learned. Right now, there are two broad types of test studies:
1. System test studies
These are comprehensive evaluations based on a reference architecture. They provide:
- The system’s foundational workflows, capabilities, and components.
- The physical architecture used.
- Description of key design considerations and practices for systems of this type.
- A testing methodology and test results showing system utilization and user experience under different workloads.
- Lessons learned you can apply to ArcGIS systems of this type.
Today, there are four system test studies available, and additional test studies are in the works!
- Network Information Management System: Gas Utility (SQL Server)
- Network Information Management System: Gas Utility (SAP HANA)
- Network Information Management System: Electric Utility (Oracle)
- Network Information Management System: Electric Utility (PostgreSQL)
2. Configuration test studies
These assess how certain configurations of a system’s components, like instance sizes or application settings, can impact overall performance and user experience. They:
- Often build on a design used previously in a system test study.
- Describe the configuration(s) tested and the methodology used.
- Control other variables to ensure meaningful results.
- Assess whether a certain configuration is more optimal than another.
- Typically have applications across many systems well beyond the system being tested.
Today, there are four configuration test studies available. New configuration studies will be available later this year:
- Network Information Management System: Performance and Usability Comparison of SAP HANA deployed with SAP Private Cloud Edition
- Evaluating GPU and CPU Impact On Desktop Editing Workflows
- Evaluating impact of adding mobile capabilities to a foundational network information management system
- Evaluating the impact of visibility range on system performance and user experience
How should I use test studies?
Evaluate your current practices:
1. Assess the way staff participate in the system today: Do they perform their work in a well-defined, consistent way in alignment with best practices?
➡️ Use the provided steps as a starting point to document your organization’s workflows.
2. Review the testing tools and methods: How are your system changes evaluated and implemented today?
➡️ Identify gaps in your organization’s testing tools and practices needed to validate that your technical architecture meets service-level, functional, and nonfunctional requirements.
3. Evaluate your observability practices: How do you monitor your system today to adjust when thresholds are exceeded?
➡️ Review your observability requirements. Document the telemetry you capture, and how you capture it. Consider whether it is sufficient to support and inform operational activities.
Identify improvement opportunities:
4. Compare your design to the one tested: How is your system designed differently?
➡️ Evaluate whether the discrepancy could be causing degraded performance or reliability that your users need.
5. Look for areas to optimize: Where might you improve your system design to balance performance with cost?
➡️ Weigh infrastructure costs against operational costs and start by reviewing whether there are lower-cost opportunities to tune your apps and services.
Justify investment and prepare for change:
6. Anticipate the impact: Does your system’s behavior and user experience indicate a needed change in system design or configuration?
➡️ Review the shared testing strategy as a starting point to scope and vet changes in a way that protects your production environment.
7. Provide justification: Have you been struggling to justify further investment in ArcGIS systems?
➡️ Document potential negative consequences and use the quantitative evidence to support the need for system changes or investment.
💬 How are you using test studies in your work today?
Article Discussion: