How to Test the Stability of an Application

Testing the stability of an application is critical.  It can prevent system outages by identifying problems before they occur in production.  Outages can severely damage a business, in some cases permanently.  The following outline provides a reasonable template for testing application stability.

• Ramp load up incrementally to the breaking point of the system.  Do not stop at expected peak load because bursts or unexpected traffic can entail load far higher than anticipated.
• Load should cover critical dimensions such as transaction rate/throughput, connections, concurrent users, range of use cases/functionality
• When the application breaks, investigate what broke
• If the test infrastructure broke (test client capacity hit, test network capacity hit, test case crashed, etc.), the test infrastructure must be repaired so that the application is what breaks, not the test infrastructure.
• If the application broke, diagnose the type of breakage and what broke.
• Is breakage recoverable?
• Does breakage affect already connected users, or just block new users?
• Did the application code break (errors, deadlocks, thread blocking, etc.)?
• Was a system resource limit hit (cpu, memory, network, disk)?
• If system resource limits were not hit, does the application need to be fixed so that it is not the bottleneck?  The system should scale up so that system limits are hit, whether CPU, network, disk I/O, or network bandwidth.
• Did a downstream service break?
• How can the downstream service be improved to provide more capacity and stability?
• Did the system just slow down, remaining functional?
• Is a restart required, and what must be restarted (services, server, downstream services, etc.)? 
• Can the system be scaled out or scaled up to improve the capacity?  
• If not, why not?  Is there an architectural limitation preventing further scalability?  How can scalability be improved?
• From the test determine the peak capacity of the application and verify that proper production monitoring is in place to detect this threshold.
• Run at near peak capacity for an extended period of time (this could be one day or more depending on uptime requirements)
• Is the application stable when run for a long time or does it eventually crash?
• Why does it crash?
• Does performance degrade over time?
• Why does it degrade?
• Perform administrative operations that may need to be performed during production usage while system is near peak load.
• Is the system stable when this happens?
• Perform the full suite of functional tests while the system is near peak load.
• Is the system stable when this happens?

Document the results of the test carefully.  Do not ignore crashes and instability.  Spend the time and effort to understand the behavior and harden the application to behave well under any conditions, anticipated or not.