Extending the Load Test Plan

The previous post covered the minimal plan for load testing which included testing scalability and stability testing.  This post covers additional testing needed to ensure a stable, scalable, and well-performing application, specifically:

1. Performance regression
2. Fault tolerance
3. Horizontal scalability

1. Performance Regression

A performance regression test consists of running the same performance test on the prior version of the application and then on the current version of the application using identical or at least equivalent hardware.  This will show whether performance has degraded in the current release versus the previous release.  The test should be run under load to include the impact of any concurrency or other load-related issues affecting performance.  The load could be at various levels, i.e., running the vertical scalability test already discussed on each version of the application.  Or, if a single load is used for performance regression, the load should be selected at slightly less than peak capacity.

Ideally, such a performance regression test or standard performance benchmark would be run on a variety of versions of the application over time which will provide a performance trend.  This will show whether performance is slowly degrading or improving over time.

In addition to response time performance regressions, regressions should also be looked for in server resource usage.  You want to know if the application or client is burning more CPU to do the same amount of work, or whether more memory or network resource is needed.

If there is a performance regression, it should be investigated carefully and fixed if possible. 

2. Fault Tolerance

Fault tolerance testing involves running various negative or destructive tests while the application is under load.  These could include the following:   

• Bringing a downstream system down under load (stopping a downstream database or downstream webservice)
• Slowing down a downstream service under load.
• Applying a sudden heavy burst of traffic under load.
• Triggering error scenarios under load.
• Dropping network connections under load.  (using a tool such as tcpview)
• Bouncing the application under load.
• Failing over to another server under load.
• Imparing the network (reducing bandwidth, dropping packets, etc.)

The behavior of the application is observed in each test:

• Does the application recover automatically?  
• Does it crash?  
• Does it cause a cascading effect, affecting other systems? 
• Does it enter into a degraded state and never recover?  
• Can the event be monitored and alerted on with available tools?  
• Are appropriate events logged?

In each case, the behavior could be as designed, or it could be unexpected and be a scenario that must be fixed prior to production deployment.  This type of testing can find issues that would otherwise not be found in the test lab and can greatly improve the stability of the application.

3. Horizontal Scalability

Vertical scalability of the application on a single server has already been tested and bugs fixed allowing the application to scale up and use a majority the resources of a single server.  A horizontal scalability test addresses the question of whether adding additional servers running the application into the cluster allows the application to scale further.  Once one server is nearing peak capacity, can another server be added to the cluster to double capacity?  How far out does this scale?  Does one server double capacity, but no further gains can be obtained beyond that level?

The simplest way to test this is to literally expand the cluster one server at a time and test peak capacity with each addition.  However, at some point this may be unfeasible due to lack of test servers. In that case, one strategy might be to focus on individual downstream applications and verify that they can scale to the required level.  For example, a downstream database could be load tested directly to see how many queries per second can be supported.  This will provide a cap on the scalability of the upstream application.  The same thing can be done with a downstream web service.

Any capacity ceilings hit should be studied so that it is understood what is limiting further horizontal scalability, whether it is network bandwidth, capacity of a database, an architectural flaw in the application, etc.

The load balancing mechanism should also be examined carefully to make sure it does not become a bottleneck, particularly if it has not been used previously or if load is to be increased substantially.

Another possibility might be to deploy the application to the cloud and scale out using large numbers of virtual servers.