Performance Test Team Blueprint

The following is a blueprint of how to set up an effective performance test team.

TEAM OBJECTIVE

• Verify all software is:
• scalable
• stable
• fault tolerant
• Resolve bottlenecks whenever encountered
• Automate wherever possible
• Use consistent processes, test cases, and techniques
• Provide good documentation

TEST METHODOLOGY
Test software performance at all levels of granularity:

• Design
• review architecture and design, verify that design will scale in theory
• Module
• Unit performance tests for critical modules, both white box and functional as needed
• Use mocking or stubbing to isolate module from external dependencies
• Run performance tests in continuous integration environment
• Test scalability, stability
• Service/Product
• Use stubbing or spoofing to isolate service
• Cover key functional use cases
• Test scalability, capacity, stability, fault tolerance
• End to end
• Black box performance tests
• Use consistent, stable benchmark on all builds/versions
• Trend metrics
• Look for performance regressions
• Test scalability, capacity, stability, fault tolerance
• Infrastructure
• Ensure infrastructure is scalable
• Ensure infrastructure is stable

TEST METRICS

• Ensure application has adequate monitoring tooling/hooks (for example, metrics on all entry points and downstream calls)
• Collect uniform and meaningful metrics during performance testing
• Ensure performance test metrics are comparable to production metrics

TOOLS

• UI/end to end load generator
• Service/web service load generator
• Unit test load generator
• Code/CPU profiler
• Memory profiler

PERSONNEL

• Hire software engineers in test with specialization in performance whenever possible.  Ensure hires can automate repetitive tasks and automate the performance testing process.