What return on investment/ROI can be expected from performance testing? There are two categories of "return" for performance and stability testing.
One category of return is a simple, quantifiable return in which performance testing results allow the amount of hardware required to be reduced in a couple of ways:
- Application tuning and optimization allowing the same amount of hardware to handle additional traffic
- Proof of hardware overcapacity, showing that a reduced amount of hardware would be sufficient to handle peak traffic, allowing hardware inventory to be reduced
The second category of return is not a return so much as an insurance policy, or protection against a fat tail or black swan event. In this case, you are protecting yourself against system stability bugs whose consequences can be absolutely catastrophic, up to the complete destruction of the business.
Consider the example of Knight Capital's botched software rollout on August 1st, 2012. The rollout triggered unexpected trades that cost the firm $440 million, depleting its operating capital. This weakened position allowed outside investors to take a controlling 70% stake in the company as terms of its bailout.
By doing performance and stability testing, catastrophic stability bugs may be missed due to limitations of the test environment or test data, or failure to anticipate the nature of production behavior leading to the disaster. However, if a catastophic stability bug is discovered and fixed, which happens regularly, a black swan event has been quietly and successfully sidestepped. The value is avoiding a devastating black swan event that brings down the company through loss of customers, loss of reputation, lawsuits, takeovers, etc. Not to mention more individual losses such as loss of job, bonus, promotion, career, etc. The cost of the testing is fixed and predicable, simply the cost of supporting ongoing performance testing, in effect the insurance policy premiums protecting against the black swan.
No comments:
Post a Comment