Performance Test Plan Template

The following can be used as a performance test plan template:

Summary

Summary contains information such as the following:

• Which project to be tested
• Architectural diagram
• Test strategy
• Timeline
• Versions
• Deliverables (e.g., performance test report, application capacity analysis, SLA analysis, etc.)

Use Cases

• Which business transactions will be tested
• X% transaction A, Y% transaction B, etc.
• Data parameterization for each transaction

Performance Requirements

• Response time SLA needed if any.  E.g., 95% of transactions must complete within 100ms and 100% within 5 seconds.
• Transaction rate targets if any.  E.g., peak transaction rate of 1,000 requests per second must be supported.

Test Scripts

• Details on test scripts covering use cases, XML, JSON, etc.
• Details on data to be used for parameterization

Environment

• Which servers to use
• Configuration to use
• Any spoofing, stubbing, etc. to be used.

Test Tool

• Whether LoadRunner, Jmeter, Grinder, an internally developed tool, etc.  
• Any issues or details relevant to the tool.

Monitoring

• Metrics to be collected
• Response times
• Entry point response times
• Downstream call response times
• Transaction rates
• Server resource usage
• CPU
• Memory
• Network
• Disk
• Tools to be used to collect the metrics.

Scalability Testing

This section contains details on scalability testing to be done.

Load will be driven up in increments to the point at which peak capacity is reached.  A bottleneck analysis will be done to attempt to determine the factor limiting capacity to this point, whether some server resource limitation, some downstream call to a database or service, some internal thread blocking, etc.  This will provide information such as the following:

Discussion of whether any horizontal scalability testing will be done.

Scalability testing will allow the following to be determined:

• Bottlenecks
• Capacity
• Response Time SLAs

Stability Testing

This section contains details on stability tests to be run.

To verify stability, a high load should be run against the application for an extended period of time, at a minimum 24 hours and ideally for days or weeks.  A high load can be determined from the results of the scalability test, just below peak capacity, just below the point at which response time takes a turn for the worse. 

During the run, server resource usage should be captured and monitored, and error logs monitored.  Trends should be monitored closely:

• Does response time degrade over time?  That indicates a resource leak. 
• Does CPU usage increase over time?  That indicates a design or implementation error.  
• Does memory leak?  
• Do errors begin to occur at some point or occur in some pattern?  
• Does the application eventually crash?

Stability testing should also include verification of fault tolerance, which could include the following:   

• Bringing a downstream system down under load (stopping a downstream database or downstream webservice)
• Slowing down a downstream service under load.
• Applying a sudden heavy burst of traffic under load.
• Triggering error scenarios under load.
• Dropping network connections under load.  (using a tool such as tcpview)
• Bouncing the application under load.
• Failing over to another server under load.
• Imparing the network (reducing bandwidth, dropping packets, etc.)

The behavior of the application is observed in each test:

• Does the application recover automatically?  
• Does it crash?  
• Does it cause a cascading effect, affecting other systems? 
• Does it enter into a degraded state and never recover?  
• Can the event be monitored and alerted on with available tools?  
• Are appropriate events logged?

Performance Baseline/Maintenance/Regression Testing

If relevant to the project, the performance test can be used at a fixed load as a standard benchmark test to be run against each build or version of the application.  This will establish trends over time of each metric and will help verify that performance does not degrade over time or regress in a particular version.

How to Performance Test in a Service-Oriented Architecture

How do you performance test, stress test, and load test in the world of service-oriented architecture (SOA)?

The answer is that you test it at different levels of granularity, typically three levels.  One, obvious level is the service.  Another level is end to end.  The third level is the module level, the low-level building blocks making up the service.

One necessary precondition to adequate performance testing services is proper instrumentation providing key performance metrics.  Response times, transaction rates, and and success/failure counts must be available for all service entry points and downstream calls.  This allows response time contributions to be allocated accurately to the proper services and allows fast performance problem debugging.

The three layers of SOA performance testing share the following in common.

• Scalability Testing
• What is capacity?
• What bottleneck is limiting capacity?
• What is response time at various loads?
• What is the canonical performance chart?

• Stability Testing
• Is the application stable?
• Is the application fault tolerant?

• Performance Regression Testing
• Does performance degrade from build to build?
• Does server resource usage increase from build to build?

• Metrics on server resource usage
• CPU
• Memory
• Network
• Disk

Module level SOA performance testing is done as follows:

• Test key functional code paths at module level
• Use multi-threaded, concurrent execution
• Run within unit test framework
• Run within continuous integration framework
• Run frequently, each check-in, build, or version

Service level SOA performance test is done as follows:

• Test through public entry point
• Isolate service under test from other services
• Use spoofing or stubbing of backend services, mimicking their response time behavior
• Determine response time and availability service level agreements (SLAs) based on test results
• Thoroughly test the clustering or load balancing mechanism used to scale the service out horizontally.

End to end level SOA performance test is done as follows:

• Test public entry point into the application
• Verify that bottlenecks hit are consistent with capacity of individual services discovered in service-level testing.
• Verify fault tolerance of unavailable downstream services

An additional layer is infrastructure testing.  This could include messaging infrastructure, caching infrastructure, storage infrastructure, database infrastructure, etc.  Key infrastructure should be directly tested for scalability and stability in some cases to ensure that it behaves as expected and scales as expected.

SOA performance testing can be summarized in the following conceptual chart:

How to Use Regular Expressions in Loadrunner Using Gnu Regex for Windows Library

The following describes how to use the Gnu Regex for Windows regular expression library with loadrunner.  The following summarizes the steps:

• Download gnuwin32 Regex for Windows at  http://gnuwin32.sourceforge.net/packages/regex.htm
• From gnuwin32 zip, copy regex.h to <loadrunner_install_directory>/include
• In regex.h, comment out or remove the line  "#include <sys/types.h>" 
• From gnuwin32 zip, copy regex2.dll to <loadrunner_install_directory>/bin
• In the loadrunner script, load the regex2.dll
• In the loadrunner script, compile the regular expression pattern
• Call the regexec method with a string and the regular expression pattern
• Check the return code for match or no-match

Simplest Example

/* 
 * Example of how to use gnuwin32 Regex with loadrunner
 * Download gnuwin32 Regex for Windows 2.7
 *   http://gnuwin32.sourceforge.net/packages/regex.htm
 * Comment out line "#include <sys/types.h>" from gnuwin32 regex.h
 * Put gnuwin32 regex.h in <loadrunner_install_directory>/include
 * Put gnuwin32 regex2.dll in <loadrunner_install_directory>/bin
 */
#include "regex.h" 


// preprocessor macros giving names to regex response codes
#define MATCH 0
#define NOMATCH 1
#define ERROR   2


int dll_loaded = 0;


Action()
{
  char *test_string = "test";
  char *pattern = ".est";
  regex_t pattern_buffer;
  int status;


  // load gnuwin32 regular expression dll if not already loaded
  if (!dll_loaded) 
  {
    lr_load_dll("regex2.dll"); 
  }


  // compile the regular expression pattern
  if (regcomp(&pattern_buffer, pattern, REG_EXTENDED)) // REG_EXTENDED = POSIX Extended Regular Expression syntax
  {
    lr_error_message("Failed to parse regex pattern: %s", pattern);
    return ERROR;
  }


  // see if the pattern matches the input string
  status = regexec(&pattern_buffer, test_string, 0, NULL, 0);  // For meaning of last 3 params, see api doc: http://gnuwin32.sourceforge.net/packages/regex.htm


  // log the results
  lr_log_message("%s string \"%s\" to regex pattern \"%s\"",
  status==MATCH ? "Matched" : "Failed to match",
  test_string,
  pattern);
}

Console Output

Running Vuser...
Starting iteration 1.
Starting action Action.
Matched string "test" to regex pattern ".est"
Ending action Action.
Ending iteration 1.
Ending Vuser...

Test Script
The following example runs through a list of tests of various regular expression patterns used with both match expected and no-match expected test cases.


/* 

 * Example of how to use gnuwin32 Regex with loadrunner
 * Download gnuwin32 Regex for Windows 2.7
 *   http://gnuwin32.sourceforge.net/packages/regex.htm
 * Comment out line "#include <sys/types.h>" from gnuwin32 regex.h
 * Put gnuwin32 regex.h in <loadrunner_install_directory>/include
 * Put gnuwin32 regex2.dll in <loadrunner_install_directory>/bin
 */
#include "regex.h" 


// preprocessor macros giving names to regex response codes
#define MATCH 0
#define NOMATCH 1
#define ERROR   2


int dll_loaded = 0;


Action()
{
  regex_t pattern_buffer;


  // load gnuwin32 regular expression dll if not already loaded
  if (!dll_loaded) 
  {
 lr_load_dll("regex2.dll"); 
  }


  // Run tests  expected to match
  RunTest("test", ".est", MATCH);  // regex any char operator
  RunTest("test", "a*test", MATCH); // regex zero or more operator
  RunTest("test", "t+est", MATCH); // regex one or more operator
  RunTest("tttest", "t{3}est", MATCH); // regex interval operator
  RunTest("test", "t(ES|es)t", MATCH); // regex alternation operator
  RunTest("test", "t[^abcdfghijklmnopqruvwxyz]*t", MATCH); // regex list operator, with negation and repetition
  RunTest("test", "[t-z]est", MATCH);  // regex range operator
  RunTest("test", "(([u-z]est)|([t-z]es[t-z]))", MATCH);  // regex grouping operator
  RunTest("tests", "^te.*", MATCH);  // regex beginning of line operator
  RunTest("tests", ".*s$", MATCH);  // regex end of line operator
// 
  // Run tests  expected to not match
  RunTest("test", ".test", NOMATCH);  // regex any char operator
  RunTest("est", "t+est", NOMATCH); // regex one or more operator
  RunTest("test", "t(ES|Es)t", NOMATCH); // regex alternation operator
  RunTest("test", "t[^abcdefghijklmnopqrsuvwxyz]*t", NOMATCH); // regex list operator, with negation and repetition
  RunTest("tes2t", "tes[:alpha:]*t", NOMATCH); // regex character class operator: alpha   // 
  RunTest("testA", "test[:digit:]", NOMATCH); // regex character class operator: digits
  RunTest("test", "[u-z]est", NOMATCH);  // regex range operator
  RunTest("test", "(([u-z]est)|([a-s]es[t-z]))", NOMATCH);  // regex grouping operator
  RunTest("tests", "^es.*", NOMATCH);  // regex beginning of line operator
  RunTest("tests", ".*t$", NOMATCH);  // regex end of line operator


  return 0;
}


/*
 *  RunTest
 *  input: test string to match to regex pattern
 *  input: regex pattern
 *  input: expected result: match or no matc
 */
void RunTest(char *test_string, char *pattern, int expectedResult)
{
int status = TestMatch(test_string, pattern);


// Report whether or not test passed
lr_log_message("%s %s string \"%s\" to regex pattern \"%s\"",
  expectedResult==status ? "TEST SUCCESS: As expected," : "TEST FAILURE: Not expected,",
  status==MATCH ? "matched" : "failed to match",
  test_string,
  pattern);
}


/*
 *  TestMatch
 *  input: test string to match to regex pattern
 *  input: regex pattern
 *  output results of match attempt
 */
int TestMatch(char *test_string, char *pattern)
{
regex_t pattern_buffer;
int status;


if (regcomp(&pattern_buffer, pattern, REG_EXTENDED)) // REG_EXTENDED = POSIX Extended Regular Expression syntax
{
lr_error_message("Failed to parse regex pattern: %s", pattern);
return ERROR;
}


    status = regexec(&pattern_buffer, test_string, 0, NULL, 0);  // For meaning of last 3 params, see api doc: http://gnuwin32.sourceforge.net/packages/regex.htm
return status;
}

Console Output


Running Vuser...
Starting iteration 1.


Starting action Action.
TEST SUCCESS: As expected, matched string "test" to regex pattern ".est"
TEST SUCCESS: As expected, matched string "test" to regex pattern "a*test"
TEST SUCCESS: As expected, matched string "test" to regex pattern "t+est"
TEST SUCCESS: As expected, matched string "tttest" to regex pattern "t{3}est"
TEST SUCCESS: As expected, matched string "test" to regex pattern "t(ES|es)t"
TEST SUCCESS: As expected, matched string "test" to regex pattern "t[^abcdfghijklmnopqruvwxyz]*t"
TEST SUCCESS: As expected, matched string "test" to regex pattern "[t-z]est"
TEST SUCCESS: As expected, matched string "test" to regex pattern "(([u-z]est)|([t-z]es[t-z]))"
TEST SUCCESS: As expected, matched string "tests" to regex pattern "^te.*"
TEST SUCCESS: As expected, matched string "tests" to regex pattern ".*s$"
TEST SUCCESS: As expected, failed to match string "test" to regex pattern ".test"
TEST SUCCESS: As expected, failed to match string "est" to regex pattern "t+est"
TEST SUCCESS: As expected, failed to match string "test" to regex pattern "t(ES|Es)t"
TEST SUCCESS: As expected, failed to match string "test" to regex pattern "t[^abcdefghijklmnopqrsuvwxyz]*t"
TEST SUCCESS: As expected, failed to match string "tes2t" to regex pattern "tes[:alpha:]*t"
TEST SUCCESS: As expected, failed to match string "testA" to regex pattern "test[:digit:]"
TEST SUCCESS: As expected, failed to match string "test" to regex pattern "[u-z]est"
TEST SUCCESS: As expected, failed to match string "test" to regex pattern "(([u-z]est)|([a-s]es[t-z]))"
TEST SUCCESS: As expected, failed to match string "tests" to regex pattern "^es.*"
TEST SUCCESS: As expected, failed to match string "tests" to regex pattern ".*t$"
Ending action Action.
Ending iteration 1.
Ending Vuser...

Loadrunner How To Read a File into a Parameter

Here is a method for reading the contents of a file into a loadrunner parameter.  This can be useful for various situations such as reading XML or JSON requests from file, reading data, etc.

Script



//#include <stdio.h>
#define SEEK_SET 0 /* beginning of file. */
#define SEEK_CUR 1 /* current position. */
#define SEEK_END 2   /* end of file */


Action()
{
long infile; // file pointer
char *buffer; // buffer to read file contents into
char *filename = "test.txt"; // file to read
int fileLen; // file size
int bytesRead; // bytes read from file
// 
// open the file
infile = fopen(filename, "rb");
if (!infile) {
  lr_error_message("Unable to open file %s", filename);
  return;
}


// get the file length
fseek(infile, 0, SEEK_END);
fileLen=ftell(infile);
fseek(infile, 0, SEEK_SET);
lr_log_message("File length is: %9d bytes.", fileLen);


// Allocate memory for buffer to read file
buffer=(char *)malloc(fileLen+1);
if (!buffer) {
  lr_error_message("Could not malloc %10d bytes", fileLen+1);
  fclose(infile);
  return;
}


// Read file contents into buffer
bytesRead = fread(buffer, 1, fileLen, infile);
if (bytesRead != fileLen) 
{
  lr_error_message("File length is %10d bytes but only read %10d bytes", fileLen, bytesRead);
}
else
{
  lr_log_message("Successfully read %9d bytes from file: ", bytesRead);
}
fclose(infile);


// Save the buffer to a loadrunner parameter
lr_save_var( buffer, bytesRead, 0, "fileDataParameter");
free(buffer);
lr_log_message("File contents: %s", lr_eval_string("{fileDataParameter}"));
}




Step by Step Walk Through
Constants needed for fseek calls, also available in stdio.h:

#define SEEK_SET 0 /* beginning of file. */
#define SEEK_CUR 1 /* current position. */
#define SEEK_END 2   /* end of file */


Open file:

infile = fopen(filename, "rb");
if (!infile) {
lr_error_message("Unable to open file %s", filename);
return;
}


Get file length:

        fseek(infile, 0, SEEK_END);
fileLen=ftell(infile);
fseek(infile, 0, SEEK_SET);
lr_log_message("File length is: %9d bytes.", fileLen);


Allocate memory to read file into:

buffer=(char *)malloc(fileLen+1);
if (!buffer) {
lr_error_message("Could not malloc %10d bytes", fileLen+1);
fclose(infile);
return;
}


Read file into memory:

bytesRead = fread(buffer, 1, fileLen, infile);
if (bytesRead != fileLen) 
{
lr_error_message("File length is %10d bytes but only read %10d bytes", fileLen, bytesRead);
}
else
{
lr_log_message("Successfully read %9d bytes from file: ", bytesRead);
}

Close file handle:
fclose(infile);


Save the file to a loadrunner parameter:

lr_save_var( buffer, bytesRead, 0, "fileDataParameter");
free(buffer);
        lr_log_message("File contents: %s", lr_eval_string("{fileDataParameter}"));


File
The following is an example input file "test.txt" which would be located in the script directory (<loadrunner installation directory>/scripts/<scriptname>/test.txt:


First line of file.
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><Node>Data</Node></soap:Body></soap:Envelope>
Last line of file

Console Output

Running Vuser...
Starting iteration 1.
Starting action Action.
File length is:       168 bytes.
Successfully read       168 bytes from file: 
File contents: First line of file.
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><Node>Data</Node></soap:Body></soap:Envelope>
Last line of file
Ending action Action.
Ending iteration 1.
Ending Vuser...