Virtualization Performance Testing

Virtualization technologies (such as VMware, Parallels, Xen, KVM, Waratek, etc.) have become very popular for good reasons, including but not limited to the following:

• Application density can be increased
• Server resource usage can be maximized
• Hardware costs can be reduced many-fold
• The number of servers required to run the same load can be reduced
• Applications become more portable

It is critical to do performance, load, and stress testing of a virtualization solution before rolling it out for several reasons:

• Different applications behave differently in a virtualized environment and some are more suitable than others
• Performance or stability could be affected in a virtualized environment
• Virtual machine sizing can drastically affect application behavior
• Application configuration can be different when run in a virtualized environment.

This post gives a brief outline of how to do performance, load and stress testing of a virtualization solution.

1. Include online and batch test cases as well as administrative operations
2. Test bare metal (physical server) as a baseline to be compared to virtualization results
1. See how many instance of the application can be run on bare metal
2. Push bare metal up to server capacity
3. Test virtualization
1. Run VMs on the same bare metal server or an equivalent one
2. Push one VM up to capacity, then two VMs, then three VMs, etc., up to the highest density desired.
3. Capture hypervisor server resource usage metrics (cpu, memory, disk, network)
4. Verify that VM performance compares reasonably relative to bare metal
5. Verify that VM capacity compares reasonably relative to bare metal
6. Verify that VM application stability compares reasonably relative to bare metal
7. If performance or stability is not acceptable, see if application tuning is needed or virtual machine tuning is needed.
8. Include stability tests in which high load is applied for an extended period of time.
9. Perform administrative operations while system is under high load.
10. Once performance and stability are acceptable, publish the results and the tuning and configuration needed to achieve acceptable results.

Using Loadrunner Pacing to Hit a Specific Transaction Rate

It is often necessary to apply load at a specific transaction rate per second (TPS).  For example, it may be necessary to test every build or patch of an application by running a fixed benchmark of 20 queries per second against the application, measuring the response time and server utilization at that fixed load.  By then examining the trend over time of response time and server resource utilization under uniform load, performance regressions can be easily identified.

One way to set a specific transaction rate using Loadrunner is by using the pacing feature.

Pacing specified how often an iteration starts.  For example, if an iteration runs in 300 - 500 milliseconds, setting pacing to 1 second for that script will case a user to run the iteration once every second as illustrated below:

Each second the iteration starts and then ends after 400 ms or so.  At the next second interval, the next iteration starts.  An exact transaction rate of 1 iteration per second is reached in this way.  By increasing the user count in this case to 10, a transaction rate of 10 TPS can be achieved.  

An iteration can have multiple transactions per iteration.  If the above example was an iteration that included three transactions (query1, query2, and query3), a single user would run three transactions per second and 10 users would give 30 TPS.

The following formula provides a calculator for determining what the pacing in seconds should be set to given three input parameters, transactions per iteration, user count, and target transaction rate:

• transactionsPerIteration  (the number of transactions included in an iteration)
• users (the number of users to be run)
• tps (the target transaction rate per second)

For example, suppose transactionsPerIteration = 3, users = 50, and target transaction rate = 10 TPS.  The equation gives a pacing in seconds of (3 * 50)/10 = 1.5 seconds.

One thing that can prevent the target transaction rate from being reached is when the iteration slows down when run under multiple users such that the time an iteration runs is longer than the pacing time.  In this case a warning is generated and the transaction rate is not reached.  This can be sometimes be avoided by increasing the user count somewhat.

However, the problem could also be due to limitations in the application's vertical scalability.  The application may have concurrency problems preventing concurrent users from executing the transaction without blocking other users.  In a worst case, the application's concurrency problems may make it impossible to reach the target transaction rate regardless of the number of users.  In any case, it is worthwhile to investigate what the bottlenecks are that are limiting scalability and resolving those issues.

VisualCSV - A Free CSV Analyzer - CSV File to Chart, CSV File to Stats, CSV File to Histograms, CSV File Correlation Charts

VisualCSV         Download Now

VisualCSV is a free CSV analyzer that is:

• A Java application 
• Converts CSV File to Charts
• Converts CSV File to Statistics
• Converts CSV File to Histograms
• Converts CSV File to Correlation Charts

Performance testing, system monitoring, etc. often generate CSV files that must be analyzed, and typically you need to chart columns, calculate stats on columns, correlate columns, etc.

VisualCSV is a java application that, given a csv file input, will generate most of the data you need automatically, including:

• statistics on each column in the CSV file 
• min
• avg
• max
• standard deviation
• variance, median
• percentiles (such as 95th and 98th percentiles, or any list of percentiles you specify)

• trend charts for each column in jpeg format
• histogram charts for each column
• correlation metrics between any pairs of columns
• correlation charts showing correlations between any pairs of columns
• transposed correlation charts reversing x and y axis.
• charts created using jfreechart
• stats created using apache commons math

The following sections shos sample output generated from processing a csv file from windows performance monitor (perfmon) having the following columns:

(PDH-CSV 4.0) (Pacific Standard Time)(480)

\\CHELSTRPL006\Memory\Available Bytes

\\CHELSTRPL006\Network Interface(HP NC382i DP Multifunction Gigabit Server Adapter _4)\Bytes Total/sec

\\CHELSTRPL006\Network Interface(HP NC382i DP Multifunction Gigabit Server Adapter _4)\Output Queue Length

\\CHELSTRPL006\PhysicalDisk(_Total)\% Disk Time

\\CHELSTRPL006\PhysicalDisk(_Total)\Avg. Disk sec/Read

\\CHELSTRPL006\PhysicalDisk(_Total)\Avg. Disk sec/Write

\\CHELSTRPL006\Processor(_Total)\% Privileged Time

\\CHELSTRPL006\Processor(_Total)\% Processor Time

Summary stats

The following csv file showing summary stats for each column is generated.

Trend Charts

For each column, a chart is generated such as this:

Histograms

For each column, a frequency histogram showing number of occurrences by bucket is generated such as this:

The number of buckets is configurable.

For each chart, a relative frequency histogram  is generated showing relative frequency by bucket is generated such as this:

Correlations Between Columns

For each pair of columns, a csv file showing correlations between pairs of columns is generated such as this:

Correlation Charts

For each pair of columns, a correlation chart and a transposed correlation chart (x and y swapped) is generated such as this:

Download

VisualCSV can be downloaded here:

• https://docs.google.com/open?id=0B4gruS14dMq4N2M4ZDZjNWYtMTIzYy00NmRmLWJiNTYtOWVhYTc4NjJiMWU2

Usage

• VisualCSV requires java in the system path.  It has been tested with java 1.6 and probably runs fine with 1.5 and 1.7 as well.
• Download and unzip the files.
• Analyze the sample CSV file from command line by running:
• run.bat TestPerfmonFile.csv
• This will analyze the test file and generate output in directory .\out.  The output should 

Configuration

Client configuration is found in the file client.config.

Columns can be specifically included or excluded by putting them in specified include or exclude files which by default (as specified in client.config) are called 

Column-Include-Names.properties

Column-Exclude-Names.properties

The include file can be empty or missing in which case all columns are analyzed.  Otherwise, the columns analyzed are all columns in the include file minus those in the exclude file.

Likewise columns can be specifically included or excluded from being correlated with each other by putting them in specified include or exclude files which by default (as specified in client.config)  are called 

Correlation-Include-Names.properties

Correlation-Exclude-Names.properties

Again, the include file can be empty or missing in which case all columns are correlated. Otherwise, the columns in the exclude file are subtracted from the includes.

The client.config file includes the following values by default:

# --- Functionality
createTrendCharts=true
computeCorrelations=true
createCorrelationCharts=true


# --- whether to include or exclude columns from analysis
# --- put columns in this file to include them in analysis.  empty file or no file means analyze all columns.
columnIncludeFile=Column-Include-Names.properties


# --- put columns in this file to exclude them from analysis.
columnExcludeFile=Column-Exclude-Names.properties


# --- whether to include or exclude columns from correlation with other columns
correlationIncludeFile=Correlation-Include-Names.properties
correlationExcludeFile=Correlation-Exclude-Names.properties




# --- Timestamp column index (0 = first column)
hasTimestampColumn=true
timestampColumnIndex=0


# --- Java SimpleDateFormat date format for parsing timestamps
timestampFormat=MM/dd/yyyy HH:mm:ss.S

# Letter Date or Time Component Presentation Examples
# G Era designator Text AD
# y Year Year 1996; 99
# M Month in year Month July; Jul; 07
# w Week in year Number 27
# W Week in month Number 2
# D Day in year Number 189
# d Day in month Number 10
# F Day of week in month Number 2
# E Day in week Text Tuesday; Tue
# a Am/pm marker Text PM
# H Hour in day (0-23) Number 0
# k Hour in day (1-24) Number 24
# K Hour in am/pm (0-11) Number 0
# h Hour in am/pm (1-12) Number 12
# m Minute in hour Number 30
# s Second in minute Number 55
# S Millisecond Number 978
# z Time zone General time zone Pacific Standard Time; PST; GMT-08:00
# Z Time zone RFC 822 time zone -0800


# --- chart settings
chartEveryNthDataPoint=1
histogramLogScale=false


# --- statistics settings
createChartCorrelationGreaterThan=0.0
createChartCorrelationLessThan=-0.0
numHistogramBuckets=10
percentiles=80,90,95,98,99


# --- Output file locations
outDir=out
deleteStatsDirs=true
outTrendDir=out/Trend
outCorrelationDir=out/Correlation
outHistogramDir=out/Histogram
outputStatsFile=SummaryStats.csv

Monitoring Linux Servers during Performance Tests

The server resource usage of the servers used in a load or performance test should be monitored and saved to be able to later verify system bottlenecks and behavior.  Linux servers can be monitored in a number of different ways.

If you are using Performance Center to drive load, linux monitors can be enabled as part of the load test, allowing the linux metrics to be conveniently combined with other metrics in a single report.  That requires the rstat daemon to be running on the linux servers monitored.  For CentOS, rstat daemon can be installed as follows:

sudo yum install rpcbind
sudo wget http://www.wyzaerd.com/public/rpc.rstatd-4.0.1.tar.gz
sudo tar xvzf rpc.rstatd-4.0.1.tar.gz
cd rpc.rstatd-4.0.1
sudo ./configure
sudo make
sudo make install
sudo /usr/local/sbin/rpc.rstatd

Monitors can then be configured in performance center to collect stats from the linux server running rstatd.

Another tool for viewing and collecting server resource usage on linux is jperfmeter (http://jperfmeter.sourceforge.net/).  Jperfmeter provides a gui view of resource usage metrics of a list of servers, showing values over time.  For example the following shows the jmeter console for a group of 4 linux servers:

The jmeter metrics can be saved to a delimited text file for later analysis:

A third tool for monitoring linux server resource usage is sar. The following script will collect server resource usage on CPU, memory, network, and disk utilization:


# Run sar every 10 seconds until stopped
# cpu
sar -u 10  > sar.cpu.log &
# free memory
sar -r 10  > sar.freememory.log &
# disk total
sar -b 10  > sar.disk.log &
# network by device
#    - Note that you need to filter by the adaptor in use.  
#    - Run "sar -n DEV 10" to see which adaptor is being used
sar -n DEV 10 |grep eth1 > sar.network.log &

On some versions of sar, it is necessary to provide a number of data points to collect such as the following:

# Run sar every 10 seconds until stopped
# cpu
sar -u 10 99999 > sar.cpu.log &
# free memory
sar -r 10 99999 > sar.freememory.log &
# disk total
sar -b 10 99999 > sar.disk.log &
# network by device
#    - Note that you need to filter by the adaptor in use.  
#    - Run "sar -n DEV 10" to see which adaptor is being used
sar -n DEV 10 99999 |grep eth1 > sar.network.log &


For a test of one or two hours, a 10 second collection interval is convenient, providing a couple of hundred data points.  For an overnight test, a 300 second collection interval is usually appropriate.

Output

The cpu log file shows user and system CPU % utilization:

03:07:55 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle

03:08:55 PM     all     73.99      0.00      2.43      0.21      0.00     23.37

03:09:55 PM     all     81.79      0.00      2.67      0.21      0.00     15.34

03:10:55 PM     all     82.29      0.00      2.68      0.17      0.00     14.86

The free memory log file shows how much memory is free and used:

03:07:55 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit

03:08:55 PM 110106128  88246712     44.49    356468  42850352  30363972      7.61

03:09:55 PM 110053452  88299388     44.52    356472  42879192  30371420      7.61

03:10:55 PM 109989584  88363256     44.55    356484  42914152  30372688      7.61

The disk log file shows read and write transfers per second and bytes read and written per second

03:07:55 PM       tps      rtps      wtps   bread/s   bwrtn/s

03:08:55 PM   7889.59      0.00   7889.59      0.00  58582.09

03:09:55 PM   8454.59      0.00   8454.59      0.00  62458.76

03:10:55 PM   8456.30      0.00   8456.30      0.00  62645.15

03:11:55 PM   7257.61      0.00   7257.61      0.00  57384.76

The network log file shows packets received and transmitted per second and bytes received and transmitted per second.

03:00:01 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s

03:08:55 PM      eth1   3285.46   2965.12    956.75   1824.97      0.00      0.00      1.05

03:09:55 PM      eth1   3640.33   3307.06   1053.38   2074.92      0.00      0.00      1.14

03:10:55 PM      eth1   3617.67   3283.23   1047.62   2061.22      0.00      0.00      1.65

03:11:55 PM      eth1   2917.34   2657.74    842.35   1686.10      0.00      0.00      1.38

03:12:55 PM      eth1   3859.74   3502.98   1119.06   2194.43      0.00      0.00      1.15

These can be charted to provide a simple overview of server resource usage as follows:

A Minimal Load Test Plan

What is a minimal load/stress test plan for a new service or application?  A minimal plan covers two scenarios:

1. Scalability Test
2. Stability Test

1. Scalability

Vertical scalability is a measure of how effectively an application can handle increasing amounts of load on a single server.   Ideally, the application can handle increasing amounts of load without significant degradation in response time until reaching some server resource limit such as CPU limits or network adaptor bandwidth limits. The results of a scalability test can be presented in a chart such as the following:

The chart shows, for each of 8 tested load levels, the response time and the transaction rate of the application.  In this case each load level is a number of concurrent requests in increments of one.  In other cases other increments may be appropriate (such as increments of 10) and other measures of load may be appropriate (message size, etc.).  Load should be driven high enough that throughput levels off.  Response time will ideally remain flat as load increases, eventually turning a knee or corner and heading upwards as capacity is reached.  In this case, the application scales nearly perfectly up to 5 concurrent requests, then begins to degrade, with peak throughput around 4,500 queries per second.

The scalability chart provides a large amount of information that can be used for capacity planning, production configuration, etc.  It shows what response times are under typical loads.  It shows the throughput capacity of a single server running the application, and it shows the behavior as capacity is exceeded.

As part of a scalability test, metrics showing server resource usage at each load level should be captured, such as CPU usage, network usage, disk usage, and memory usage.  Logs and errors should be captured.  Similar information should be captured on any downstream systems involved in the test if any, such as databases or services.

Part of the test analysis should involve bottleneck analysis, which is analyzing and determining what is limiting the capacity of the application, what is limiting it to 4,500 queries per second.  This could be server resource usage (hitting CPU, network or disk limits), it could be an increase in response time of a downstream database or server as load increases, it could be contention within the application such as thread blocking or error paths hit at high loads, etc.

An appropriate environment for a scalability test would involve two servers, one for the application and one to act as the client driving the load:

Server resource usage on the load generator should be monitored as well to verify that it is not the bottleneck.

This type of vertical scalability test does not guarantee that the application will scale out horizontally for two reasons: (1) there may be downstream systems such as databases that become bottlenecks at higher loads, and (2) load balancing or clustering systems may not scale as expected.  More extended testing beyond a minimal plan would have to cover these factors as well.

For the vertical scalability test, it is important to drive the application up to peak capacity, regardless of what expected load may be.  Usage may be different than expected, spikes in load may occur, business might grow, etc.  Some performance or stability problems only manifest themselves at higher loads, and it is important to identify these even if production load is expected to be much lower.

2. Stability

The second part of the minimal load test plan is the stability test.  To verify stability, a high load should be run against the application for an extended period of time, at a minimum 24 hours and ideally for days or weeks.  A high load can be determined from the results of the scalability test, just below peak capacity, just below the point at which response time takes a turn for the worse.  In the example above, a load of 5 concurrent requests could be used, assuming those are the final results following resolution of performance bottlenecks.

During the run, server resource usage should be captured and monitored, and error logs monitored, as with the scalability test.  Trends should be monitored closely.  Does response time degrade over time?  That indicates a resource leak.  Does CPU usage increase over time?  That indicates a design or implementation error.  Does memory leak?  Do errors begin to occur at some point or occur in some pattern?  Does the application eventually crash?

Beyond the Minimal Plan

Beyond the minimal plan, other tests are required to ensure the application is completely performant and stable.  These will be covered later (http://sub-second.blogspot.com/2011/11/extending-load-test-plan.html) and include:

1. Performance regression
2. Horizontal scalability
3. Fault tolerance