Tuesday, September 29, 2015

Load Testing with Selenium Grid

Selenium grid is an open source browser-based test tool that allows multiple concurrent browsers to run at the same time, spread across a number of nodes.  With each node running multiple browsers and many nodes in place, a significant number of concurrent browsers be pointed at a web site and substantial load generated.

Some Advantages

Selenium webdriver runs real browsers which execute javascript on the client-side and can give realistic response times and realistic user behavior.

Selenium integrates well with modern java stacks such as junit, testng, jenkins, gradle, ant, etc.

Cost.


Some Drawbacks

There may be bugs, such as limited compatibility with some versions of some browsers.

There may be maintenance overhead.

The amount of load that can be generated is limited.  Real browsers are expensive to run in terms of cpu and memory requirements, so a lot of client nodes can be required to generate enough load.  It is more appropriate for moderate load.  I'm not sure what the limits of the number of concurrent browsers this approach will support, but dozens of concurrent browsers can easily be supported.  It is possible that hundreds of concurrent browsers may be too much.  For very large tests, commercial tools and more efficient load generation approaches targeting lower-level traffic may be more suitable.


How

Set up a selenium grid hub and a number of selenium grid nodes registered to that hub.  This can be done in a cloud provider such as AWS or using on-premise hardware.
The desired versions of browsers to be tested should be installed on the nodes.
Nodes should have graphics cards that give responsive browser behavior.
Limit the max number of browsers per node to be no more than the number of CPUs per node.
Other than for node debugging, disable node logging which can be very verbose and fill up the file system.
In the java client test application, each test thread should get a thread local instance of Selenium WebDriver.  This will result in a thread-safe WebDriver, with browsers not interfering with one another.
Test the node infrastructure by, for example, running five browsers on one node versus running one browser on each of five nodes.
Once the grid infrastructure has been validated, the target website can then be tested for scalability by slowly ramping up the number of concurrent browsers until response time knee is hit.
Stability of the target website can be tested by running near peak load for an extended period of time.  Be sure to log results along the way in case of a client-side crash.
Browser snapshots can be saved when errors are encountered, making bug identification simple.