At Points.com, we take our continuous integration servers pretty seriously. We have a large number of unit, integration, and web tests that run on TeamCity whenever one of our developers pushes a change to our code repository. This allows us to be alerted of regressions in our software and provides global references to successful or failed builds for anyone to see. Historically, we have always run our web tests on Firefox using Selenium.

What about Internet Explorer?

Many large corporations use one browser across all of their employees. They might be using Internet Explorer 6, Firefox 3.6, or some other major browser such as Safari or Chrome. This in turn means that a company providing software to such a corporation will have to support not only multiple browsers, but ancient browser versions.

There are a number of sane ways of dealing with this. Every software company that provides support for a web-based solution has to make a choice about which browser versions to officially support. While this reduces the number of difficulties, it does not fix the problem.

Regression Testing

Every software release, we found ourselves – guided by our heroic QAs – having to perform regression testing on each of the supported browsers. Any similar gaps in automated testing cause a longer and longer release cycle, decreasing the agility of our company as a whole. I can personally attest to the task of manual cross-browser testing being a royal pain.

To the Rescue!

Enter our co-ops. Michael Gershunovsky (https://github.com/shellderp, mgershun [at] uwaterloo.ca) and Calvin Ng (ckwng [at] uwaterloo.ca) came on board with Points.com from the University of Waterloo in September during their Fall work term. Their ultimate goal has been to reduce our release cycles through more powerful automated testing. Not only have they become experts on automated testing within our suite of tools; they have also created a large amount of documentation to aid us in the future, as well as reduce the amount of time we have to spend regression testing during each release cycle.

 

Selenium Grid – An Extensible Solution

The solution can be seen below. Selenium Grid is used as a framework to establish a set of continuous integration agents, a set of nodes upon which to test, and a hub to act as a broker between the two. In practice, the nodes are virtual machines which contact the hub to let it know what they can do and that they are available. Each VM defines a unique combination of an operating system and a browser – for example, Windows Vista and IE8. TeamCity then contacts the hub for information about the resources to which it can delegate.

  1. Node registers itself with hub declaring capabilities
  2. Agent sends session request asking for capabilities
  3. Hub looks for free node with desired capabilities and starts session on it
  4. Hub tells Agent that Node was found
  5. Agent starts sending test commands to Hub
  6. Hub forwards commands to Node
  7. Node browser hits agent servers as necessary for command
  8. Node returns results to Hub
  9. Hub forwards results to Node

This solution is also very much extensible. We can always add another VM that takes a new browser/OS combination that we did not previously support. The hub will take that information and allow the TeamCity agents to use it as they please!

One interesting note is that this allowed us to benchmark the differences between IE and the other browsers, with both Selenium 1 (RC) & 2 (Web Driver). Google Chrome using Selenium RC took, on average, about 3 ½ minutes to complete. Internet Explorer 9, in comparison, took more than 46 minutes. Internet Explorer 9 on Web Driver took only 4 ½ minutes, indicating the order of magnitude of improvement in the push from RC – which uses JavaScript – to Web Driver, which uses browser-specific commands to control the browser. Google Chrome and Firefox both remained very fast at approximately 3 minutes each.

Michael and Calvin have devised a system that we can use for years to come. A huge thanks to our co-op students for this achievement!

Shares 0