These are notes on how to integrate Selenium tests into a Jenkins-based continuous integration setup. Since I had next to no experience with either project, there are several moving parts, and there are many different ways to achive this goal, I decided to write up my notes. Note also that until very recently, Jenkins was known as Hudson; in the interests of being forward-thinking I'm going to refer to it by its new name.

Preliminaries

At work we already had a CI server configured and running, so I'll skip the basic Jenkins configuration stuff here. But the upshot is, it was configured for a fairly standard procedure:

1. Poll our git repository for changes.
2. When a change happens, make a new virtualenv for the project, install the necessary packages, and run our Django tests.
3. Email the devs if something's busted.

We now want to add "run some Selenium test suites" somewhere in step 2, and "show me reports" somewhere in step 3. I'm going to assume that you can install the Selenium IDE Firefox plugin and author some simple tests, which is very easy: just open the plugin window, start recording, click around, and save the results somewhere. For getting-started purposes I made one test suite, consisting of two test cases: one to check that an unsuccessful login attempt keeps me on the same page and shows an error, and one to check that a successful login takes me to the page I expect. These are saved as "Selenese" HTML files for now. (How to write good test suites, and what specifically to test, seems like a topic in and of itself.)

Selenium server setup

First, a quick overview of the different bits of the Selenium project:

• Selenium IDE ==> A Firefox plugin used to record and playback tests. Note that you'll probably want to edit and tweak these tests after creating them with a regular text editor.
• Selenium RC ==> Two parts: one, a server that handles taking the above test commands and sending them to a client (ie, a browser) for execution; and two, language drivers that allow you to do that from different programmming languages (which will allow for greater flexibility for your tests)
• Seleniun Grid ==> Allows you to do what RC does above, except on multiple client machines at once.

So, we know we need to author tests (IDE) and we know we need to run them against a browser (RC). For ease of setup and because I don't have more than one machine to play with at the moment, I'm not going to set up Grid for now (although it definitely seems like a good idea, to cut down on the time it takes to get feedback on a build).

Additionally, we need to decide what machine to run the RC server on. I had a couple of choices:

1. A regular desktop machine (eg Windows, Mac or Linux box) with a GUI and browser(s), separate from whatever Jenkins is running on. Many people use a spare machine in their office for this purpose, but note that you'll then need to deal with telling Jenkins how to talk to this machine.
2. A faked-out headless Linux box. For convenience, on the same machine as Jenkins. The browser stuff takes a bit of setting up but it's easier for Jenkins to talk to since all paths are local, etc.

I chose the latter. Our CI server is actually an EC2 instance running Ubuntu, but Selenium needs an actual browser to do its stuff against and obviously we don't have (or want) that on a remote server. However, it's possible to run GUI apps headless in a virtual display. This was not difficult at all, thanks to these three posts. Summarizing those, first we need to install some packages:

sudo aptitude install xvfb          # virtual display
sudo aptitude install x11-apps      # installs xclock (to test things are working), and xwd (for taking screenshots)
sudo aptitude install imagemagick   # for converting screenshots
sudo aptitude install firefox

Then, we start the virtual display, launch the GUI app xclock (so we have something to look at), take a screenshot of it, and (since xwd uses its own image output format) convert that into a regular image file:

/usr/bin/Xvfb :5 -ac -screen 0 1024x768x8 &
xclock -display :5.0 &
xwd -root -display :5.0 -out outputfile
convert outputfile outputfile.png

I scped that image back to my desktop, and voila, there's a clock. Now we can copy up our HTML test files, and the Selenium server file (part of the Selenium RC download), and on the server, try running our tests:

export DISPLAY=:5.0 # firefox needs this to know where to find a display to run on
java -jar /path/to/selenium-server.jar -htmlSuite *firefox "http://www.example.com/" "/path/to/tests/suite.html" "/path/to/results/results.html"
w3m results.html

Sweet! Note that you'll want to set up your server to somehow launch xvfb on startup, so the virtual display is always available. Ok, onwards.

Build a test site

If you noticed above, our previous Jenkins build steps only configured the bare minimum Python environment to run our Django tests: just installthe necessary packages, make sure PYTHONPATH is correct, etc. We now need a complete setup of our site, including a database with known data to test against, and a way to reach it in a browser. Documenting this in a generic way is obviously not possible, since it's very project-specific. But Jenkins build steps are nice in that they're just shell commands, so just do whatever you need to do. I had to do stuff like set up a test domain name, configure Apache to point to the directory with the project code, dump in some known data into a test DB, copy a settings file to the right place, compile our JavaScript code into a production file, etc. And if you haven't already, be sure to add your Selenium test files to your code repository, since obviously we don't want to have to copy them up manually like we did above. You could potentially wrap all that up in a shell script, or, I guess people are using Ant or Maven for this kind of thing as well. Make sure you can visit your site and it looks normal before proceeding.

Also, be sure to first run your tests manually from the server (adjust the above command for any new paths) against the test site before attempting to get Jenkins to do it. Doing this I discovered that I needed to make a custom Firefox profile; since our site is using an SSL certificate and our test app is not using the correct one, my sample tests begain failing because the Firefox that Selenium launches was presenting the security warning page first (where locally I had confirmed an exception). This is easy to resolve, but yet another step. On the Mac, launch /Applications/Firefox.app/Contents/MacOS/firefox-bin -ProfileManager -no-remote, choose to create a new profile and name it Selenium-something, and then navigate to the test app. Get the warning, confirm and save the exception (or install the certificate), then quit Firefox. Then find that profile you just created, and copy it up to the server. Then you can launch your tests from the server like:

java -jar /path/to/selenium-server.jar -htmlSuite *firefox "http://www.example.com/" "/path/to/tests/suite.html" "/path/to/results/results.html" -firefoxProfileTemplate "/path/to/profile/foo.Selenium"

Now when you view your results.html, they should be all nice and green!

Tell Jenkins to report

So now that the test site is available and Selenium is happy running tests, we can (finally) get to having Jenkins do all this for us. This simply involves adding another build step to kick off running the test suite and getting the results. We'll need the Selenium HTML report plugin (note there is also the Seleniumhq plugin, but I couldn't get that to work). From within Jenkins, go to Manage Jenkins > Manage Plugins, click the Available tab, find and select it, click Install, and restart Jenkins.

Now, add the additional shell build step to your project and simply enter the same command that launched your tests above (don't forget to set the DISPLAY env var as well). Finally, under "Post-build Actions" check off "Publish Selenium Html Report" and point it to the directory of where your results are -- that is, if the command is outputting to /path/to/results/results.html, you enter /path/to/results, since it can parse more than one HTML file of results (meaning you can add as many build steps to run other test suites as you want, as long as they output to the same place). You should get details in the Console Output for a job (for debugging) and can see the results by clicking "Selenium Html Report" in the left column.

That's it for now, and I hope it works for you -- happy testing!