More and more useful applications, libraries and plugins are appearing that take away some of the pain that is sometimes involved in making software tests useful for entire teams. Examples of excellent tools that we have at our disposal right now are Hudson for continuous integration, Cucumber for integration tests and Selenium for automated testing of web applications in an actual browser. But making all these play together nicely involves more and more setting up of build servers and configuration.

Our goal is to document the steps required to overcome one of the hurdles that we kept running into, namely running our full Cucumber test suite including all Selenium scenarios on our Hudson integration server.

Ingredients

We’re going to use the following ingredients in this post:

• 1 Debian 5.0.4 ‘Lenny’ installation
• 1 installation of Xvfb
• 1 web browser (in our case iceweasel, Firefox rebranded by Debian)
• 1 Ruby on Rails 2.3.7 application
• 1 Capybara 0.3.8 gem
• As many Cucumbers (0.7.3) as necessary

We won’t go into the details of creating a Rails application, installing the gems and writing the Cucumber scenarios here. Their manuals are perfect guides for this.

Installing required packages

We start with installing a couple of packages that allow us to run headless tests in a web browser.

Virtual framebuffer with Xvfb

“In the X Window System, Xvfb or X virtual framebuffer is an X11 server that performs all graphical operations in memory, not showing any screen output.” Source: http://en.wikipedia.org/wiki/Xvfb

$ apt-get install xvfb

Web browser

After installing Xvfb we can go ahead and install a web browser.

$ apt-get install iceweasel

Before we continue we want to configure our browser profile so that it doesn’t whine about tabs being closed or not resuming correctly after it has crashed. If we don’t do this, our test suite might crash or run forever.

$ cd ~/.mozilla/firefox/xxxxxxxx.default/
$ vim user.js

Now put the following two lines in the user.js file:

user_pref("browser.sessionstore.enabled", false);
user_pref("browser.sessionstore.resume_from_crash", false);

Test Display

Before we start running our tests we will ant to verify that all packages were installed successfully. To do so start a virtual framebuffer (an Xvfb session) on display 99 with screen 0:

$ Xvfb :99 -ac -screen 0 1024x768x16

In a different terminal window do:

$ DISPLAY=:99.0 iceweasel http://example.org

This will start our web browser in the virtual framebuffer and open the example.com homepage in this browser. Next up is makinga ‘screenshot’ so that we can see what is going on inside our virtual framebuffer.

$ xwd -root -display :99.0 -out xwdout

And subsequently view the screenshot using:

$ xwud -in xwdout

Do you see the example.org homepage? Then Xvfb and iceweasel were installed successfully and we are now ready to do some testing.

Iceweasel in Xvfb

Running cucumbers

Before we integrate this setup into our continuous integration environment we do a test run to see if the cucumbers work with our new configuration. We can do this using the following command, keep in mind that we have to explicitly tell the cucumbers to use our virtual framebuffer display:

$ DISPLAY=:99.0 rake cucumber

If all goes well then we will see that all scenarios passed. If not all scenarios pass first check if they do run on in a non-headless situation.

Configuring Hudson

Now we have come to the point where we can tie all strings together. We need to add a new build step to the job that needs to run headless cucumbers. But, before adding this build step, we will create a start/stop script for our virtual framebuffer. This script can be used to start a buffer before running the scenarios, and stop the buffer after the scenarios completed. You can save this script to /etc/init.d. Make sure that the permissions are set so that the user that runs Hudson can execute it.

XVFB=/usr/bin/Xvfb
XVFBARGS="$DISPLAY -ac -screen 0 1024x768x16"
PIDFILE=/var/hudson/xvfb_${DISPLAY:1}.pid
case "$1" in
  start)
    echo -n "Starting virtual X frame buffer: Xvfb"
    /sbin/start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $XVFB -- $XVFBARGS
    echo "."
    ;;
  stop)
    echo -n "Stopping virtual X frame buffer: Xvfb"
    /sbin/start-stop-daemon --stop --quiet --pidfile $PIDFILE
    echo "."
    ;;
  restart)
    $0 stop
    $0 start
    ;;
  *)
  echo "Usage: /etc/init.d/xvfb {start|stop|restart}"
  exit 1
esac
exit 0

Source: http://marc.info/?l=tomcat-user&m=102335321103262&w=2

Final step is to add an ‘Execute shell’ build script to our Hudson job. You can use the following set of commands to run your cucumbers.

#!/bin/bash
export DISPLAY=:99
/etc/init.d/xvfb start
rake cucumber
RESULT=$?
/etc/init.d/xvfb stop
exit $RESULT

After adding this build step we save our job and let Hudson build it. If all goes well the cucumber scenarios should now be running as part of our continuous integration process. You can check the ‘Console output’ page of a Build in Hudson for hints when it didn’t run successfully.

Conclusion

If you have followed along with this post then you should have a working, headless setup by now. The advantage of this setup is that it is pretty light-weight and easy to setup. But this setup is not suited for you if you need to test across multiple browsers on different browsers. For such a setup you will want to have a look at virtual machines and Hudson slave agents.

Note: the following examples have been tested with DataMapper version 0.10.2.

One of the questions I had was how to perform multiple queries within a single (database) transaction, so that I can ‘stage’ a number of database queries and commit (or rollback) them all in one go. Fortunately, after some investigation of the DataMapper source code, it turns out to be an extremely simple task:

User.transaction do
  User.first.update(:name => "John Smith")
  Address.first.update(:street => "") # Invalid: street must not be blank
end

With this code neither the address nor the user will be updated in the database if one of the updates fails. The syntax is actually the same as for the ActiveRecord ORM. However, triggering a rollback is done differently, as can be seen in the following example.

User.transaction do |t|
  User.first.update(:name => "John Smith")
  Address.first.update(:street => "Oak St. 400")
  t.rollback # Transaction will be rolled back even when updates are OK.
end

That’s it for now. I’m sure there are some additional features that I haven’t mentioned here, but this is all I had to know at this point.

For the first time in its history Ruby en Rails was a multiple day event. The first day, October 31, was packed with interesting presentations on various Ruby and Rails related topics, by great speakers including renowned community idols like Yehuda Katz, Jeremy Kemper, Jonathan Weiss and many more. The second day (the ‘geek-day’) focused on more practical sessions and lightning talks. On this day another event was held: The RubyEnRails Rumble (RERR), a competition where teams of two had to complete an assignment within a certain time.

Since we from Kabisa consider ourselves to be Ruby programmers we wanted to take the challenge and see how we would do in this competition. Michel de Graaf and Ludo van den Boom were given the honor to rumble for Kabisa. This post will describe how we prepared for the RubyEnRails Rumble and how we have experienced it.

Rumble Preparations

After deciding that we would partake in the Rumble we knew that we would have to prepare ourselves as good as possible so that we could utilize our time at the Rumble as efficiently as possible. Neither of us had ever participated in an event like this, so we started of with a session in which we tried to decide what we could do to help us prepare for the Rumble. Together with our Kabisa colleagues we made a list of items and topics that could prove to be handy during the Rumble.

The hard part was that we had no experience and that we did not know what the assignment would be, so we dove into a broad range of subjects. With help from our colleagues we were able to refresh our knowledge of Ruby and Rails related technologies and built up a collection of snippets/links to quickly be able to perform standard tasks like retrieving tweets from Twitter and adding jQuery UI sugar to an application. In the end we prepared a Rails application template so that we would be able to quickly set-up a Rails skeleton application including basic functionality like authentication and authorization. Besides this template we also prepared a user interface template builder that could help us to quickly apply a ‘theme/skin’ to an application (interfaceLift).

For scaffolding we used dry_scaffold: This gem contains a rails generator that will replace the default rails scaffolding with more up to date DRY controllers and views. Controllers are implemented using inherited_resources. And the views are generated in haml that make use of Formtastic Forms to generate nice looking forms. The fork we where using also generated a search form that was implemented using searchlogic.

Another challenge was that we had not worked together before. So, apart from the technical toolkit that we set-up, we also had to find a way to get more acquainted with each others good and bad sides. We decided that the best way to do this would be to simulate a Rumble, and that’s what we did. Our colleague Ralph came up with an assignment that we worked on for about six hours. In these six hours we managed to crank out some decent results and learned how we could best divide the development tasks to our individual skill sets. This simulation prepared us for lots of things but it did not really simulate the most vital element of a competition: stress.

Rumble Ramblings

Some hints on what the assignment would be were dropped during the presentations on the first day. Still, we could only guess what exactly to expect. At the start of the Rumble the assignment was introduced and explained, the assignment was titled ‘Community Love’. It turned out that we had to create a system that would keep users of Ruby libraries up-to-date with updates about these libraries. The assignment was kept vague on purpose so that the teams had the ability to put their creative minds to work with finding a solution to the problems laid out in the assignment description.

So, now we had a little under eight hours left to design and build something that would convince the jury of our programming skills. After a short stand-up meeting we came up with a rough idea for what we would create.

The Plan…

As a Ruby/Rails developer you will probably make use of lots of gems in your application. You can specify these gem dependencies in your Rails configuration. Or make use of the awesome gem bundler. This will help you manage gem dependencies but they will not inform you when there are updates to the gems you are using. We figured that, as a developer who uses gems in a Ruby project you should be able to send a list of gem dependencies you use in a specific application to a service that will inform you when there are updates to the gems that you are using. An update could, for example, be a new version, a reported security vulnerability or a blog post. Since we had only one day to build something, we decided to focus on Rubygems exclusively. Mostly because there are already good centralized repositories where we would be able to easily find information about gems (gemcutter).

… And the Solution …

Our solution consists of three parts: (1) a gem that you use locally to update the list of gems you use in particular applications, (2) an online service (Rails application) where you can see the gems you use and (3) notification channels (i.e. email and twitter) through which news about gems is pushed to you. We named our solution ‘gemstreamer’, for it would stream information about gems to you as a user of these gems. For those of you who are interested to see what we actually made: you can see the ‘online service’ Rails application that is used as the central ‘information hub’ where information is gathered and distributed at http://github.com/ludo/rerr2009. For the gem that you would use locally to push the gems you use to the central hub see http://github.com/michel/gemstreamer. We deliberately kept the code as simple as possible and leveraged as many existing tools as possible to be able to quickly get things done. The end result is pretty simple, but the ideas behind it might actually be useful when extended and polished for real-life usage. This was also one of the comments from the jury, where they believed that something like this could make a great extension to http://gemcutter.org.

… At a Price

Did we mention the stress? Apparently we were both not very satisfied with how we made progress. There was so much that we wanted to do, and so little time to do it all. This is probably the biggest flaw in how we approached the Rumble. Before the Rumble we had already discussed that it would be best to make something small that works instead of something big that does not work. This reasoning sounded really good before the Rumble and after the Rumble, but we actually forgot it during the Rumble… Still, we managed to get our main ideas implemented which we were able to present at the end of the day.

Rumble Opponents

With five pretty strong teams the competition was fierce. It was interesting to see that every team approached the assignment from a different angle. Where our end result can be seen as an interesting prototype, others had created something that could be of use to developers right away. Here is a short summary of what the other teams created:

• The Fingertips team created Apprise for which they patched Bundler. This patch to was applied to Bundler By Yehuda the very next day.
• Iain Hecker and Marcel de Graaf built an extension to Webistrano that shows outdated gems, named ‘deproll‘.
• The i76 team (Tom-Eric Gerritsen and Rik Tonnard) built a Rails plugin that you can hook-up in your application so that you can see the dependencies that your app uses through a web interface (http://github.com/i76/rrrer2009).
• Leon Berenschot and Klaas-Jan Wierenga gave a cool presentation and won the Twitter vote (http://github.com/LeipeLeon/Dev-Enter).

Rumble Results

Victory! Completely unexpected we were awarded the first prize: two tickets to RailsConf 2010 in Baltimore, MD. The winner was selected by a jury and through public voting via Twitter. In the Twitter vote we ended as third, which we thought was already a great achievement. But the professional jury had unanimously picked us! These two votes combined made us the overall winners of the Ruby en Rails Rumble 2009: awesome!

In hindsight we can conclude that our plan was a little too ambitious as it turned out that eight hours is really not that much. Nonetheless, the rumble turned out to be a great experience that was really fun doing. It was a crazy day with a lot of stress, but it was also an interesting experiment to see how we would do against some other very good teams. The end result turned out to be even better than what we’ve hoped for. We want to thank the jury for choosing us as well as the people who voted for us on Twitter. Last but not least many thanks to the people who made Ruby en Rails possible this year, we had a great time!