Blog » Wizardry

Three Go Wild In Berlin

 

One bright Monday morning in September, 3 wizards set out for Berlin. Their quest: to attend RailsConf Europe 2007.

Railsconf logo

A smooth journey was followed by disaster at the hotel - one of our rooms was not ready yet. Fortunately, it turned out we were on the executive floor and so got to enjoy beers, newspapers and nibbles in Hotel Melia’s Executive Lounge.

How many techies does it take to find a conference?
Having checked out the room (bounced on the bed, put the towels and slippers in suitcases, eaten all complementary biscuits and/or gummy sheep), we then went to check out the town. As none of us had the sense to bring any info on the conference’s whereabouts, we made sure to find the hotel that housed it. This turned out to be easy - just follow the stream of t-shirts with computer “jokes” on them. “Run Dos Run” never gets old.

Day 2 in the big Berlin hotel…
…and we nearly missed the conference. I showed off my travel experience by setting my alarm but forgetting to set the clock an hour forward. Fortunately, we were saved by Fred’s ninja-like breakfast sense - there was no way he was going to miss breakfast in the Executive Lounge before the conference.

Fred at Eurails
(Original photograph copyright James Duncan Davidson)

Hang on a minute, why are we here again?
And so we finally made it RailsConf Europe 2007. There followed 2 days of varied talks, demonstrations, general hobnobbing and excellent food, including sightings of celebrities such as his DHHness and Martin Fowler, a trip to RejectConf in a bar called the Pirates Cove (which means what with also visiting the Ambulance Bar, I can only assume all bars in Berlin have black walls and look like they used to be boiler rooms) and my personal highlight, Evan Phoenix’s wonderful graphs to demonstrate that Rubinius, while slower than the space shuttle, was faster than a tortoise and even a 3-legged dog.

Homeward Bound

Mostly due to Steve’s shoe exploding only after we’d got through security, the journey back was relatively unexciting and, of course, we found that it was raining in Stansted.

Loading Fixtures With Parsimony

 

A little while ago I came across this interesting blog post about making tests run faster. In a nutshell the author managed to speed up their unit tests considerably by preloading the fixture data into the database (via a Rake task) and then not using any of the fixture stuff that Rails gives you. There’s a hefty price to pay though: you can’t use the fixture accessors, so users(:bob) becomes User.find(1) and comments(:bobs_comment_to_alice) becomes Comment.find(12).

We’re in a similar situation: we have several reasonably large Rails apps and tests that were taking more and more time to run. Other people have tried completely mocking out the database altogether. This speeds things up massively, but we really don’t want to go there: our interactions with the database are an important part of our application and often non trivial: it is vital they be covered by our tests. The speedup described above sounded good, but we weren’t ready to get rid of the fixtures accessors as they really help with the readability and maintainability of the tests.

A New Hope

How much can we claw back without having to give up fixture accessors? It turns out the answer is a lot. Most of our tests use several models (via the associations for example), for example a lot of the tests involve a question in someway (it is after all our business). Rails already caches fixtures on a per testcase basis, but that still means fixtures are reloaded once for each foo_test.rb file. Like any sane person we use transactional tests and so the tests themselves never clobber test data: we should never have to reload anything. Surely we can do better.

Plugin Baby!

We thought about this a few months ago and initially came up with a crude solution: put all the fixtures in test_helper.rb and only have them loaded once. Effective, although it would nice not to have this requirement. With a little more work a cleverer solution presents itself: override Fixture.create_fixtures. We stash away fixtures when they are loaded and if the same fixture is loaded again we just retrieve it from out cache. Using the plugin is easy, just grab it from here and add require 'faster_fixtures' to your test_helper.rb

A Wrinkle in Time

There’s one extra little wrinkle. Because previously fixtures were reloaded once per testcase, the data in the test database was never more then a few seconds old. Now that the data is only loaded once per test run, it might easily be 2-3 minutes old by the time the last tests run. Depending on how your tests are written this might be a problem. Consider for example the following:

Some fixtures:

recent_post:
  created_at: <%= 4.minutes.ago.to_s(:db)%>
comment_on_recent_post:
  created_at: <%= 3.minutes.ago.to_s(:db)%>

Some code:

def find_recent_posts
  Post.find :all, :conditions => ['created_at > ?', 5.minutes.ago]
end

and a test

def test_find_recent_posts
  assert_equal [posts(:recent_post)], Post.find_recent_posts
end

Nothing very exciting. However the change to the way fixtures are loaded can cause this test to fail when all the tests to run: if it runs towards then end, then the test data may be over 1 minute old and so posts(:recent_post) will be over 5 minutes old. There are also issues of consistency: if the comments fixture is loaded more than a minute before the posts fixture then you’d have a comment created before the post it is commenting on!

Time Travel

So, fix all tests to remove this stuff? Ugly. Marty, you’re not thinking fourth dimensionally! All we need is a little time travel, and for that we can use mocha. Mocha is a nice little mocking and stubbing library (although I expect you could use a different one).

The faster_fixtures plugin defines the following method

def freeze_now(frozen_now)
  Time.stubs(:now).returns(frozen_now)
end

Calling this method stops time (at least if you’re inside the Ruby interpreter). We also ensure this is called as part of the test setup.

60% of the time, works every time

Well your mileage may vary, but we’ve used it on all of our apps. On the biggest app it cut test run time by 35%, and for the next biggest it cut test time by just a shade over 50%. Other people have reported similar gains. Typically unit tests show bigger gains, but functional tests usually show an appreciable gain.

Update

As of 7662, faster fixtures (minus the time correcting stuff) has been committed to the rails trunk. I will probably retool the plugin so that it can offer time freezing if required.

Escape from Zurg

 

Ruby Logo “Escape from Zurg” is a puzzle (featuring characters from the movie “Toy Story”) which has been used to teach students Logic Programming (in, for example, Prolog). Recently I came across a very interesting paper which presents an elegant Haskell solution to the riddle. This got me wondering about whether I could come up with a similarly satisfying solution in Ruby.

After a few false starts (mainly a result of me trying to transliterate the Prolog or Haskell solutions rather than come up with something which worked well in Ruby) I’ve come up with something that I’m pretty happy with. Not only is it (I think) concise and clear, but it’s also efficient. Given that Ruby doesn’t have built-in searching (like Prolog) and and doesn’t support lazy evaluation or list comprehension (like Haskell), the fact that the Ruby implementation is as straightforward as it turns out to be is, I think, a testament to the power of the language.

Here’s the puzzle:

Buzz, Woody, Rex, and Hamm have to escape from Zurg. They merely have to cross one last bridge before they are free. However, the bridge is fragile and can hold at most two of them at the same time. Moreover, to cross the bridge a flashlight is needed to avoid traps and broken parts. The problem is that our friends have only one flashlight with one battery that lasts for only 60 minutes (this is not a typo: sixty). The toys need different times to cross the bridge (in either direction):

Buzz: 5 minutes
Woody: 10 minutes
Rex: 20 minutes
Hamm: 25 minutes

Since there can be only two toys on the bridge at the same time, they cannot cross the bridge all at once. Since they need the flashlight to cross the bridge, whenever two have crossed the bridge, somebody has to go back and bring the flashlight to those toys on the other side that still have to cross the bridge.

The problem now is: In which order can the four toys cross the bridge in time (that is, in 60 minutes) to be saved from Zurg?

Like the Haskell solution, the Ruby solution depends on a SearchProblem class which looks after the task of generating candidate solutions:

SearchProblem = Struct.new(:initial) do
  def each_candidate(&proc)
    step [], initial, &proc
  end
 
  def step(history, state, &proc)
    if state.terminal?
      yield history
    else
      state.each_successor do |move, state|
        step history + [move], state, &proc
      end
    end
  end
end

SearchProblem takes an initial State which provides two methods; terminal? returns true if the state is terminal (i.e. in our case, all the toys are on the right hand side of the bridge) and each_successor calls a block with each valid move from that state, together with the new state after that move.

SearchProblem provides an each_candidate method which calls the provided block with each candidate solution in turn. Crucially, candidate solutions are generated as needed, so we don’t have to hold the entire search tree in memory (not a big deal for a problem of this size, but definitely an issue for a “real world” problem!). The Haskell solution achieves a similar effect through lazy evaluation (contrast with this Erlang solution which does have to generate the entire search tree).

To define the particular problem at hand, we first define our Toys:

Toy = Struct.new(:name, :time)
Toys = [
  Toy.new(:buzz, 5),
  Toy.new(:woody, 10),
  Toy.new(:rex, 20),
  Toy.new(:hamm, 25)]

Next, we define Move which we will use to represent transitions between states. In this case, a move consists of a direction (right or left) and a group of toys (two toys for a move to the right, one toy for a move to the left):

Move = Struct.new(:direction, :who) do
  def cost
    who.collect(&:time).max
  end
end

Move provides one method, cost which returns the time taken to complete the move (the implementation of which makes use of the Symbol#to_proc trick).

Most of the heavy lifting is performed by the State class:

State = Struct.new(:pos, :group) do
  def terminal?
    group.empty?
  end
 
  def each_successor
    case pos
      when :left
        group.each_pair do |pair|
          yield Move.new(:right, pair), State.new(:right, group - pair)
        end
      when :right
        (Toys - group).each do |toy|
          yield Move.new(:left, [toy]), State.new(:left, group + [toy])
        end
    end
  end
end

A State comprises two attributes, pos which represents the current flashlight position and group which represents the toys remaining on the left-hand side of the bridge (so the toys on the right-hand side will be Toys - group).

The implementation of State#terminal? is obvious — if group is empty (i.e. if there are no toys on the left-hand side) then we’re done.

State#each_successor has to deal with two cases, depending on the position of the flashlight. The implementation utilises a small utility method on Array which allows all unique pairs of elements of the array to be generated:

class Array
  def each_pair
    each_with_index do |x, i|
      self[i + 1 ... length].each {|y| yield [x, y]}
    end
  end
end

Finally, we create an instance of SearchProblem with our initial state and print out the first candidate solution we find in which the time taken is no more than 60 minutes:

problem = SearchProblem.new State.new(:left, Toys)
puts problem.each_candidate {|history|
  break history if history.inject(0) {|acc, move| acc + move.cost} <= 60
}

As it happens (by sheer fluke) with the Toys array initialized in the natural order, the very first solution generated turns out to be the correct one, so the amount of work performed is the bare minimum. Clearly we can’t always rely on being so lucky in general, however!

To my mind, this solution is at least as easy to understand (and as flexible) as the Haskell equivalent (and both are a significant improvement on the Prolog). I would very much welcome comments and suggestions, however!

The complete source code can be downloaded here.

Update (2007-09-11)

There’s an interesting discussion about this problem on the ruby-talk mailing list.

Hidden feature in Ruby’s Struct

 

Ruby LogoThe Ruby core library contains a nice little utility class called Struct, which provides a convenient way to bundle a number of attributes together, using accessor methods, without having to write an explicit class. So this:

class Customer
  attr_accessor :name
  attr_accessor :address
 
  def initialize(name, address)
    @name = name
    @address = address
  end
end

is (broadly) equivalent to this:

Customer = Struct.new(:name, :address)

Much nicer (and DRYer)!

But what if you want to define methods on the new class that you’ve just created? The Pickaxe Book says:

Ruby 1.9 and later allow you to pass a block to a Struct’s constructor. This block is evaluated in the context of the new struct’s class and hence allows you conveniently to add instance methods to the new struct.

Unfortunately most of us haven’t (yet) moved to Ruby 1.9, but there is good news! It turns out that this functionality has actually been present ever since 1.8.3 (here’s the relevant ChangeLog entry, although note that when it refers to [ruby-talk:02606], it should really refer to [ruby-core:02606]).

Imagine, for example, that we wanted to add a custom to_s method to our Customer class:

Customer = Struct.new(:name, :address) do
  def to_s
    "A customer called '#{name}' living at '#{address}'"
  end
end

Sweet!

Update (2007-09-05)

It turns out that there is another commonly used idiom which achieves much the same effect:

class Customer < Struct.new(:name, :address)
  def to_s
    "A customer called '#{name}' living at '#{address}'"
  end
end

However, it also turns out that both of these idioms may have problems when combined with Rails. For further information see this discussion on the ruby-talk mailing list.

82ASK wins National Business Award

 

National Business Awards logo 82ASK is going to have to invest in a new trophy cabinet after picking up another gong in the National Business Awards Midlands and East of England region.

82ASK walked away with The Orange Best Use of Technology in Business Award and was highly commended in the Business Innovation of the Year Award.

82ASK CTO Paul Butcher said “We have an outstanding technical team here at 82ASK, and this award is a vindication of their hard work and dedication over the last two years as we’ve built the technical platform which will allow us to scale even greater heights in the coming years.”

82ASK at the National Business Awards
Henry Addison, Fred Cheung, Sarah McVittie and Paul Butcher representing 82ASK at the National Business Awards