Creating Rake::TestTasks on-the-fly

Published February 06, 2007

My new hacker-for-hire position at JobFutures (yes, a new public site will be live in a matter of weeks) has me working on some interesting new projects, one of which is extracting out some core models into plugins to share between our applications. Each plugin is testable on its own, but we need a way to test the main application and the core plugins in one hit.

My first idea was just to have a TestTask to run all the test files. That seemed like a simple enough approach, the only problem though is that the test_helper’s all mix into Test::Unit::TestCase and set their fixture_paths willy nilly.

Eventually our good old friend eval came to the rescue. In the code below we dynamically create TestTaskss, invoking them along the way.


CORE_PLUGINS = %w(jf_core_models ...)

desc "Test the core plugins and the main application"
task :the_lot do
  had_errors = false
  
  CORE_PLUGINS.each do |p|
    test_name = "test_#{p}"
    eval <<-CODE
      Rake::TestTask.new(:#{test_name} => :environment) do |t|
        t.libs << "test"
        t.verbose = true
        t.pattern = "vendor/plugins/#{p}/test/**/*.rb"
      end
    CODE
    Rake::Task[test_name].invoke rescue had_errors = true
  end
  Rake::Task["test"].invoke rescue had_errors = true

  raise "There were test failures" if had_errors
end

The most unsexy part is the had_errors, but I can’t think of nice why to write that. We need to catch the errors because when a Rake::TestTask fails it throws an exception, but we want to run all the tests, even if some fail, and still have Ruby give the correct error return code to shell.

It’s not the first time lately that eval has come to the rescue. I have a feeling that I’m soon going to be known around the office as “eval black magic boy”.

Archived comments

Comments were previously allowed on articles. Though no new comments are being accepted you can see the old comments below.

James Adam
Is there any reason why you don’t generate the TestTasks outside of your “task :the_lot” task, and then have all the generated tasks as dependencies?
```
namespace :test do
  CORE_PLUGINS = %w(jf_core_models ...)
  CORE_PLUGINS.each do |p|
    Rake::TestTask.new(p => :environment) do |t|
      t.libs << "test" 
      t.verbose = true
      t.pattern = "vendor/plugins/#{p}/test/**/*.rb" 
    end
  end
  desc "Test the core plugins and the main application" 
  task :the_lot => CORE_PLUGINS
end
```
I’m not sure if that would retain the error-handling behaviour that you’re after, however.

If the fixture paths are the real issue, have a look at the testing extensions that the engines plugin 1.2 release provides – it was developed with testing multiple plugins in mind.
Tim Lucas
James: Nice, I like it. It does though have the problem of failing if any of the plugin tests fail… you’d need to expand the task to do that, as you said.

Cheers for that link. We handle fixture paths using a plugin we’ve developed, ExternalFixtures, so we can specify:
```
class Admin::UsersControllerTest < Test::Unit::TestCase
  fixtures :access_logs
  external_fixtures PLUGIN_PATHS[:jf_core_models] => [:users, :roles, :user_roles]
end
```
I’m not a fan of the idea of copying around fixture files.
James Adam

I’d be interested to see your plugin. Earlier versions of the engines plugin tried to add more “smarts” to the fixtures method, specifying the file and the class to be used, but it became too complicated to support as Rails itself evolved.

The current approach of mirroring all the fixtures to a known temporary location when the tests are run is the simplest, least intrusive solution that should still work for everyone, which is why I opted for that rather than some (much needed) patching to the fixtures mechanism itself. It’s pragmatism over elegance, unfortunately.

Keep up the good work!
Tim Lucas

Our External Fixtures plugin is a bit of a hack. We had to call the original Test::Unit::TestCase#load_fixtures from within our Test::Unit::TestCase#load_fixtures_with_external_fixtures, and then copy+paste the original #load_with_fixtures implementation.

It’d be a simple core patch to parameterize #load_with_fixtures method so we don’t have to reimplement it. That’d need to be accepted into core before we could release the plugin.

For the moment, the Engines way of doing things is probably best for general consumption.

Previously: Logging Rails activity in script/console

Next up: HTML encoding your flash in your controller

Creating Rake::TestTasks on-the-fly

Archived comments

Thoughts

toolmantim