Auto-Saving Screenshots on Test Failures & Other Capybara Tricks

Recently, when working on a JavaScript-heavy Backbone.js + Rails application, I spent a significant amount of time working with Capybara and the JavaScript driver Poltergeist for automated browser testing. After struggling with some specific asynchronous test steps, I stumbled across a great blog post detailing some effective asynchronous helper methods: wait_for_ajax, wait_for_dom, and the recently removed Capybara method wait_until. I ended up using a slightly modified version of wait_until that allows me to pass in a block that must return true:

 def wait_until
 require "timeout"
 Timeout.timeout(Capybara.default_wait_time) do
 sleep(0.1) until yield == true
 end
end

wait_until { dom_element['class'].include? "is-active" }

While these asynchronous helper methods are great for assisting with finicky JS tests, I ran into some frustration when tests would fail on our Continuous Integration (CI) server (at Viget we use Jenkins). Tests involving JavaScript rely heavily on timing. It is fairly easy to have a test pass on your speedy development machine, only to have the test fail on CI either because it is running slower due to parallel builds happening or simply because of the timing of executing the assertions. Even after increasing the Capybara.default_wait_time setting, I was still getting seemingly random test failures.

After a few times ssh'ing into our Jenkins server, modifying the test to save a screenshot, and then scp'ing that file back to my machine in order to view it, I decided to try and automate this process. My first idea was to make a plugin that would automatically save a screenshot of a failing test and then email that image to a configurable set of receivers. After playing around with it a bit, I realized that you could just access the images hosted right on the CI server.

This bit of code will automatically take a screenshot of a failing JavaScript test. I've decided to assert that the type of test is ':js => true' because the default Capybara driver (RackTest) does not support saving screenshots.

 RSpec.configure do |config|
 config.after(:each) do
 if example.exception && example.metadata[:js]
 meta = example.metadata
 filename = File.basename(meta[:file_path])
 line_number = meta[:line_number]
 screenshot_name = "screenshot-#{filename}-#{line_number}.png"
 screenshot_path = "#{Rails.root.join("tmp")}/#{screenshot_name}"

 page.save_screenshot(screenshot_path)

 puts meta[:full_description] + "\n Screenshot: #{screenshot_path}"
 end
 end
end

Now when a finicky JavaScript test fails on Jenkins, the auto-emailed report includes a URL to the saved screenshot:

I was pleasantly surprised to find that Jenkins converts a path from the console output into a convenient URL.

This allows me to simply click the provided link to see what is happening in the browser, which has been an immense time-saver in diagnosing failing tests.

What kind of tricks do you use for asynchronous browser tests and/or debugging them? Please leave a comment below!