The most basic testing techniques will get you far, but when things get complicated, you need to break out the big guns. What follows are a few tricks to try out when you run into a roadblock.
In a perfect world, all the resources that we needed would be self-contained in our application, and all interactions would take place in constant time. In our real work, life is nothing like this. We’ve got to deal with user input, database interaction, web service calls, file I/O, and countless other moving parts that live outside of our application. Testing these things can be painful.
Sure, we could set up a development database that gets blown out and reloaded every time our tests run—that’s what Rails does. We could read and write from temporary files, clearing out our leftovers after each example runs. For things like web services, we could build a fake service that acts the way we expect our live service to act and run it on a staging server. The question here is not whether it is possible to do this, but whether it is necessary.
Sometimes, you really do need to deal with real-world data. This is especially true when you want to tune and optimize performance or test resource-dependent interactions. However, in most cases, our code is mainly interested only in the behavior of the things we interact with, not what they really are. This is where either a mock or a stub could come in handy.
There are additional benefits to removing dependencies on external code and resources as well. By removing these extra layers, you are capable of isolating your examples so that they test only the code in question. This purposefully eliminates a lot of interdependencies within your tests and helps make sure that you find and fix problems in the right places, instead of everywhere their influence is felt.
Let’s start with a trivial example, to help you get your head around the concepts of mocks and stubs, and form a working definition of what they are.
What follows is some basic code that asks a user a yes or no
question, waits for input, and then returns true or
false based on the answer. A basic implementation
might look like this:
class Questioner
def ask(question)
puts question
response = gets.chomp
case(response)
when /^y(es)?$/i
true
when /^no?$/i
false
else
puts "I don't understand."
ask question
end
end
endGo ahead and toy around with this a bit by executing something similar to this little chunk of code, to get a sense for how it works:
q = Questioner.new
puts q.ask("Are you happy?") ? "Good I'm Glad" : "That's Too Bad"Interacting with this code by just running a simple script in the console is enough to show that it pretty much works as expected. However, how do we test it? Is it enough to break down the code so that it’s a bit more testable, allowing us to write tests for everything but the actual user interaction?
class Questioner
def ask(question)
puts question
response = yes_or_no(gets.chomp)
response.nil? ? ask(question) : response
end
def yes_or_no(response)
case(response)
when /^y(es)?$/i
true
when /^no?$/i
false
end
end
endNow most of the work is being done in yes_or_no, which is easily testable:
class QuestionerTest < Test::Unit::TestCase
def setup
@questioner = Questioner.new
end
%w[y Y YeS YES yes].each do |yes|
must "return true when yes_or_no parses #{yes}" do
assert @questioner.yes_or_no(yes), "#{yes.inspect} expected to parse as true"
end
end
%w[n N no nO].each do |no|
must "return false when yes_or_no parses #{no}" do
assert ! @questioner.yes_or_no(no), "#{no.inspect} expected to parse as false"
end
end
%w[Note Yesterday xyzaty].each do |mu|
must "return nil because #{mu} is not a variant of 'yes' or 'no'" do
assert_nil @questioner.yes_or_no(mu), "#{mu.inspect} expected to parse as nil"
end
end
endThese examples will all pass, and most of your code will be
tested, except for the trivial ask()
method. However, what if we wanted to build code that relies on the
results of the ask() method?
class Questioner
def inquire_about_happiness
ask("Are you happy?") ? "Good I'm Glad" : "That's Too Bad"
end
def ask(question)
puts question
response = yes_or_no(gets.chomp)
response.nil? ? ask(question) : response
end
def yes_or_no(response)
case(response)
when /^y(es)?$/i
true
when /^no?$/i
false
end
end
endIf we want to write tests that depend on the return value of
ask(), we’ll need to do something to
prevent the need for direct user input. A relatively simple way to test
inquire_about_happiness() is to
replace the ask() method with a stub
that returns our expected values
for each scenario:
class HappinessTest < Test::Unit::TestCase
def setup
@questioner = Questioner.new
end
must "respond 'Good I'm Glad' when inquire_about_happiness gets 'yes'" do
def @questioner.ask(question); true; end
assert_equal "Good I'm Glad", @questioner.inquire_about_happiness
end
must "respond 'That's Too Bad' when inquire_about_happiness gets 'no'" do
def @questioner.ask(question); false; end
assert_equal "That's Too Bad", @questioner.inquire_about_happiness
end
endIf we wanted to be a bit more formal about things, we could use a third-party tool to make our stubbing more explicit and easier to work with. There are lots of options for this, but one I especially like is the flexmock gem by Jim Weirich. We’ll look at this tool in much greater detail when we discuss formal mocking, but for now, let’s just look at how it can be used to clean up our stubbing example:
require "flexmock/test_unit"
class HappinessTest < Test::Unit::TestCase
def setup
@questioner = Questioner.new
end
must "respond 'Good I'm Glad' when inquire_about_happiness gets 'yes'" do
stubbed = flexmock(@questioner, :ask => true)
assert_equal "Good I'm Glad", stubbed.inquire_about_happiness
end
must "respond 'That's Too Bad' when inquire_about_happiness gets 'no'" do
stubbed = flexmock(@questioner, :ask => false)
assert_equal "That's Too Bad", stubbed.inquire_about_happiness
end
endThe example code accomplishes the same task as our manual stubbing, but does so in an arguably more pleasant and organized way. Though it might be overkill to pull in a third-party package just to stub out a method or two, you can see how this interface would be preferable if you needed to write tests that were a little more complicated, or at least more involved.
No matter how we implement them, stubs do allow us to improve our test coverage a bit more here. Still, let’s pause for a moment and ask ourselves a question: did we really finish our job? Looking at the code, we find that our naive implementation sans tests looks like this:
class Questioner
def inquire_about_happiness
ask("Are you happy?") ? "Good I'm Glad" : "That's Too Bad"
end
def ask(question)
puts question
response = gets.chomp
case(response)
when /^y(es)?$/i
true
when /^no?$/i
false
else
puts "I don't understand."
ask question
end
end
endOur test-driven results turn out like this:
class Questioner
def inquire_about_happiness
ask("Are you happy?") ? "Good I'm Glad" : "That's Too Bad"
end
def ask(question)
puts question
response = yes_or_no(gets.chomp)
response.nil? ? ask(question) : response
end
def yes_or_no(response)
case(response)
when /^y(es)?$/i
true
when /^no?$/i
false
end
end
endThough we’ve successfully split out our
yes_or_no parser for testing, we still don’t have any
automated checks for how our code will display a question to the user
and how it will respond based on that code. Presently, the only safety
net we have for our I/O code is our limited testing in our terminals,
which can hardly be called robust. Although it is of course better to
have some coverage than no coverage at all, we can do better
here.
Ruby ships with a StringIO
class, which essentially is an IO
object that is implemented to work against a string rather than the
typical file handles. Although I hesitate to call this a mock object, it
comes close in practice. We’ll take a quick look at how you might use it
to test I/O code, which is a nice stepping stone that can lead us into
real mock territory.
But before we can test with StringIO, we need to make it so that our
Questioner class allows us to swap
out the input and output sources for our own custom objects:
class Questioner
def initialize(in=STDIN,out=STDOUT)
@input = in
@output = out
end
def ask(question)
@output.puts question
response = @input.gets.chomp
case(response)
when /^y(es)?$/i
true
when /^no?$/i
false
else
@output.puts "I don't understand."
ask question
end
end
endBy default, nothing will change and I/O will still go to STDIN and STDOUT. However, this opens the door for
replacing these I/O objects with a pair of StringIO
objects, allowing us to totally rethink our tests:
class QuestionerTest < Test::Unit::TestCase
def setup
@input = StringIO.new
@output = StringIO.new
@questioner = Questioner.new(@input,@output)
@question = "Are you happy?"
end
["y", "Y", "YeS", "YES", "yes"].each do |y|
must "return false when parsing #{y}" do
provide_input(y)
assert @questioner.ask(@question), "Expected '#{y}' to be true"
expect_output "#{@question}\n"
end
end
["n", "N", "no", "nO"].each do |no|
must "return false when parsing #{no}" do
provide_input(no)
assert !@questioner.ask(@question)
expect_output "#{@question}\n"
end
end
[["y", true],["n", false]].each do |input,state|
must "continue to ask for input until given #{input}" do
provide_input "Note\nYesterday\nxyzaty\n#{input}"
assert_equal state, @questioner.ask(@question)
expect_output "#{@question}\nI don't understand.\n"*3 + "#{@question}\n"
end
end
def provide_input(string)
@input << string
@input.rewind
end
def expect_output(string)
assert_equal string, @output.string
end
endWithout too much more effort, we were able to specify and test the
full behavior of this trivial little program. We are able to test both
the logic, and the actual I/O operations, to verify that they work as we
expect them to. In this particular case, we were pretty lucky that Ruby
ships with a library that acts like an I/O object and makes our testing
easier. We won’t always be so lucky. What’s more, we don’t really need
most of what StringIO has to offer here. A lighter
(albeit more abstract) approach would be to use a formal mocking
framework to do the job. Let’s take a look at how this problem might be
solved in flexmock, to make things a bit clearer:
require "flexmock/test_unit"
class QuestionerTest < Test::Unit::TestCase
def setup
@input = flexmock("input")
@output = flexmock("output")
@questioner = Questioner.new(@input,@output)
@question = "Are you happy?"
end
["y", "Y", "YeS", "YES", "yes"].each do |y|
must "return false when parsing #{y}" do
expect_output @question
provide_input(y)
assert @questioner.ask(@question), "Expected '#{y}' to be true"
end
end
["n", "N", "no", "nO"].each do |no|
must "return false when parsing #{no}" do
expect_output @question
provide_input(no)
assert !@questioner.ask(@question)
end
end
[["y", true], ["n", false]].each do |input, state|
must "continue to ask for input until given #{input}" do
%w[Yesterday North kittens].each do |i|
expect_output @question
provide_input(i)
expect_output("I don't understand.")
end
expect_output @question
provide_input(input)
assert_equal state, @questioner.ask(@question)
end
end
def provide_input(string)
@input.should_receive(:gets => string).once
end
def expect_output(string)
@output.should_receive(:puts).with(string).once
end
endThe interesting thing about this example is that flexmock() returns a completely generic
object, yet this accomplishes the same results as using
StringIO, which is finely tuned for emulating a real
IO object. The end result is that the
latter example tends to focus on the interactions between your code and
the resource, and that the former example is more directly bound to what
an I/O object actually is. It can be beneficial to avoid such tight
distinctions, especially when working in Ruby, where what an object
actually is tends to be less important than what it can do.
To generalize: mock objects essentially break interactions down into the messages that an object should receive, the arguments that accompany the messages, the return values of the methods, whether a block is yielded, and whether any errors should be raised. If this sounds like a lot, don’t worry too much. The beauty of a mock object is that you need to specify only those things that are necessary to handle in your code.
Flexmock (like many of the other Ruby mocking options) is quite robust, and to go over it extensively here would take more than just a single section of a chapter. However, through this simple example, you can see that there are ways to avoid actively hitting your external resources while still being able to test your interactions with them.
Of course, using a mock object comes with its own cost, like
anything else. In this example, if we changed the internal code to use
print() instead of puts(), we would need to modify our mock
object, but we would not need to modify our StringIO-based solution. Although a mock
object completely eliminates the need to worry about the internal state
of your dependencies, it creates a tighter coupling to their interfaces.
This means that some care should be taken when deciding just how much
you want to mock out in any given test suite.
Learning how to build decent mock objects without going overboard takes some practice, but is not too hard once you get the hang of it. It ultimately forms one of the hard aspects of testing, and once that bridge is crossed, only a few more remain.
Dealing with programs that need to generate complex output can be a pain. Verifying that things actually work as you expect them to is important, but simply comparing raw output values in an automated test leads to examples that are nearly impossible to follow. However, we often resort to just dumping our expected data into our tests and comparing it to what we’re actually generating. This sort of test is useful for detecting when a problem arises, but finding the source of it, even with decent diff utilities, can be a real pain.
Imagine we’ve got a basic blog that needs to output RSS, which is really just a specialized XML format. The following example is a simplified version of what I use to generate the feeds in my blog. James Gray actually wrote the code for it, using XML Builder, another great gem from Jim Weirich:
require "builder"
require "ostruct"
class Blog < OpenStruct
def entries
@entries ||= []
end
def to_rss
xml = Builder::XmlMarkup.new
xml.instruct!
xml.rss version: "2.0" do
xml.channel do
xml.title title
xml.link "http://#{domain}/"
xml.description description
xml.language "en-us"
@entries.each do |entry|
xml.item do
xml.title entry.title
xml.description entry.description
xml.author author
xml.pubDate entry.published_date
xml.link entry.url
xml.guid entry.url
end
end
end
end
end
endWe need to test that the output of this to_rss method is what we expect it to be. The
lazy approach would look like this:
require "time"
class BlogTest < Test::Unit::TestCase
FEED = <<-EOS
<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
><channel><title>Awesome</title><link>http://majesticseacreature.com/</link>
<description>Totally awesome</description><language>en-us</language><item>
<title>First Post</title><description>Nothing interesting</description>
<author>Gregory Brown</author><pubDate>2008-08-08 00:00:00 -0400</pubDate>
<link>http://majesticseacreature.com/awesome.html</link>
<guid>http://majesticseacreature.com/awesome.html</guid></item></channel></rss>
EOS
def setup
@blog = Blog.new
@blog.title = "Awesome"
@blog.domain = "majesticseacreature.com"
@blog.description = "Totally awesome"
@blog.author = "Gregory Brown"
entry = OpenStruct.new
entry.title = "First Post"
entry.description = "Nothing interesting"
entry.published_date = Time.parse("08/08/2008")
entry.url = "http://majesticseacreature.com/awesome.html"
@blog.entries << entry
end
must "have a totally awesome RSS feed" do
assert_equal FEED.delete("\n"), @blog.to_rss
end
endYou could make this slightly less ugly by storing your output in a file, but it’s not much better:
class BlogTest < Test::Unit::TestCase
def setup
@blog = Blog.new
@blog.title = "Awesome"
@blog.domain = "majesticseacreature.com"
@blog.description = "Totally awesome"
@blog.author = "Gregory Brown"
entry = OpenStruct.new
entry.title = "First Post"
entry.description = "Nothing interesting"
entry.published_date = Time.parse("08/08/2008")
entry.url = "http://majesticseacreature.com/awesome.html"
@blog.entries << entry
end
must "have a totally awesome RSS feed" do
assert_equal File.read("expected.rss"), @blog.to_rss
end
endIn the end, the issue boils down to the fact that you’re definitely not focusing on the important parts of the problem if you have to check the output character by character. An RSS feed with some extra whitespace in it would be no less valid than the file shown here, yet it would cause an annoying failure in your tests.
Unless it really isn’t worth your time, the best way to deal with complex output is to parse it into a workable dataset before doing your comparisons. There are a few RSS feed parsers out there that would make quick work of a file like this. However, in the interest of generality, we could use a generic XML parser without much more effort.
There are a few solid choices for XML parsing in Ruby, and even support for it in the standard library. However, the library that I find most pleasant to work with is the nokogiri gem, written by Aaron Patterson. Here’s what part of the tests look like after they’ve been reworked to use Nokogiri:
require "time"
require "nokogiri"
class BlogTest < Test::Unit::TestCase
def setup
@blog = Blog.new
@blog.title = "Awesome"
@blog.domain = "majesticseacreature.com"
@blog.description = "Totally awesome"
@blog.author = "Gregory Brown"
entry = OpenStruct.new
entry.title = "First Post"
entry.description = "Nothing interesting"
entry.published_date = Time.parse("08/08/2008")
entry.url = "http://majesticseacreature.com/awesome.html"
@blog.entries << entry
@feed = Nokogiri::XML(@blog.to_rss)
end
must "be RSS v 2.0" do
assert_equal "2.0", @feed.at("rss")["version"]
end
must "have a title of Awesome" do
assert_equal "Awesome", text_at("rss", "title")
end
must "have a description of Totally Awesome" do
assert_equal "Totally awesome", text_at("rss", "description")
end
must "have an author of Gregory Brown" do
assert_equal "Gregory Brown", text_at("rss", "author")
end
must "have an entry with the title: First Post" do
assert_equal "First Post", text_at("item", "title")
end
def text_at(*args)
args.inject(@feed) { |s,r| s.send(:at, r) }.inner_text
end
endThis is a huge improvement! Now, our tests actually look like they’re verifying the things we’re interested in, rather than simply checking our output against some amorphous code blob that we can’t easily inspect and verify.
Of course, this approach to testing complex data requires you to trust whatever you are using to parse your output, but as long as you can do that, the ability of whatever library you use to parse your output is from the very start an indication that you are producing meaningful results.
Not every file format you will encounter will have parsers available for it, of course. Some of the formats you need to produce may even be fully custom-made. However, providing that it isn’t impossible to build one, a parser will come in handy for making your tests more flexible and expressive. Consider this possibility before turning to direct file comparison as a last resort only.
We’re about to wrap up with a mixed bag of tips and tricks for keeping your test suite maintainable, but before we do that, let’s go over some of the highlights of the advanced testing techniques discussed in this section:
Mocks and stubs can be used to remove external dependencies from tests while still verifying proper behavior and interaction.
Stubs are used when we want to replace some functionality with canned results to make testing other code easier.
Mocks are used to create objects that can act in place of an external resource for the purpose of testing. Mock objects are set up with expected responses, which are then verified when the tests are run. This means that if you have something like
my_obj.should_receive(:foo).onceandfoois never called onmy_obj, this will result in a test failure. This is the primary difference between mocks and stubs.When testing complex output, it is best to find a tool that parses the output format you are generating, and write your tests against its results.
When you can’t find a tool for parsing your output format, you might consider building one that parses only the values you are interested in, in addition to necessary basic validation of the document’s structure.
If it isn’t possible to parse your generated data without great effort, consider storing your expected output in its own file and loading it into your tests as needed, using a diff utility to compare expected and actual output.
For most XML formats, Nokogiri does a great job of parsing the document and making it easily searchable.
Get Ruby Best Practices now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
