Perl Testing: A Developer's Notebook

BUY THIS BOOK

Print Book $29.95

Safari Books Online

What is this?

Print Book £20.95

What is this?

Looking to Reprint this content?

Tell a friend

Perl Testing: A Developer's Notebook

By Ian Langworth, chromatic
Price: $29.95 USD
£20.95 GBP

Cover | Table of Contents | Colophon

Chapter 1: Beginning Testing

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

You've heard about the benefits of testing. You know that it can improve your code's reliability and maintainability as well as your development processes. You may even know about the wide range of available modules and idioms that Perl offers for testing Perl and non-Perl programs. In short, you may know everything except where to start.

The labs in this chapter walk through the most basic steps of running and writing automated tests with Perl. By the end of the chapter, you'll know how to start and continue testing, how Perl's testing libraries work, and where to find more libraries to ease your workload.

One of Perl's greatest strengths is the CPAN, an archive of thousands of reusuable code libraries—generally called modules—for almost any programming problem anyone has ever solved with Perl. This includes writing and running tests. Before you can use these modules, however, you must install them. Fortunately, Perl makes this easy.

The best way to install modules from the CPAN is through a packaging system that can handle the details of finding, downloading, building, and installing the modules and their dependencies.

Section 1.1.1.1: Through the CPAN shell

On Unix-like platforms (including Mac OS X) as well as on Windows platforms if you have a C compiler available, the easiest way to install modules is by using the CPAN module that comes with Perl. To install a new version of the Test::Simple distribution, launch the CPAN shell with the cpan script:

    % cpan
    cpan shell -- CPAN exploration and modules installation (v1.7601)
    ReadLine support enabled

    cpan> install Test::Simple
    Running install for module Test::Simple
    Running make for M/MS/MSCHWERN/Test-Simple-0.54.tar.gz

    <...>

    Appending installation info to /usr/lib/perl5/5.8.6/powerpc-linux/perllocal.pod
      /usr/bin/make install UNINST=1 -- OK

You can also run the CPAN shell manually with perl-MCPAN-e shell.

If Test::Simple had any dependencies (it doesn't), the shell would have detected them and tried to install them first.

If you haven't used the CPAN

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Installing Test Modules

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The best way to install modules from the CPAN is through a packaging system that can handle the details of finding, downloading, building, and installing the modules and their dependencies.

Section 1.1.1.1: Through the CPAN shell

    % cpan
    cpan shell -- CPAN exploration and modules installation (v1.7601)
    ReadLine support enabled

    cpan> install Test::Simple
    Running install for module Test::Simple
    Running make for M/MS/MSCHWERN/Test-Simple-0.54.tar.gz

    <...>

    Appending installation info to /usr/lib/perl5/5.8.6/powerpc-linux/perllocal.pod
      /usr/bin/make install UNINST=1 -- OK

You can also run the CPAN shell manually with perl-MCPAN-e shell.

If Test::Simple had any dependencies (it doesn't), the shell would have detected them and tried to install them first.

If you haven't used the CPAN module before, it will prompt you for all sorts of information about your machine and network configuration as well as your installation preferences. Usually the defaults are fine.

Section 1.1.1.2: Through PPM

By far, most Windows Perl installations use ActiveState's Active Perl distribution (http://www.activestate.com/Products/ActivePerl/), which includes the ppm utility to download, configure, build, and install modules. With ActivePerl installed, open a console window and type:

    C:\>PPM
    PPM> install Test-Simple

ActivePerl also has distributions for Linux and Solaris, so these instructions also work there

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Running Tests

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Before you can gain any benefit from writing tests, you must be able to run them. Fortunately, there are several ways to do this, depending on what you need to know.

To see real tests in action, download the latest version of Test::Harness (see http://search.cpan.org/dist/Test-Harness) from the CPAN and extract it to its own directory. Change to this directory and build the module as usual (see "Installing Test Modules," earlier in this chapter). To run all of the tests at once, type make test:

    $ make test
    PERL_DL_NONLAZY=1 /usr/bin/perl5.8.6 "-MExtUtils::Command::MM" "-e" \
        "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
    t/00compile.........ok 1/5# Testing Test::Harness 2.46
    t/00compile.........ok
    t/assert............ok
    t/base..............ok
    t/callback..........ok
    t/harness...........ok
    t/inc_taint.........ok
    t/nonumbers.........ok
    t/ok................ok
    t/pod...............ok
    t/prove-globbing....ok
    t/prove-switches....ok
    t/strap-analyze.....ok
    t/strap.............ok
    t/test-harness......ok
            56/208 skipped: various reasons
    All tests successful, 56 subtests skipped.
    Files=14, Tests=551,  6 wallclock secs ( 4.52 cusr +  0.97 csys =  5.49 CPU)

make test is the third step of nearly every Perl module installation. This command runs all of the test files it can find through Test::Harness, which summarizes and reports the results. It also takes care of setting the paths appropriately for as-yet-uninstalled modules.

How do I run tests for distributions that don't use Makefile.PL?

make test comes from ExtUtils::MakeMaker, an old and venerable module. Module::Build is easier to use in some cases. If there's a Build.PL file, instead use the commands perl Build.PL, perl Build, and perl Build test. Everything will behave as described here.

How do I run tests individually?

Sometimes you don't want to run everything through make test, as it runs all of the tests for a distribution in a specific order. If you want to run a few tests individually, use

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Interpreting Test Results

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Perl has a wealth of good testing modules that interoperate smoothly through a common protocol (the Test Anything Protocol, or TAP) and common libraries (Test::Builder). You'll probably never have to write your own testing protocol, but understanding TAP will help you interpret your test results and write better tests.

All of the test modules in this book produce TAP output. Test:: Harness interprets that output. Think of it as a minilanguage about test successes and failures.

Save the following program to sample_output.pl:

    #!perl

    print <<END_HERE;
    1..9
    ok 1
    not ok 2
    #     Failed test (t/sample_output.t at line 10)
    #          got: '2'
    #     expected: '4'
    ok 3
    ok 4 - this is test 4
    not ok 5 - test 5 should look good too
    not ok 6 # TODO fix test 6
    # I haven't had time add the feature for test 6
    ok 7 # skip these tests never pass in examples
    ok 8 # skip these tests never pass in examples
    ok 9 # skip these tests never pass in examples
    END_HERE

Using Windows and seeing an error about END_HERE? Add a newline to the end of sample_output. pl, then read perldoc perlfaq8.

Now run it through prove (see "Running Tests," earlier in this chapter):

    $ prove sample_output.pl
    sample_output....FAILED tests 2, 5
        Failed 2/9 tests, 77.789 okay (less 3 skipped tests: 4 okay, 44.44%)
    Failed Test      Stat Wstat Total Fail  Failed  List of Failed
    ------------------------------------------------------------------------
    sample_output.pl                9    2  22.22%  2 5
    3 subtests skipped.
    Failed 1/1 test scripts, 0.00% okay. 2/9 subtests failed, 77.79% okay.

prove interpreted the output of the script as it would the output of a real test. In fact, there's no effective difference—a real test might produce that exact output.

The lines of the test correspond closely to the results. The first line of the output is the test plan. In this case, it tells the harness to plan to run 9 tests. The second line of the report shows that 9 tests ran, but two failed: tests 2 and 5, both of which start with

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Writing Your First Test

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This lab introduces the most basic features of Test::Simple, the simplest testing module. You'll see how to write your own test for a simple "Hello, world!"-style program.

Open your favorite text editor and create a file called hello.t. Enter the following code:

    #!perl

    use strict;
    use warnings;

    use Test::Simple tests => 1;

    sub hello_world
    {
        return "Hello, world!";
    }

    ok( hello_world() eq "Hello, world!" );

Save it. Now you have a simple Perl test file. Run it from the command line with prove:

    $ prove hello.t

You'll see the following output:

    hello....ok
    All tests successful.
    Files=1, Tests=1,  0 wallclock secs ( 0.09 cusr +  0.00 csys =  0.09 CPU)

hello.t looks like a normal Perl program; it uses a couple of pragmas to catch misbehavior as well as the Test::Simple module. It defines a simple subroutine. There's no special syntax a decent Perl programmer doesn't already know.

The first potential twist is the use of Test::Simple. By convention, all test files need a plan to declare how many tests you expect to run. If you run the test file with perl and not prove, you'll notice that the plan output comes before the test output:

    $ perl hello.t
    1..1
    ok 1

The other interesting piece is the ok() subroutine. It comes from Test::Simple and is the module's only export. ok() is very, very simple. It reports a passed or a failed test, depending on the truth of its first argument. In the example, if whatever hello_world() returns is equal to the string Hello, world!, ok() will report that the test has passed.

Anything that can go in an if statement is fair game for ok().

As the output shows, there's one test in the file, and it passed. Congratulations!

In some cases, the number of tests you run is important, so providing a real plan is a good habit to cultivate.

How do I avoid changing the plan number every time I add a test?

Writing 'no_plan' on the use line lets Test::Simple know that you're playing it by ear. In this case, it'll keep its own count of tests and report that you ran as many as you ran.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Loading Modules

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Most of the Perl testing libraries assume that you use them to test Perl modules. Modules are the building blocks of larger Perl programs and well-designed code uses them appropriately. Loading modules for testing seems simple, but it has two complications: how do you know you've loaded the right version of the module you want to test, and how do you know that you've loaded it successfully?

This lab explains how to test both questions, with a little help from Test::More.

Imagine that you're developing a module to analyze sentences to prove that so-called professional writers have poor grammar skills. You've started by writing a module named AnalyzeSentence that performs some basic word counting. Save the following code in your library directory as AnalyzeSentence.pm:

Perl is popular among linguists, so someone somewhere may be counting misplaced commas in Perl books.

    package AnalyzeSentence;

    use strict;
    use warnings;

    use base 'Exporter';

    our $WORD_SEPARATOR = qr/\s+/;
    our @EXPORT_OK      = qw( $WORD_SEPARATOR count_words words );

    sub words
    {
        my $sentence = shift;
        return split( $WORD_SEPARATOR, $sentence );
    }

    sub count_words
    {
        my $sentence = shift;
        return scalar words( $sentence );
    }

    1;

Besides checking that words() and count_words() do the right thing, a good test should test that the module loads and imports the two subroutines correctly. Save the following test file as analyze_sentence.t:

    #!perl

    use strict;
    use warnings;

    use Test::More tests => 5;

    my @subs = qw( words count_words );

    use_ok( 'AnalyzeSentence', @subs   );
    can_ok( _ _PACKAGE_ _, 'words'       );
    can_ok( _ _PACKAGE_ _, 'count_words' );

    my $sentence =
        'Queen Esther, ruler of the Frog-Human Alliance, briskly devours a
        monumental ice cream sundae in her honor.';

    my @words = words( $sentence );
    ok( @words =  = 17, 'words() should return all words in sentence' );

    $sentence = 'Rampaging ideas flutter greedily.';
    my $count = count_words( $sentence );

    ok( $count =  = 4, 'count_words() should handle simple sentences' );

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Improving Test Comparisons

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

ok() may be the basis of all testing, but it can be inconvenient to have to reduce every test in your system to a single conditional expression. Fortunately, Test::More provides several other testing functions that can make your work easier. You'll likely end up using these functions more often than ok().

This lab demonstrates how to use the most common testing functions found in Test::More.

The following listing tests a class named Greeter, which takes the name and age of a person and allows her to greet other people. Save this code as greeter.t:

    #!perl

    use strict;
    use warnings;

    use Test::More tests => 4;

    use_ok( 'Greeter' ) or exit;

    my $greeter = Greeter->new( name => 'Emily', age => 21 );
    isa_ok( $greeter, 'Greeter' );

    is(   $greeter->age(),   21,
        'age() should return age for object' );
    like( $greeter->greet(), qr/Hello, .+ is Emily!/,
        'greet() should include object name' );

The examples in "Writing Your First Test," earlier in this chapter, will work the same way if you substitute Test::More for Test::Simple; Test::More is a superset of Test:: Simple.

Now save the module being tested in your library directory as Greeter.pm:

    package Greeter;

    sub new
    {
        my ($class, %args) = @_;
        bless \%args, $class;
    }

    sub name
    {
        my $self = shift;
        return $self->{name};
    }

    sub age
    {
        my $self = shift;
        return $self->{age};
    }

    sub greet
    {
        my $self = shift;
        return "Hello, my name is " . $self->name() . "!";
    }

    1;

Running the file from the command line with prove should reveal three successful tests:

    $ prove greeter.t
    greeter.t....ok
    All tests successful.
    Files=1, Tests=4,  0 wallclock secs ( 0.07 cusr +  0.03 csys =  0.10 CPU)

This program starts by loading the Greeter module and creating a new Greeter object for Emily, age 21. The first test checks to see if the constructor returned an actual Greeter object. isa_ok() performs several checks to see if the variable is actually a defined reference, for example. It fails if it is an undefined value, a non-reference, or an object of any class other than the appropriate class or a derived class.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Chapter 2: Writing Tests

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Perl has a rich vocabulary, but you can accomplish many things using only a fraction of the power available. In the same way, Perl has an ever-increasing number of testing modules and best practices built around the simple ok() function described in Chapter 1.

The labs in this chapter guide you through the advanced features of Test::More and other commonly used testing modules. You'll learn how to control which tests run and why, how to compare expected and received data effectively, and how to test exceptional conditions. These are crucial techniques that provide the building blocks for writing comprehensive test suites.

Some tests should run only under certain conditions. For example, a network test to an external service makes sense only if an Internet connection is available, or an OS-specific test may run only on a certain platform. This lab shows how to skip tests that you know will never pass.

Suppose that you're writing an English-to-Dutch translation program. The Phrase class stores some text and provides a constructor, an accessor, and an as_dutch() method that returns the text translated to Dutch.

Save the following code as Phrase.pm:

    package Phrase;
    use strict;

    sub new
    {
        my ( $class, $text ) = @_;
        bless \$text, $class;
    }

    sub text
    {
        my $self = shift;
        return $$self;
    }

    sub as_dutch
    {
        my $self = shift;
        require WWW::Babelfish;
        return WWW::Babelfish->new->translate(
            source      => 'English',
            destination => 'Dutch',
            text        => $self->text(),
        );
    }

    1;

A user may or may not have the WWW::Babelfish translation module installed. That's fine; you've decided that Phrase's as_dutch() feature is optional. How can you test that, though?

Save the following code as phrase.t:

    #!perl

    use strict;

    use Test::More tests => 3;
    use Phrase;

    my $phrase = Phrase->new('Good morning!');
    isa_ok( $phrase, 'Phrase' );

    is( $phrase->text(), 'Good morning!', "text() access works" );

    SKIP:
    {
        eval 'use WWW::Babelfish';

        skip( 'because WWW::Babelfish required for as_dutch()', 1 ) if $@;

        is(
            $phrase->as_dutch,
            'Goede ochtend!',
            "successfully translated to Dutch"
          );
    }

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Skipping Tests

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Save the following code as Phrase.pm:

    package Phrase;
    use strict;

    sub new
    {
        my ( $class, $text ) = @_;
        bless \$text, $class;
    }

    sub text
    {
        my $self = shift;
        return $$self;
    }

    sub as_dutch
    {
        my $self = shift;
        require WWW::Babelfish;
        return WWW::Babelfish->new->translate(
            source      => 'English',
            destination => 'Dutch',
            text        => $self->text(),
        );
    }

    1;

A user may or may not have the WWW::Babelfish translation module installed. That's fine; you've decided that Phrase's as_dutch() feature is optional. How can you test that, though?

Save the following code as phrase.t:

    #!perl

    use strict;

    use Test::More tests => 3;
    use Phrase;

    my $phrase = Phrase->new('Good morning!');
    isa_ok( $phrase, 'Phrase' );

    is( $phrase->text(), 'Good morning!', "text() access works" );

    SKIP:
    {
        eval 'use WWW::Babelfish';

        skip( 'because WWW::Babelfish required for as_dutch()', 1 ) if $@;

        is(
            $phrase->as_dutch,
            'Goede ochtend!',
            "successfully translated to Dutch"
          );
    }

Run the test file with prove in verbose mode. If you have WWW::Babelfish installed, you will see the following output:

    $ prove -v phrase.t
    phrase....1..3
    ok 1 - The object isa Phrase
    ok 2 - text() access works
    ok 3 - successfully translated to Dutch
    ok
    All tests successful.
    Files=1, Tests=3,  1 wallclock secs ( 0.16 cusr +  0.01 csys =  0.17 CPU)

If you run the test

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Skipping All Tests

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The preceding lab demonstrated how to skip certain tests under certain conditions. You may find cases where an entire test file shouldn't run—for example, when testing platform X-specific features on platform Y will produce no meaningful results. Test::More provides a bit of useful syntax for this situation.

Use the plan function on its own instead of specifying the tests in the use() statement. The following code checks to see if the current weekday is Tuesday. If it is not, the test will skip all of the tests. Save it as skip_all.t:

    use Test::More;

    if ( [ localtime ]->[6] != 2 )
    {
        plan( skip_all => 'only run these tests on Tuesday' );
    }
    else
    {
        plan( tests => 1 );
    }

    require Tuesday;
    my $day = Tuesday->new();
    ok( $day->coat(), 'we brought our coat' );

Tuesday.pm is very simple:

    package Tuesday;

    sub new
    {
        bless {  }, shift;
    }

    # wear a coat only on Tuesday
    sub coat
    {
        return [ localtime ]->[6] =  = 2;
    }

    1;

Run this test file on a Tuesday to see the following output:

    $ prove -v skip_all.t
    chapter_01/skipping_all_tests....1..1
    ok 1 - we brought our coat
    ok
    All tests successful.
    Files=1, Tests=1,  1 wallclock secs ( 0.13 cusr +  0.04 csys =  0.17 CPU)

A real test file would have more tests; this is just an example.

Run it on any other day of the week to skip all of the tests:

    $ prove -v skip_all.t
    chapter_01/skipping_all_tests....1..0 # Skip only run these tests on Tuesday skipped
            all skipped: only run these tests on Tuesday
    All tests successful, 1 test skipped.
    Files=1, Tests=0,  0 wallclock secs ( 0.14 cusr +  0.05 csys =  0.19 CPU)

Instead of immediately reporting the test plan by passing extra arguments to the use keyword, skip_all.t uses Test::More's plan() function to determine the test plan when the script runs. If the current weekday is not Tuesday, the code calls plan() with two arguments: an instruction to run no tests and a reason why. If it is Tuesday, the code reports the regular test plan and execution continues as normal.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Marking Tests as TODO

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Even though having a well-tested codebase can increase your development speed, you may still have more features to add and bugs to fix than you can program in the current session. It can be useful to capture this information in tests, though they'll obviously fail because there's no code yet! Fortunately, you can mark these tasks as executable, testable TODO items that your test harness will track for you until you get around to finishing them.

Take a good idea for some code: a module that reads future versions of files. That will be really useful. Call it File::Future, and save the following code to File/Future.pm, creating the File/ directory first if necessary:

    package File::Future;

    use strict;

    sub new
    {
        my ($class, $filename) = @_;
        bless { filename => $filename }, $class;
    }

    sub retrieve
    {
        # implement later...
    }

    1;

The File::Future constructor takes a file path and returns an object. Calling retrieve() on the object with a date retrieves that file at the given date. Unfortunately, there is no Perl extension to flux capacitors yet. For now, hold off on writing the implementation of retrieve().

There's no sense in not testing the code, though. It'll be nice to know that the code does what it needs to do by whatever Christmas Acme::FluxFS finally appears. It's easy to test that. Save the following code as future.t:

    use Test::More tests => 4;
    use File::Future;

    my $file = File::Future->new( 'perl_testing_dn.pod' );
    isa_ok( $file, 'File::Future' );

    TODO: {
        local $TODO = 'continuum not yet harnessed';

        ok( my $current = $file->retrieve( 'January 30, 2005' ) );
        ok( my $future  = $file->retrieve( 'January 30, 2070' ) );

        cmp_ok( length($current), '<', length($future),
            'ensuring that we have added text by 2070' );
    }

Run the test with prove. It will produce the following output:

    $ prove -v future.t
    future.t....1..4
    ok 1 - The object isa File::Future
    not ok 2 # TODO continuum not yet harnessed
    #     Failed (TODO) test (future.t.pl at line 14)
    not ok 3 # TODO continuum not yet harnessed
    #     Failed (TODO) test (future.t.pl at line 15)
    not ok 4 - ensuring that we have added text by 2070 # TODO
          continuum not yet harnessed
    #     Failed (TODO) test (future.t at line 13)
    #     '0'
    #         <
    #     '0'
    ok
    All tests successful.
    Files=1, Tests=4,  0 wallclock secs ( 0.02 cusr +  0.00 csys =  0.02 CPU)

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Simple Data Structure Equality

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Test::More's is() function checks scalar equality, but what about more complicated structures, such as lists of lists of lists? Good tests often need to peer into these data structures to test whether, deep down inside, they are truly equal. The first solution that may come to mind is a recursive function or a series of nested loops. Hold that thought, though—Test::More and other test modules provide a better way with their comparison functions.

Save this code as deeply.t:

    use Test::More tests => 1;

    my $list1 =
    [
        [
            [ 48, 12 ],
            [ 32, 10 ],
        ],
        [
            [ 03, 28 ],
        ],
    ];

    my $list2 =
    [
        [
            [ 48, 12 ],
            [ 32, 11 ],
        ],
        [
            [ 03, 28 ],
        ],
    ];

    is_deeply( $list1, $list2, 'existential equivalence' );

Run it with prove -v to see the diagnostics:

    $ prove -v deeply.t
    deeply....1..1
    not ok 1 - existential equivalence
    #     Failed test (deeply.t at line 23)
    #     Structures begin differing at:
    #          $got->[0][1][1] = '10'
    #     $expected->[0][1][1] = '11'
    # Looks like you failed 1 tests of 1.
    dubious
      Test returned status 1 (wstat 256, 0x100)
    DIED. FAILED test 1
      Failed 1/1 tests, 0.00% okay
    Failed 1/1 test scripts, 0.00% okay. 1/1 subtests failed, 0.00% okay.
    Failed Test Stat Wstat Total Fail  Failed  List of Failed
    ---------------------------------------------------------------------------
    deeply.t       1   256     1    1 100.00%  1

The example test compares two lists of lists with the is_deeply() function exported by Test::More. Note the difference between the two lists. Because the first array contains a 10 where the second contains an 11, the test failed.

The output shows the difference between $list1 and $list2. If there are multiple differences in the data structure, is_deeply() will display only the first. Additionally, if one of the data structures is missing an element, is_deeply() will show that as well.

How do I see the differences, but not the similarities, between data structures in my test output?

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Data Composition

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

As the data structures your code uses become more complex, so will your tests. It's important to verify what actually makes up a data structure instead of simply comparing it to an existing structure. You could iterate through each level of a complex nested hash of arrays, checking each and every element. Fortunately, the Test::Deep module neatens up code testing complicated data structures and provides sensible error messages.

Save the following as cmp_deeply.t:

    use Test::More tests => 1;
    use Test::Deep;

    my $points =
    [
        { x => 50, y =>  75 },
        { x => 19, y => -29 },
    ];

    my $is_integer = re('^-?\d+$');

    cmp_deeply( $points,
      array_each(
        {
          x => $is_integer,
          y => $is_integer,
        }
      ),
            'both sets of points should be integers' );

Now run cmp_deeply.t from the command line with prove. It will show one successful test:

    $ prove cmp_deeply.t
    cmp_deep....ok
    All tests successful.
    Files=1, Tests=1,  0 wallclock secs ( 0.06 cusr +  0.00 csys =  0.06 CPU)

cmp_deeply(), like most other testing functions, accepts two or three arguments: the data structure to test, what you expect the structure to look like, and an optional test description. The expected data, however, is a special test structure with a format containing special Test::Deep functions.

The test file begins by creating a regular expression using re(), a function exported by Test::Deep. re() declares that the data must match the given regular expression. If you use a regular expression reference instead, Test::Deep believes you expect the data to be a regular expression instead of matching the data against it.

re() also lets you perform checks on the data it matches. See the Test::Deep documentation for details.

Test::Deep's array_each() function creates the main test structure for the test. To pass the test, $points must be an array reference. Every element of the array must validate against the test structure passed to array_each().

Passing a hash reference as the test structure declares that every element must be a hash reference and the values of the given hash must match the values in the test structure's hash. In

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Testing Warnings

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The only parts of your code that don't need tests are those parts that you don't need. If your code produces warnings in certain circumstances and they're important to you, you need to test that they occur when and only when you expect them. The Test::Warn module provides helpful test functions to trap and examine warnings.

Save the following code as warnings.t:

    use Test::More tests => 4;
    use Test::Warn;

    sub add_positives
    {
        my ( $l, $r ) = @_;
        warn "first argument ($l) was negative"  if $l < 0;
        warn "second argument ($r) was negative" if $r < 0;
        return $l + $r;
    }

    warning_is { is( add_positives( 8, -3 ), 5 ) }
        "second argument (-3) was negative";

    warnings_are { is( add_positives( -8, -3 ), -11 ) }
        [
            'first argument (-8) was negative',
            'second argument (-3) was negative'
        ];

There are no commas between the first and second arguments to any of Test:: Warn's test functions because their prototypes turn normallooking blocks into subroutine references.

Run the file with prove to see the following output:

    $ prove -v warnings.t
    warnings....1..4
    ok 1
    ok 2
    ok 3
    ok 4
    ok
    All tests successful.
    Files=1, Tests=4,  0 wallclock secs ( 0.04 cusr +  0.00 csys =  0.04 CPU)

The test file declares and tests a trivial function, add_positives(). The function adds two numbers together and warns the user if either number is less than zero.

warning_is() takes a block of code to run and the text of the warning expected. Like most other test functions, it takes an optional third argument as the test description. Passing two less-than-zero arguments to add_positives() causes the subroutine to produce two warnings. To test for multiple warnings, use Test::Warn's warnings_are(). Instead of a single string, warnings_are() takes a reference to an array of complete warning strings as its second argument.

What if the warning I'm trying to match isn't an exact string?

Test::Warn also exports warning_like()

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Testing Exceptions

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Sometimes things go wrong. That's okay; sometimes the best thing to do in code that detects an unrecoverable error is to pitch a fit and let higher-level code figure out what to do. If you do that, though, you need to test that behavior. As usual, there's a module to make this easy. Test::Exception provides the functions to test that a block of code throws (or doesn't throw) the exceptions that you expect.

Suppose that you're happy with add_positives() from "Testing Warnings," but your coworkers can't seem to use it correctly. They happily pass in negative numbers and ignore the warnings, and then blame you when their code fails to work properly. Your team lead has suggested that you strengthen the function to hate negative numbers—so much so that it throws an exception if it encounters one. How can you test that?

Save the following listing as exception.t:

    use Test::More tests => 3;
    use Test::Exception;
    use Error;

    sub add_positives
    {
        my ($l, $r) = @_;
        throw Error::Simple("first argument ($l) was negative")  if $l < 0;
        throw Error::Simple("second argument ($r) was negative") if $r < 0;
        return $l + $r;
    }

    throws_ok { add_positives( -7,  6 ) } 'Error::Simple';
    throws_ok { add_positives(  3, -9 ) } 'Error::Simple';
    throws_ok { add_positives( -5, -1 ) } 'Error::Simple';

There are no commas between the first and second arguments to any of Test:: Exception's test functions.

Run the file with prove:

    $ prove -v exception.t
    exception....1..3
    ok 1 - threw Error::Simple
    ok 2 - threw Error::Simple
    ok 3 - threw Error::Simple
    ok
    All tests successful.
    Files=1, Tests=3,  0 wallclock secs ( 0.03 cusr +  0.00 csys =  0.03 CPU)

The call to throws_ok() ensures that add_positives() throws an exception of type Error::Simple. throws_ok() performs an isa() check on the exceptions it catches, so you can alternatively specify any superclass of the exception thrown. For example, because exceptions inherit from the Error class, you can replace all occurrences of Error::Simple in

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Chapter 3: Managing Tests

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

All the normal rules of programming apply to tests: stay organized, reduce duplication, and don't take on more technical debt than you need. For small projects, it's easy to create and manage single test files. Large or important projects need more thought and care. Where do you put your tests? How do you organize them between files? What options do you have to reduce complexity to manageable levels?

This chapter's labs explain how to organize your test files into test suites, know and improve the reach of your tests, write your own custom testing libraries, and interpret test results.

Writing tests is easy. Managing tests well is more difficult. Having complete test coverage is worthless if running the complete test suite is so difficult that no one ever does it. Making your tests easy to run without user intervention and making it easy to interpret their results will pay off over and over again.

Using the standard testing tools that understand the Test Anything Protocol is just one part of the process. Organizing your tests sensibly is another.

Consider the tests for the Test::Harness module. Download the latest distribution from the CPAN and extract it. Change into the newly created directory, run Makefile.PL, and build and test the module:

Look for the Download link at http://search.cpan.org/dist/Test-Harnes/ .

    $ perl Makefile.PL
    Checking if your kit is complete...
    Looks good
    Writing Makefile for Test::Harness
    $ make
    cp lib/Test/Harness/TAP.pod blib/lib/Test/Harness/TAP.pod
    cp lib/Test/Harness/Iterator.pm blib/lib/Test/Harness/Iterator.pm
    cp lib/Test/Harness/Assert.pm blib/lib/Test/Harness/Assert.pm
    cp lib/Test/Harness.pm blib/lib/Test/Harness.pm
    cp lib/Test/Harness/Straps.pm blib/lib/Test/Harness/Straps.pm
    cp bin/prove blib/script/prove
    /usr/bin/perl5.8.6 "-MExtUtils::MY" -e "MY->fixin(shift)" blib/script/prove
    <output snipped>
    $ make test
    PERL_DL_NONLAZY=1 /usr/bin/perl5.8.6 "-MExtUtils::Command::MM" "-e"
            "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
    <output snipped>

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Organizing Tests

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Using the standard testing tools that understand the Test Anything Protocol is just one part of the process. Organizing your tests sensibly is another.

Look for the Download link at http://search.cpan.org/dist/Test-Harnes/ .

    $ perl Makefile.PL
    Checking if your kit is complete...
    Looks good
    Writing Makefile for Test::Harness
    $ make
    cp lib/Test/Harness/TAP.pod blib/lib/Test/Harness/TAP.pod
    cp lib/Test/Harness/Iterator.pm blib/lib/Test/Harness/Iterator.pm
    cp lib/Test/Harness/Assert.pm blib/lib/Test/Harness/Assert.pm
    cp lib/Test/Harness.pm blib/lib/Test/Harness.pm
    cp lib/Test/Harness/Straps.pm blib/lib/Test/Harness/Straps.pm
    cp bin/prove blib/script/prove
    /usr/bin/perl5.8.6 "-MExtUtils::MY" -e "MY->fixin(shift)" blib/script/prove
    <output snipped>
    $ make test
    PERL_DL_NONLAZY=1 /usr/bin/perl5.8.6 "-MExtUtils::Command::MM" "-e"
            "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
    <output snipped>

Until now, all of the examples have mixed code and tests in the same file. That's fine for teaching, but it won't work as well in production code. There's no technical reason to keep all of the tests for a particular program or module in a single file, so create as many test files as you need, organizing them by features, bugs, modules, or any other criteria.

The only technical requirement when using separate test files is that the files must be able to load the modules they test. That means you must manage Perl's library paths appropriately. Fortunately, most good CPAN modules handle this. The magic of making these tests work is the magic of Perl module installation tools such as

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Checking Your Coverage

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Having some tests is better than having no tests, but having enough tests is better yet. Code coverage is one way to measure how much of the code the tests actually test. Analyzing code coverage by hand is tedious. Fortunately, the Devel::Cover module from the CPAN automates the analysis and reporting for you. Best of all, it works with the standard Perl test harness.

Install Devel::Cover and its dependencies (see "Installing Test Modules" in Chapter 1). You need the ability to build XS modules, unless you install it via ppm or some other binary package.

XS is the Perl extension system. It allows the use of code written in languages other than Perl and requires a working C development environment.

From the top level of a module directory, such as Test::Harness (see "Organizing Tests," earlier in this chapter), build the module, and then run the following commands:

If your module uses Module:: Build, use perl Build testcover instead of make test. Otherwise, install ExtUtils:: MakeMaker:: Coverage and use make testcover.

    $ cover -delete
    Deleting database /home/chromatic/dev/install/Test-Harness-2.46/cover_db
    $ HARNESS_PERL_SWITCHES=-MDevel::Cover make test
    PERL_DL_NONLAZY=1 /usr/bin/perl5.8.6 "-MExtUtils::Command::MM" "-e"
    "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
    t/00compile.........ok 1/5# Testing Test::Harness 2.46
    t/00compile.........ok
    t/assert............ok
    t/base..............ok
    t/callback..........ok
    t/harness...........ok
    t/inc_taint.........ok
    t/nonumbers.........ok
    t/ok................ok
    t/pod...............ok
    t/prove-globbing....ok
    t/prove-switches....ok
    t/strap-analyze.....ok
    t/strap.............ok
    t/test-harness......ok
            56/208 skipped: various reasons
    All tests successful, 56 subtests skipped.
    Files=14, Tests=551, 255 wallclock secs
        (209.59 cusr +  4.98 csys = 214.57 CPU)

    $ cover
    Reading database from /home/chromatic/dev/Test-Harness-2.46/cover_db


    ---------------------------------- ------ ------ ------ ------ ------ ------
    File                               stmt branch   cond    sub   time  total
    ---------------------------------- ------ ------ ------ ------ ------ ------
    blib/lib/Test/Harness.pm             71.6   51.6   61.1   80.8    0.0   65.9
    blib/lib/Test/Harness/Assert.pm     100.0  100.0    n/a  100.0    0.0  100.0
    blib/lib/Test/Harness/Iterator.pm    70.0   25.0    n/a   80.0   98.9   65.5
    blib/lib/Test/Harness/Straps.pm      92.9   82.7   69.0   96.2    1.0   87.6
    Total                                80.8   66.0   65.4   88.3  100.0   76.0
    ---------------------------------- ------ ------ ------ ------ ------ ------


    Writing HTML output to /home/chromatic/dev/Test-Harness-2.46/cover_db/coverage.html ...
    done.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Writing a Testing Library

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

By now, tests should seem less daunting. They're just programs; all of the normal good advice about design and organization applies. It makes sense to keep related code in the same place. In Perl terms, this is a module: a self-contained library of code.

Previous labs have demonstrated how to use several testing libraries. Some come with Perl's standard library. Others live on the CPAN, with new modules released frequently. If you find yourself solving the same testing problem repeatedly by twisting existing test modules in new directions, consider writing your own testing library. Test::Builder makes it possible.

The following example implements one function of a very simple testing library. It adds one new function, is_between(), that tests whether a given value is between two other values. Save the code under a lib/ directory where you can reach it (see "Installing Test Modules," in Chapter 1) as Test/Between.pm:

The use of subroutine prototypes is a convention in testing modules, but they're not like subroutine signatures in other languages. See the Prototypes section in perldoc perlsub for more information.

    package Test::Between;

    use strict;
    use warnings;

    use base 'Exporter';

    our@EXPORT = qw( is_between );

    use Test::Builder;
    my $Test = Test::Builder->new();

    sub is_between ($$$;$)
    {
        my ($item, $lower, $upper, $desc) = @_;

        return
        (
            $Test->ok( "$lower" le "$item" && "$item" le "$upper", $desc ) ||
            $Test->diag( "        $item is not between $lower and $upper" )
        );
    }

    1;

Now you can use it within your own test programs. Save the following code as test_between.t:

    #!perl

    use strict;
    use warnings;

    use Test::More tests => 3;

    use Test::Between;

    is_between(    'b', 'a', 'c', 'simple alphabetical comparison' );
    is_between(     2 ,  1 ,  3 , 'simple numeric comparison'      );
    is_between(  "two",  1 ,  3 , 'mixed comparison'               );

By design, Test:: Between doesn't allow its users to set the plan. Why reinvent the wheel when it's likely that users will use the module with Test:: Simple or Test:: More anyway?

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Testing a Testing Library

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Test::Builder makes writing custom testing libraries easy (see the previous lab, "Writing a Testing Library") by handling all of the distracting test bookkeeping and management. They're just code. Good libraries need good tests, though.

Fortunately, using Test::Builder makes writing tests for these custom libraries easier too, with a little help from Test::Builder::Tester.

Consider a test suite for Test::Between (from "Writing a Testing Library"). Save the following test file as between.t:

    #!perl

    use strict;
    use warnings;

    use Test::Between;
    use Test::Builder::Tester tests => 3;

    my $desc;

    $desc = 'simple alphabetical comparison';
    test_pass( $desc );
    is_between( 'b', 'a', 'c',  $desc );
    test_test( $desc );

    $desc = 'simple numeric comparison';
    test_pass( $desc );
    is_between(  2,  1,  3, $desc );
    test_test( $desc );

    $desc = 'mixed comparison';
    test_out( "not ok 1 - $desc" );
    test_fail( +2 );
    test_diag( '        two is not between 1 and 3' );
    is_between(  "two",  1,  3, $desc               );
    test_test( 'failed comparison with diagnostics' );

The $desc variable appears multiple times so as to avoid copying and pasting the test description multiple times. Avoid repetition in tests as you would in any other code.

Run it with perl:

    $ perl between.t
    1..3
    ok 1 - simple alphabetical comparison
    ok 2 - simple numeric comparison
    ok 3 - failed comparison with diagnostics

between.t looks almost like any other test that uses Test::Between except for one twist: instead of using Test::More to declare a test plan, it uses Test::Builder::Tester, which provides its own test plan. From there, it has three blocks of tests that correspond to the tests shown in "Writing a Testing Library"--an alphabetical comparison that should pass, a numeric comparison that should also pass, and a mixed comparison that should fail.

Test::Builder::Tester works by collecting information about what a test should do, running the test, and comparing its actual output to the expected output. Then it reports the results. This requires you to know if the test should pass or fail and what kind of output it will produce.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Writing a Testing Harness

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

TAP is a simple protocol (see "Interpreting Test Results" in Chapter 1), but you shouldn't have to write your own parser when Test::Harness already knows how to interpret the results. However, Test::Harness only prints out what it discovers.

Test::Harness uses Test::Harness:: Straps internally

Test::Harness::Straps is a thin wrapper around a TAP parser. It collects the results in a data structure but does not analyze or print them. Writing a program to report those results in an alternate format is easy. If you want to do something when tests fail, or if you want to do something more complicated than simply reporting test results, why not write your own testing harness?

Save the following program somewhere in your path as new_harness.pl and make it executable:

    #!perl

    use strict;
    use warnings;


    use Test::Harness::Straps;
    my $strap = Test::Harness::Straps->new();

    for my $file (@ARGV)
    {
        next unless -f $file;
        my %results = $strap->analyze_file( $file );
        printf <<END_REPORT, $file, @results{qw( max seen ok skip todo bonus )};
    Results for %s
        Expected tests:    %d
        Tests run:         %d
        Tested passed:     %d
        Tests skipped:     %d
        TODO tests:        %d
        TODO tests passed: %d
    END_REPORT
    }

Run it on a directory full of tests (the Test::Harness suite, for example):

    $ new_harness t/strap*t
    Results for t/strap-analyze.t
        Expected tests:    108
        Tests run:         108
        Tested passed:     108
        Tests skipped:     0
        TODO tests:        0
        TODO tests passed: 0
    Results for t/strap.t
        Expected tests:    176
        Tests run:         176
        Tested passed:     176
        Tests skipped:     0
        TODO tests:        0
        TODO tests passed: 0

Your shell should expand the file pattern t/strap*. t to include only the straps tests shown in the output.

The first few lines start the program as normal, loading a few modules and pragmas and creating a new

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Testing Across the Network

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Test::Harness::Straps makes writing custom test harnesses easy, but it's more flexible than you might think. Its input can come from anywhere. Have you ever wanted to run tests on a remote machine and summarize their output locally? That's no problem.

Save the following code as network_harness.pl:

    use Net::SSH::Perl;
    use Test::Harness::Straps;

    my $strap   = Test::Harness::Straps->new();
    my $ssh     = Net::SSH::Perl->new( 'testbox' );
    $ssh->login(qw( username password ));

    my ($stdout, $stderr, $exit) = $ssh->cmd( 'runtests' );
    my %results = $strap->analyze_fh( 'testbox tests', $stdout );

    # parse %results as normal

The first argument to analyze_fh() is the test's name, corresponding to the test file name used with analyze_file().

Suppose that you have code running on a machine named testbox. You have access to that machine via SSH, and you have a program on that machine called runtests that knows how to run tests for your application. Run network_harness.pl as a normal Perl program and it will gather and parse the output from testbox, reporting the results.

The harness connects to the testbox machine through SSH by using the provided username and password. Then it issues the runtests command to the remote machine, collects the results, and passes the output of the command to the TAP parser object. From there, do whatever you like with the results.

Does the other machine have to have Perl running?

No, it can use any other language as long as it produces TAP output.

What if you don't want to or are unable to read from a socket on the remote machine?

Put the test output into an array of lines, perhaps by reading it from a web page on a remote server, and then use the analyze() method:

    use LWP::Simple;
    use Test::Harness::Straps;

    my $strap   = Test::Harness::Straps->new();
    my $output  = get( 'http://testbox/tests/smoketest.t' );
    my @lines   = split( /\n/, $output );
    my %results = $strap->analyze( 'testbox smoketest', \@lines );

    # parse %results as normal

The only trick to this example is that

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Automating Test Runs

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Improving code quality is the primary benefit of writing a large test suite, but there are several other benefits, such as encouraging more careful coding and better design. Well-written tests provide feedback on the state of the project. At any point, anyone can run the tests to find out what works and what has broken.

This is valuable enough that, besides encouraging developers to run the test suite at every opportunity while developing, many projects automate their test suites to run unattended at regular intervals, reporting any failures. This smoketesting is highly valuable, as it