Chapter 4. Writing Unit Tests

The previous chapters present a simple unit test framework and the fundamentals of xUnit. The unit test framework’s architecture is important to understand, but not something you have to think about often. Most of your time should be spent writing unit tests, implementing production code to make the tests pass, or refactoring. This chapter includes examples of common patterns used when writing unit tests, as well as related tips on unit test development.

The code examples in this chapter are unit tests of additional virtual library functionality, including looking up books by author and title, looking up multiple books by one author, and removing books from the library. The Library and Book code to implement the new features is given at the end of the chapter.

Types of Asserts

The code examples shown so far use plain asserts . These are the most generic type of test assertion, which take a Boolean condition that must evaluate to TRUE for the test to succeed. A plain assert, the unit test for the Library method removeBook( ), is shown in Example 4-1.

Example 4-1. Test method testRemoveBook( ) using a plain assert
LibraryTest.java
   public void testRemoveBook( ) {
      library.removeBook( "Dune" );
      Book book = library.getBook( "Dune" );
      assertTrue( book == null );
   }

If the method removeBook( ) is stubbed out, the test fails. The following test results report the failure:

> java junit.textui.TestRunner LibraryTests
.....F.
Time: 0.06
There was 1 failure:
1) testRemoveBook(LibraryTest)junit.framework.AssertionFailedError
     at LibraryTest.testRemoveBook(LibraryTest.java:32)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

FAILURES!!!
Tests run: 6,  Failures: 1,  Errors: 0

Although the line of code where the failure occurred is shown, the output does not describe the specific cause of the failure. It often is helpful to add an informative message to the assertion. The xUnits generally have two versions of every assert method, one of which takes a message parameter describing the assert. Example 4-2 shows the test method using an assert with a message.

Example 4-2. Test method using an assert with a message
LibraryTest.java
   public void testRemoveBook( ) {
      library.removeBook( "Dune" );
      Book book = library.getBook( "Dune" );
      assertTrue( "book is not removed", book == null );
   }

With the additional message, the rest results provide better information about the cause of the test failure:

1) testRemoveBook(LibraryTest)junit.framework.AssertionFailedError: 
book is not removed

Although all assert conditions ultimately must evaluate to a Boolean result of TRUE or FALSE, it can be tedious to constantly reduce every expression to this form. The xUnits offer a variety of assert functions to help. Examples of several of the assert methods from JUnit are as follows:

assertFalse( book == null );
assertFalse( "book is null", book == null );
assertNull( book );
assertNull( "book is not null", book );
assertNotNull( book );
assertNotNull( "book is null", book );
assertEquals( "Solaris", book.title );
assertEquals( "unexpected book title", "Solaris", book.title );

These assert methods all have variants that take a message parameter to describe the failure, as shown above. The assertEquals() method has variants that take different data types as arguments.

Defining Custom Asserts

The basic assert methods cover only a few common cases. It’s often useful to extend them to cover additional test conditions and data types. Custom assert methods save test coding effort and make the test code more readable.

So far, the Library tests check a Book’s title attribute to verify the expected Book object, as shown in Example 4-3 in the test method testGetBooks( ).

Example 4-3. Test comparing two Books using their title attributes
LibraryTest.java

   public void testGetBooks( ) {
      Book book = library.getBook( "Dune" );
      assertTrue( book.getTitle( ).equals( "Dune" ) );
      book = library.getBook( "Solaris" );
      assertTrue( book.getTitle( ).equals( "Solaris" ) );
   }

To be really sure that the test Book is correct, the tests should also check the Book’s author, but this means adding extra asserts to each test. It’s clearly useful to have an assert method that compares an expected Book to the actual Book, checking all of the attributes. This new assert method is easy to implement by building on the generic assertTrue() method, as shown in Example 4-4.

Example 4-4. Custom assert method to compare Books
BookTest.java

public class BookTest extends TestCase {

   public static void assertEquals( Book expected, Book actual ) {
      assertTrue(expected.getTitle( ).equals( actual.getTitle( ) )
              && expected.getAuthor( ).equals( actual.getAuthor( ) ));
   }
}

The assert method assertEquals() takes expected and actual Book objects to compare. It succeeds if the title and author attributes of the two Books are equal. Example 4-5 shows how it is used.

Example 4-5. Using the custom assert method
LibraryTest.java
public class LibraryTest extends TestCase {

   private Library library;
   private Book book1, book2;

   public void setUp( ) {
      library = new Library( );
      book1 = new Book("Dune", "Frank Herbert");
      book2 = new Book("Solaris", "Stanislaw Lem");
      library.addBook( book1 );
      library.addBook( book2 );
   }

   public void testGetBooks( ) {
      Book book = library.getBook( "Dune" );
      BookTest.assertEquals( book1, book );
      book = library.getBook( "Solaris" );
      BookTest.assertEquals( book2, book );
   }
}

The custom assert method makes the test clear and concise and improves it by comparing all the Book attributes, not just the title. While writing tests, watch for complex assert conditions that are used repeatedly. They are good candidates for replacement with custom assert methods.

Single Condition Tests

A useful rule of thumb is that a test method should only contain a single test assertion. The idea is that a test method should only test one behavior; if there is more than one assert condition, multiple things are being tested. When there is more than one condition to test, then a test fixture should be set up, and each condition placed in a separate test method.

The xUnits tend to enforce this rule when handling test assertion failures. A test method returns as soon as a failure occurs, skipping any additional code. Running the rest of the test is unnecessary, since the result (failure) is known.

Practically speaking, test methods containing several assertions are not always a terrible thing. Tests may have conditions that can only be combined into one expression with unnecessary complication of the code. The testGetBooks( ) method in the previous section verifies that the Library contains two Books, which is most clearly expressed as two separate asserts, although they could be combined into one compound condition. A single behavior can have several side effects that you should check with separate assertions. So, it’s not a problem when a test method contains several asserts, as long as the test method is only testing a single behavior.

However, a test method with many asserts is a clear indicator that a single test is doing too much. Example 4-6 shows a test method with this problem.

Example 4-6. Poorly written unit test that tests multiple behaviors
LibraryTest.java
   public void testLookupBooksByAuthor( ) {
      // Add two books by same author
      Book book3 = new Book( "Cosmos", "Carl Sagan" );
      Book book4 = new Book( "Contact", "Carl Sagan" );
      library.addBook( book3 );
      library.addBook( book4 );
      // Look up books by title and author
      Book book = library.getBook( "Cosmos", "Carl Sagan" );
      BookTest.assertEquals( book3, book );
      book = library.getBook( "Contact", "Carl Sagan" );
      BookTest.assertEquals( book4, book );
      // Look up both books by author
      Vector books = library.getBooks( "Carl Sagan" );
      assertEquals( "two books not found", 2, books.size( ) );
      book = (Book)books.elementAt(0);
      BookTest.assertEquals( book3, book );
      book = (Book)books.elementAt(1);
      BookTest.assertEquals( book4, book );
   }

How is this test flawed? Let us count the ways. It tests two separate behaviors: getting a Book by author and title and getting multiple Books by the same author. Looking up two books by two different methods means there are several results to test; thus, there are many asserts—five in all. Although it is sensible to check the results of all the operations, there are redundant tests, such as the two tests of the getBook() method. To get the test to pass, numerous changes must be made immediately to both Book and Library. The complexity of the changes increases the chance that a coding mistake will be made. When one assert in the sequence fails, the rest will be skipped, leaving it uncertain whether those asserts would succeed. So, if the Book lookup by title and author fails, it has to be fixed before the test that gets multiple Books is run. In other words, the tests are coupled so that failure of one may affect the success of the others.

When the number of asserts in a test method is excessive, change it into a test fixture with multiple test methods, each testing one behavior. In Example 4-7, refactoring the test method makes it apparent that the two lookup methods are distinct behaviors and should be tested separately.

Example 4-7. The previous test method refactored into separate test methods
LibraryTest.java
   public void setUp( ) {
      book3 = new Book( "Cosmos", "Carl Sagan" );
                     book4 = new Book( "Contact", "Carl Sagan" );
                     library.addBook( book3 );
                     library.addBook( book4 );
   }

   public void testGetBookByTitleAndAuthor( ) {
      Book book = library.getBook( "Cosmos", "Carl Sagan" );
      BookTest.assertEquals( book3, book );
   }

   public void testGetBooksByAuthor( ) {
      Vector books = library.getBooks( "Carl Sagan" );
      assertEquals( "two books not found", 2, books.size( ) );
      Book book = (Book)books.elementAt(0);
      BookTest.assertEquals( book3, book );
      book = (Book)books.elementAt(1);
      BookTest.assertEquals( book4, book );
   }

Example 4-7 shows LibraryTest with the two separate test methods, one for each behavior. The code to add the two test Books is placed in the setUp( ) method. The tests are isolated and the code is simplified.

Testing Expected Errors

It is important to test the error-handling behavior of production code in addition to its normal behavior. Such tests generate an error and assert that the error is handled as expected. In other words, an expected error produces a unit test success.

The canonical example of a unit test that checks expected error handling is one that tests whether an expected exception is thrown, as shown in Example 4-8.

Example 4-8. Unit test for expected exception
LibraryTest.java
   public void testRemoveNonexistentBook( ) {
      try {
         library.removeBook( "Nonexistent" );
         fail( "Expected exception not thrown" );
      } catch (Exception e) {}
   }

The expected error behavior is that an exception is thrown when the removeBook( ) method is called for a nonexistent Book. If the exception is thrown, the unit test succeeds. If it is not thrown, fail() is called. The fail( ) method is another useful variation on the basic assert method. It is equivalent to assertTrue(false), but it reads better.

Since the removeBook( ) method now throws an exception, the testRemoveBook( ) unit test should be updated, as shown in Example 4-9.

Example 4-9. Unit test that fails when an exception is thrown
LibraryTest.java
   public void testRemoveBook( ) {
      try {
         library.removeBook( "Dune" );
      } catch (Exception e) {
         fail( e.getMessage( ) );
      }
      Book book = library.getBook( "Dune" );
      assertNull( "book is not removed", book );
   }

This example uses fail( ) to cause the test to fail when an unexpected exception is thrown. The exception’s message attribute is used as the assert message.

The same general pattern is followed to test expected error behavior that is not represented by an exception: the test fails if the error is not seen and succeeds if it is. Example 4-10 shows a unit test that attempts to get a nonexistent Book from the Library and asserts that the expected null Book is returned.

Example 4-10. Unit test checking the expected error getting a nonexistent Book
LibraryTest.java
   public void testGetNonexistentBook( ) {
      Book book = library.getBook( "Nonexistent" );
      assertNull( book );
   }

(Not) Testing Get/Set Methods

Every behavior should be covered by a unit test, but every method doesn’t need its own unit test. Many developers don’t test get and set methods, because a method that does nothing but get or set an attribute value is so simple that it is considered immune to failure. Tests of such methods are correspondingly trivial, as shown in the test of getTitle( ) in Example 4-11.

Example 4-11. Trivial test of getTitle( ) method
BookTest.java
   public void testGetTitle( ) {
      Book book = new Book( "Solaris", "Stanislaw Lem" );
      assertEquals( "Solaris", book.getTitle( ) );
   }

If a get or set method produces any side effects or otherwise has nontrivial functionality, it should be tested. For example, with lazy initialization, a get method may compute an attribute value before returning it — behavior that deserves a unit test.

Testing Protected Behavior

A topic of much discussion within the unit testing community is how to test protected or private methods. Since access to such methods is restricted, writing unit tests for them is not straightforward.

Some developers deal with this quandary by simply ignoring protected or private methods and testing only the public interfaces. It’s argued that most of an object’s behavior is reflected in its public methods. The behavior of the protected methods can be inferred by the exposed behavior.

There are some drawbacks to this approach. If there are private methods that contain complex functionality, they will not be tested directly. There is a tendency to make everything public so that it is testable. Some behaviors that should be private might be exposed.

It is possible to access and test protected and private methods, depending on the specifics of how a language defines and enforces object access permissions. In C++, making the test class a friend of the production class allows it to access protected interfaces:

class Library {
#ifdef TEST
   friend class LibraryTest;
#endif
}

This introduces a reference to the test code into the production code, which is not good. Preprocessor directives such as #ifdef TEST can omit such references when the production code is built.

In Java, a simple technique that allows test classes to access protected and private methods is to declare the methods as package scope and place the test classes in the same package as the production classes. The next section, “Test Code Organization,” shows how to arrange Java code this way.

For Java developers who are not satisfied with the direct approach, the Java Reflection API is a tricky way to overcome access protection. The JUnit extension “JUnit-addons” includes a class named PrivateAccessor that uses this approach to access protected or private attributes and methods.

The truly hardcore can follow the examples given here to write their own code that subverts access protection. In Example 4-12, the values of all of Book’s fields are read, regardless of protection. This approach is an ugly hack. Don’t read this code just after a meal.

Example 4-12. Example showing use of Reflection API to get private field values
BookTest.java
import java.lang.reflect.*;

   public void testGetFields( ) {
      Book book = new Book( "test", "test" );
      Field fields[] = book.getClass( ).getDeclaredFields( );
      for ( int i = 0; i < fields.length; i++ ) {
         fields[i].setAccessible( true );
         try {
            String value = (String)fields[i].get( book );
            assertEquals( "test", value );
         } catch (Exception e) {
            fail( e.getMessage( ) );
         }
      }
   }

A Book with title and author “test” is created. The Reflection API method getDeclaredFields() returns an array of all of the Book’s fields, and the call to setAccessible( ) allows access to a field. The Reflection API method get( ) is used to obtain each field’s value. The test asserts that the value of the field is test.

Similarly, in Example 4-13, all of Book’s get methods are called, ignoring access protection (although the get methods actually are public).

Example 4-13. Example using Reflection API to invoke methods
BookTest.java
   public void testInvokeMethods( ) {
      Book book = new Book( "test", "test" );
      Method[] methods = book.getClass( ).getDeclaredMethods( );
      for ( int i = 0; i < methods.length; i++ ) {
         if ( methods[i].getName( ).startsWith("get") ) {
            methods[i].setAccessible( true );
            try {
               String value = (String)methods[i].invoke( book, null );
               assertEquals( "test", value );
            } catch (Exception e) {
               fail( e.getMessage( ) );
            }
         }
      }
   }

Paralleling the previous example, the Reflection API method getDeclaredMethods() returns all of the Book’s methods, and the call to setAccessible( ) subverts the method’s access protection. The test checks the method name and calls only those that have names starting with get to avoid calling Book’s constructor. The Reflection API method invoke() is used to call the methods. Both get methods should return the value test, so this condition is asserted.

Hacks aside, the recommended approach is to design objects so that their important behaviors are public and test those behaviors. Structure the code so that the tests have access to the protected behaviors as well, so that they can be accessed if necessary.

Test Code Organization

As a project grows in size, organizing the files containing production and test code becomes an issue. Although keeping the test and production code in the same directory is the simplest solution, it is better to have a clean separation between the two categories of code. This strategy helps avoid build complications that occur when a directory contains some code that should be linked into the production application, and some that should not. Including the test code in the delivered application is undesirable because it unnecessarily increases the size of the delivery, and also because the tests may expose behavior or design details that the developer meant to keep “under the hood.”

Organizing the code is a language-specific concern. In Java, the directory path to a source file parallels its package membership. The need to test protected interfaces means that unit tests should belong to the same package as the production classes they test, so they must have the same directory path. This can be done by creating separate but parallel hierarchies for the production and test code.

Figure 4-1 shows how the source code for the final version of the virtual library application is organized. There are three Java packages, com.utf.library, com.utf.library.gui, and com.utf.library.xml.

Organization of production and test code
Figure 4-1. Organization of production and test code

The production and test code are located in separate directories, src and test, which are located under the project’s top level DEVROOT directory. For example, the production class Library resides in the directory src/com/utf/library, and the test class LibraryTest is in test/com/utf/library. The test classes’ package names parallel the production classes’ package names, so the test classes can access and test protected behavior of the production code. Since the code is in separate directory trees, it is simple to build and run only the production or test code as desired.

For many other languages, an effective way to organize the code is to place all test code in a subdirectory named test within each production code directory. This arrangement keeps the test code separate, but makes linking it to the production code simple.

Mock Objects

Applications often use interfaces to external objects such as databases, web servers, network services, or hardware devices. Sometimes you must write and test code to interface with objects before they are actually available. Even when the external object is available in the development environment, using it in testing may involve lots of time-consuming, fragile set-up effort, such as loading test data, running services, or placing hardware in a known state. Mock objects are a way of dealing elegantly with this type of situation.

A mock object is a simulation of a real object. Mocks implement the interface of the real object and behave identically with it, to the extent necessary for testing. Mocks also validate that the code that uses them does so correctly. To pass the mock’s validation, other objects must call the correct methods, with the expected parameters, in the expected order. A test object that simply stands in for a real object without providing such verification is not a mock; it is a stub.

Databases are commonly mocked objects. Code that interfaces to a database clearly is important to test. To be tested realistically, the code must be able to perform database operations such as opening and closing connections, reading and writing data, and performing transactions. However, running a live database in the development environment can be a pain. Tests often require that the database is in a specific state or that it contains a specific set of test data. If multiple developers run tests simultaneously, their database operations may interfere.

Mocking the database makes having an actual database unnecessary for testing. The mock has the same interface as the actual database object and the same behavior from the perspective of the client software, but it doesn’t need to actually contain anything but a minimal implementation and possibly some test data. Once the database mock is created, it becomes much simpler to write tests that assume that the database is in various states. Testing becomes faster and easier without the overhead of interfacing with an actual database engine.

To illustrate this, let’s create a mock object representing a database connection object. An interface called DBConnection represents a database connection, as shown in Example 4-14.

Example 4-14. The interface DBConnection, representing a database connection
DBConnection.java
public interface DBConnection {
   void connect( );
   void close( );
   Book selectBook( String title, String author );
}

The class LibraryDB retrieves Books from a database using DBConnection. It is shown in Example 4-15.

Example 4-15. The database interface LibraryDB
LibraryDB.java
public class LibraryDB {

   private DBConnection connection;

   public LibraryDB( DBConnection c ) {
      connection = c;
   }

   Book getBook( String title, String author ) {
      connection.connect( );
      Book book = connection.selectBook( title, author );
      connection.close( );
      return book;
  }
}

We would like to build a unit test for LibraryDB, but we don’t have an actual database yet. So, we’ll mock DBConnection as shown in Example 4-16.

Example 4-16. The mock object MockDBConnection
MockDBConnection.java
public class MockDBConnection implements DBConnection {

   private boolean connected = false;
   private boolean closed = false;

   public void connect( ) { connected = true; }
   public void close( ) { closed = true; }
   public Book selectBook( String title, String author ) {
      return null;
   }

   public boolean validate( ) {
      return connected && closed;
   }

}

MockDBConnection implements the public interface of DBConnection, so it can be used in the interface’s place. MockDBConnection uses the attributes connected and closed to record that the connect( ) and close() methods have been called. The validate( ) method verifies the connection’s state by checking these flags. So, the expectation set by the mock is that code using DBConnection must call both connect( ) and close( ).

The test class LibraryDBTest is shown in Example 4-17.

Example 4-17. The test class LibraryDBTest
LibraryDBTest.java
import junit.framework.*;
import java.util.*;

public class LibraryDBTest extends TestCase {

   public void testGetBook( ) {
      MockDBConnection mock = new MockDBConnection( );
      LibraryDB db = new LibraryDB( mock );
      Book book = db.getBook( "Cosmos", "Carl Sagan" );
      assertTrue( mock.validate( ) );
   }
}

The test method testGetBook( ) creates an instance of MockDBConnection, uses it to construct a LibraryDB, and then calls the LibraryDB method getBook( ). The success of the test depends on the result of the mock’s validate( ) function. If the mock is in the expected state, its validation succeeds and the test passes. The mock object verifies the expected sequence of calls to the database connection and validates that LibraryDB is interacting with it correctly. It also allows LibraryDB and DBConnection to be tested without an actual database.

More sophisticated mock objects go beyond simply setting flags for each method called by recording the arguments provided for method calls, the order of calls, and other details of the method’s state. In this way, mock objects can perform sophisticated validation of interobject interactions.

Mock objects are a deep topic, covered by numerous web sites, books, and online groups. Also, a variety of tools are available to support mock object development for various domains and languages, including EasyMock, jMock, and MockRunner.

AbstractTest

Just like regular classes, abstract classes and interfaces should have their own unit tests. Designing such tests is not straightforward, because these object types cannot be directly instantiated. We’d also like to ensure that every descendant of an abstract class passes the parent object’s tests. The AbstractTest pattern is the answer.

An AbstractTest is itself abstract, like the tested object. It contains an abstract factory method, which produces an instance of the object to test. It also contains the test methods for the abstract class. They resemble ordinary unit test methods, but test instances of the abstract class created by the factory method.

To test a concrete class that is descended from the abstract class, the unit test is subclassed from the AbstractTest. Its factory method returns an instance of the concrete class. When the concrete unit test is run, the AbstractTest is run as well. So, the AbstractTest tests every concrete implementation of the abstract class.

Let’s create an AbstractTest for the interface DBConnection . We’ll add the method isOpen( ) to it, as shown in Example 4-18.

Example 4-18. The interface DBConnection
DBConnection.java
public interface DBConnection {
   void connect( );
   void close( );
   boolean isOpen( );
   Book selectBook( String title, String author );
}

The AbstractTest should test the behavior of the interface to make sure that any concrete implementation of it is correct. Tests of the isOpen( ) method should verify that it returns TRUE after connect( ) is called, and FALSE after close() is called. The AbstractTest class AbstractDBConnectionTestCase , shown in Example 4-19, provides these tests.

Example 4-19. The AbstractTest class AbstractDBConnectionTestCase
AbstractDBConnectionTestCase.java
import junit.framework.*;

public abstract class AbstractDBConnectionTestCase extends TestCase {

   public abstract DBConnection getConnection( );

   public void testIsOpen( ) {
      DBConnection connection = getConnection( );
      connection.connect( );
      assertTrue( connection.isOpen( ) );
   }

   public void testClose( ) {
      DBConnection connection = getConnection( );
      connection.connect( );
      connection.close( );
      assertTrue( !connection.isOpen( ) );
   }
}

The AbstractTest specifies a factory method, getConnection() . Concrete tests that descend from it will implement the factory method, allowing the test methods testIsOpen( ) and testClose( ) to test an instance of the concrete class. Notice how these methods use getConnection() to get the DBConnection to test.

AbstractTests have names ending in “TestCase,” which is different from other test classes. A separate naming convention for AbstractTest classes makes them easily recognizable. Some unit test tools assume that any class named ending with “Test” are test classes that should be instantiated and run, and the different naming convention avoids confusion.

To see the AbstractTest run, we need to define a concrete class descended from DBConnection and a corresponding concrete unit test descended from AbstractDBConnectionTestCase. The concrete class JDBCConnection is shown in Example 4-20.

Example 4-20. The concrete class JDBCConnection
JDBCConnection.java
public class JDBCConnection implements DBConnection {

   private String connectString;
   private boolean open;

   public JDBCConnection( String connect ) {
      connectString = connect;
      open = false;
   }

   public void connect( ) { open = true; }
   public void close( ) { open = false; }
   public boolean isOpen( ) { return open; }
   public String getConnectString( ) { return connectString; }
   public Book selectBook( String title, String author ) {
      return null;
   }
}

JDBCConnection is an initial version of an interface to a JDBC database engine. It differs from the base DBConnection by its member connectString, which contains the URL of a JDBC database connection.

The unit test JDBCConnectionTest tests JDBCConnection. It is derived from the AbstractTest. It is shown in Example 4-21.

Example 4-21. The concrete test JDBCConnectionTest
JDBCConnectionTest.java
public class JDBCConnectionTest extends AbstractDBConnectionTestCase {

   public DBConnection getConnection( ) {
      return new JDBCConnection( "jdbc:odbc:testdb" );
   }

   public void testConnectString( ) {
      JDBCConnection connection = (JDBCConnection)getConnection( );
      String connStr = connection.getConnectString( );
      assertTrue( connStr.equals("jdbc:odbc:testdb") );
   }
}

JDBCConnectionTest implements the factory method getConnection( ) and one test method, testConnectString( ). When the test is instantiated and run, the two test methods in the parent AbstractTest also will be run to test instances of JDBCConnection. This way, the AbstractTest verifies that the concrete subclass passes the tests of the parent interface.

Performance Tests

Like the mock object, unit testing for performance is its own significant topic. Software performance often is neglected at the unit testing level, and is only taken into consideration during functional testing. However, performance-oriented unit tests are powerful tools, especially for applications that require specific performance goals be met. It’s been reported that Apple’s Safari browser was developed in an environment that automatically ran performance tests on any code that was checked in. The code was rejected if it did not meet or exceed the speed standards of previous versions. Thus, the unit tests ensured that the code’s performance is continuously improving.

When a piece of code has a performance problem, it is very useful to first write a test that reveals the problem. This performance test not only lets you know when the code has achieved the desired performance, but also acts as a “canary in the coal mine” that indicates if the performance degrades again.

Tools intended specifically for performance-oriented unit testing are available, such as JUnitPerf. However, it is not difficult to develop performance tests within any unit test framework. This section gives an example of a unit test that tests the speed of retrieving a Book from a Library.

The initial question when writing a performance test is this: what is the performance criterion that the test must meet to pass? Usually, this is expressed in terms of the amount of time that a certain action may take. If the action takes too long, the criterion has not been met, and the test fails.

The Library class developed so far has a very poorly performing algorithm to get a Book. It serially reads through the collection of Books, doing string comparisons on each one until the desired Book is found. This awful lookup stratagem is ideal for demonstrating a performance test that fails initially, but succeeds after a little refactoring. Example 4-22 shows the unit test class LibraryPerfTest .

Example 4-22. Performance unit test LibraryPerfTest
LibraryPerfTest.java
import junit.framework.*;
import java.util.*;

public class LibraryPerfTest extends TestCase {

   private Library library;

   public void setUp( ) {
      library = new Library( );
      for ( int i=0; i < 100000; i++ ) {
         String title = "book" + i;
         String author = "author" + i;
         library.addBook(new Book( title, author ));
      }
   }

   public void testGetBookPerf( ) {
      double maxTime = 100; // milliseconds
      long startTime = System.currentTimeMillis( );
      Book book = library.getBook( "book99999" );
      long endTime = System.currentTimeMillis( );
      long time = endTime-startTime;
      assertTrue( time < maxTime );
      assertEquals( "book99999", book.getTitle( ) );
   }
}

LibraryPerfTest is implemented as a test fixture since it is likely that more performance tests will be implemented. The setUp() method adds 100,000 Books to the Library. The test method testGetBookPerf( ) tests the amount of time it takes to look up a Book. It uses the method currentTimeMillis( ) to get the system time before and after the getBook( ) operation, calculates the elapsed time, and compares it to a performance criterion of 100 milliseconds (0.1 second). As a sanity check, it also asserts that the expected Book was found.

With the Vector-based implementation of Library, the unit test fails:

> java -classpath ".;junit.jar" junit.textui.TestRunner LibraryPerfTest
.F
Time: 0.562
There was 1 failure:
1) testGetBookPerf(LibraryPerfTest)junit.framework.AssertionFailedError
        at LibraryPerfTest.testGetBookPerf(LibraryPerfTest.java:23)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

FAILURES!!!
Tests run: 1,  Failures: 1,  Errors: 0

Library can be refactored to use a Hashtable to store Books. (The refactored Library code is given in the next section, “New Library and Book Code.”) With this change, lookups by title are efficient, and the test passes:

> java -classpath ".;junit.jar" junit.textui.TestRunner LibraryPerfTest
.
Time: 0.734

OK (1 test)

The total test time has increased. This is because addBook() takes longer to add a Book with the Hashtable implementation.

The hardcoded time value of 100 milliseconds used in this example can produce different results when the test is run on faster or slower platforms. Even when run on the same platform, varying machine loads and process priorities mean that a performance test can succeed or fail on subsequent runs without any code changes. Accounting for such variations can present a challenge when designing performance tests. There are a number of techniques to deal with these problems. Consistently running performance tests on the same platform is helpful. Test timing can be based on the time required to run a reference operation rather than on a hardcoded time value, allowing for system performance variations. Timing multiple repetitions of an operation reduces the effect of transient glitches. Finally, performance tests can use order-of-magnitude timing ranges rather than exact minimum timings, so that code meeting general performance goals will pass.

New Library and Book Code

Example 4-23 gives the code for the version of Book referenced in this chapter.

Example 4-23. The class Book
Book.java

public class Book {

   private String title = "";
   private String author = "";

   Book(String title, String author) {
      this.title = title;
      this.author = author;
   }

   public String getTitle( ) { return title; }
   public String getAuthor( ) { return author; }

}

The code for the final version of Library is given in Example 4-24. It uses a Hashtable to store the collection of Books.

Example 4-24. The class Library
Library.java
import java.util.*;

public class Library {

   private Hashtable books;

   Library( ) {
      books = new Hashtable( );
   }

   public void addBook( Book book ) {
      books.put( book.getTitle( ), book );
   }

   public Book getBook( String title ) {
      return (Book)books.get( title );
   }

   public Book getBook( String title, String author ) {
      return (Book)books.get( title );
   }

   public Vector getBooks( String author ) {
      Vector auth_books = new Vector( );
      for ( Enumeration e = books.elements( ); e.hasMoreElements( ); ) {
         Book book = (Book)e.nextElement( );
         if ( book.getAuthor( ).equals(author) ) 
            auth_books.add( book );
      }
      return auth_books;   
   }

   public void removeBook( String title ) throws Exception {
      if ( books.remove( title ) == null )
         throw new Exception("Book not found");
   }

   public int getNumBooks( ) {
      return books.size( );
   }

   public void empty( ) {
      books.clear( );
   }

}

Get Unit Test Frameworks now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.