There are many different kinds of tests. In this section we present the common categories of tests. Keep in mind that the dividing line between these categories is not always sharp, and there can be overlap and gray areas.
Unit tests are the most basic type of automated test. Unit tests are usually organized with an one-to-one correspondence to the physical organization of the code being tested. This is an easy-to-understand organization, so it’s not surprising that this is the most popular kind of automated testing. If you have never done any kind of automated testing, you should start with unit testing. Experienced testers, however, are starting to move to behavior testing instead of unit testing. So, if you are an adventurous novice tester, you might consider starting directly with behavior testing.
In an object-oriented language such as Java, C++, or C#, the units of code that are tested are the public methods within a class. Just as a class collects a group of related methods, a test suite for a given class will contain the unit tests for the methods within that class. There will usually be one test suite for every class.
Sometimes, when you have a small set of classes that work together intimately, you might instead have a single test suite for the group of classes. For example, if you had some kind of specialized collection class and a separate class for iterating over an instance of the collection, you would probably test both classes in a single test suite.
In functional languages such as C, Lisp, or Haskell, the units of code that are tested are the individual functions. The test suites would usually organize tests according to the natural organizational units of the target language (such as files or modules).
Examples 4-1, 4-2, and 4-3[1] show samples of unit tests in Java, C#, and C++, respectively.
Example 4-1. Java tests using JUnit
//CalculatorTest.java
package test;
import static org.junit.Assert.*;
import org.junit.Test;
public class CalculatorTest {
private Calculator calc = new Calculator();
@Test
public void testAdd() {
assertSame(15, calc.add(10, 5));
}
@Test
public void testSubtract() {
assertSame(5, calc.subtract(10, 5));
}
@Test
public void testMultiply() {
assertSame(50, calc.multiply(10, 5));
}
@Test
public void testDivide() {
assertSame(2, calc.divide(10, 5));
}
}Example 4-2. C# tests using NUnit
namespace account
{
using NUnit.Framework;
[TestFixture]
public class AccountTest
{
[Test]
public void TransferFunds()
{
Account source = new Account();
source.Deposit(200.00F);
Account destination = new Account();
destination.Deposit(150.00F);
source.TransferFunds(destination, 100.00F);
Assert.AreEqual(250.00F, destination.Balance);
Assert.AreEqual(100.00F, source.Balance);
}
}
}Example 4-3. C++ tests using cppUnit
void
MoneyTest::testConstructor()
{
// Set up
const std::string currencyFF( "FF" );
const double longNumber = 1234.5678;
// Process
Money money( longNumber, currencyFF );
// Check
CPPUNIT_ASSERT_EQUAL( longNumber, money.getAmount() );
CPPUNIT_ASSERT_EQUAL( currencyFF, money.getCurrency() );
}
void
MoneyTest::testEqual()
{
// Set up
const Money money123FF( 123, "FF" );
const Money money123USD( 123, "USD" );
const Money money12FF( 12, "FF" );
const Money money12USD( 12, "USD" );
// Process & Check
CPPUNIT_ASSERT( money123FF == money123FF );
CPPUNIT_ASSERT( money12FF != money123FF );
CPPUNIT_ASSERT( money123USD != money123FF );
CPPUNIT_ASSERT( money12USD != money123FF );
}
void
MoneyTest::testAdd()
{
// Set up
const Money money12FF( 12, "FF" );
const Money expectedMoney( 135, "FF" );
// Process
Money money( 123, "FF" );
money += money12FF;
// Check
CPPUNIT_ASSERT( expectedMoney == money );
CPPUNIT_ASSERT( &money == &(money += money12FF) );
}Ideally, a unit test should test a single unit of code in isolation, not invoking any other code unit. In practice, this is one of the biggest challenges of automated testing. Code A calls code B, which, in turn, calls code C. So, how do you test only code A?
The primary techniques used are mocks and stubs. A stub is a piece of code that does nothing but substitute for another piece of code that you don’t want called. So, if code A calls code B, you can test code A in isolation by substituting a stub for code B. The stub will usually return a hardcoded value that is suitable for testing code A.
This implies that there is a way to perform this stub substitution at runtime (or at least at link time). Writing code in a way that makes this possible is what we mean when we say that you should create code that is testable. One technique is proving to be very versatile and is growing in popularity: Dependency Injection (DI), also known as Inversion of Control. DI is useful for much more than automated testing, but here we will restrict ourselves to the testing aspects.
In a traditionally written piece of code, say code A, other pieces of code that it calls (its dependencies) will be hardcoded. In other words, code A directly calls code B, code C, etc. With DI, these calls are not hardcoded. Instead, they are called through a variable or some other lookup mechanism, allowing you to switch the actual destination of the call at runtime—in our case, to a stub.
A mock is really just a more complicated set of stubs. A stub is really dumb, typically just containing a hardcoded return value. A mock is designed to substitute for a larger subsystem. As such, it pretends to be an instance of a subsystem, returning a coordinated set of values to emulate the subsystem it is replacing. A mock will usually consist of multiple stub-like functions that contain some minimal logic.
For example, you might have a mock that is designed to stand in place of a database. This mock database might read and return a set of canned responses from a text file.
There are also prewritten mocks ready to use. For example, the Java-based Spring framework includes several prewritten mocks, including one that stands in for the Java Naming and Directory Interface (JNDI) service.
Where unit tests are designed to test individual units of code (in as much isolation as possible), integration tests allow you to determine how these code units work together. Sometimes this just means removing the mocks and stubs and letting the code call its real dependencies. It might also mean writing new tests that are specifically designed to test the interfaces between classes or modules. Figure 4-1 illustrates the basic differences between integration and unit testing.
Behavior testing is a recent development on the testing scene. Still, most of the leading advocates of automated testing are moving to behavior testing, lending it a great deal of advance credibility. Behavior testing is both a different perspective on testing and a new set of test harness tools to directly express this new perspective.
Behavior testing doesn’t take into account the physical organization of the code or things such as classes and modules. Instead, behavior testing focuses on the individual features of the application or system and the behaviors that the application must exhibit to implement the feature. It is these behaviors that are tested.
This is best illustrated by Examples 4-4 and 4-5, which contain sample behavior tests for Ruby and Java, respectively.
Example 4-4. Ruby behavior tests using RSpec
Story: transfer from savings to checking account
As a savings account holder
I want to transfer money from my savings account
to my checking account
So that I can get cash easily from an ATM
Scenario: savings account has sufficient funds
Given my savings account balance is $100
And my checking account balance is $10
When I transfer $20 from savings to checking
Then my savings account balance should be $80
And my checking account balance should be $30
Scenario: savings account has insufficient funds
Given my savings account balance is $50
And my checking account balance is $10
When I transfer $60 from savings to checking
Then my savings account balance should be $50
And my checking account balance should be $10Example 4-5. Java behavior tests using JBehave
/** set balance = 50 */
public class AccountIsInCredit extends GivenUsingMiniMock {
public void setUp(World world) {
Mock account = (Mock) world.get("account",
mock(Account.class));
account.stubs("getBalance")
.withNoArguments()
.will(returnValue(50));
}
}
. . .
public class HappyScenario extends MultiStepScenario {
public void specifySteps() {
given(new AccountIsInCredit());
when(new UserRequestsCash());
then(new ATMShouldDispenseCash());
then(new ATMShouldReturnBankCardToCustomer());
then(new AccountBalanceShouldBeReduced());
}
}
public class OverdrawnWithoutPermission extends MultiStepScenario {
public void specifySteps() {
given(new HappyScenarioWithOverdraft());
given(new AccountHasNegativeBalanceWithoutPermission());
when(new UserRequestsCash());
then(new ATMShouldRefuseCash());
then(new ATMShouldReturnBankCardToCustomer());
}
}Notice that the Ruby example reads like normal English. The Ruby programming language excels at creating domain-specific languages (DSLs) such as what you see in Example 4-4. Rest assured that this is really executable Ruby code.
Testing behaviors is closely related to the subject of the next section: executable specifications.
Another recent trend in automated tests has been the use of executable tests as specifications or requirements. These executable specifications often double as acceptance tests—when the project is completed, all of these tests should pass.
The idea of executable requirements has been developed from two separate camps. The first is Framework for Integrated Test (FIT), which uses tables on wiki pages to specify tests that are to be run by the FIT test harness. The second is a natural extension of behavior testing to a higher level of behavior that expresses the requirements of the system.
Both of these approaches to executable specifications share the same attributes: they are readable and understandable to both the business stakeholders and the software developers, and they are executable as tests by the computer.
There are a number of areas in which an application or system must be tested that are not related to its functions. Here are a few of them:
Performance testing
Load testing
Security (vulnerability) testing
Compatibility testing
Usability testing
These types of testing are much more difficult to automate than the types of testing we have presented so far. In fact, it may be impossible to satisfactorily automate some of them (how would you automate usability testing?).
Because of this, these types of testing have remained largely manual. Tools are available to assist with performance and load testing, but it is rare for them to be fully automated.
Testing a user interface, whether it is a desktop UI or the browser interface in a web application, is another testing area that has been difficult to automate. Tools are available to help automate the testing of a user interface, but they tend to be specific to a particular environment and are difficult to create and maintain.
One potential bright spot is browser-based web applications. Since the environment and technologies that underlie web applications are based on standards, it should be possible to build fairly generic tools to automate testing of the user interface. One such tool, Selenium, is written in JavaScript and runs in the browser. Selenium lets you write automated tests that can simulate the actions of a user and check for the correct responses from the application. Over time you can expect to see more tools like this.
Since it is difficult to automate user interface testing, a traditional alternative strategy is to divide the user interface code into two parts. The bulk of the code implements the “business logic” of the user interface, with a thin GUI rendering layer on top. Automated tests are then directed at the business layer.
That traditional workaround has been rendered largely obsolete by the increasing use of the Model-View-Controller (MVC) architecture. The MVC architecture produces the same separation of business logic and GUI rendering by its very nature. In an MVC architecture, the automated tests are written to test the model and controller parts.
[1] Most of the code examples in this chapter have been excerpted (with permission) from the documentation of each test framework.
Get The Art of Lean Software Development now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
