Beautiful Testing

A Balanced Breakfast Approach

So far, we have acceptance tests, unit tests, and wikitests. We are painfully aware of the bugs that wikitests fail to find, and yet every two weeks we need to do regression testing of all of our features, with a goal of moving code to internal staging within 48 hours of starting regression tests. To do that, we create a candidate test-tracking wiki page. Using the candidate page, we can assign testers to work on different pieces of the software, and report what bug reports have been filed. When the page says “ok”, “ok”, “ok” for all elements, testing for the iteration is done. (“bz” followed by a number means the tester found a bug.) Let’s look at a candidate page, already in progress, and discuss it:

Testing Iteration Ending 2009-01-30:

• test-release status on iteration page: Green

wikitests:

• FF2: PASS. Widgets tests have been fixed in master but not in 01-30

• FF3 in progress chris. (Thought this was mine. PASS.) Ken (It was, I figured I'd take it off your hands. It's snowing here. -C )

  • TC: Hidden Email Address for Public wiki must have a race condition.

• IE7: PASS mostly mcchris at step 5941 in TC: Calc Watchlist the database is corrupted and apache-perl crashed. Can not reproduce. Otherwise no errors at all. update: Stash suggests that the nlw-error.log record indicates a race condition when saving spreadsheets such that an expected db record does not exist upon a subsequent INSERT.

• IE6: mcchris PASS

  • TC: REST Workspace passes on re-run

    • original run encountered ...