Beautiful Testing

The Nitty-Gritty

OK, so enough about the history. How does it work? Sisyphus doesn’t install Firefox, but instead can launch any specific previously installed version of Firefox. This is good, in a way, as it allows you to easily substitute custom builds, such as special patched builds or debug builds. Still, it would be cool to just point at an installable binary and say, “Go!”—just pick your build, install it, and start throwing websites at it. Currently, the sequence is: pick an installed Firefox build, launch Firefox, install extensions, run the tests, exit Firefox, and repeat, a million times if you like.

The command line looks like this:

./tester.sh -t "$TEST_DIR/tests/mozilla.org/top-sites/test.sh \
-s $TEST_DIR/tests/mozilla.org/top-sites/global1000.txt -D 0 -r" \
firefox 1.9.1-tracemonkey debug

-t = which tests to run
-s = file containing URLs to test with
-D is how deep to spider (0 = just load the current URL)
-r = which revision of Firefox to test

The following are some key environment variables:

• XPCOM_DEBUG_BREAK=stack (capture the stack with assertions)

• XPCOM_MEM_LOG=1 (log leak data)

This is very flexible. We can focus on assertion testing, memory-leak testing, or whatever is most important to investigate at the time. Also, these are platform variables, so they also apply to Thunderbird, SeaMonkey, etc.

This tool allows us to load thousands of websites without having to enter them manually into the URL bar. (We actually used to do that quite a bit!) It also allows us to ...