Taking Testing Seriously

CHAPTER 14RST Meets Signals-Based Testing

—By Wayne Roseberry

Microsoft collects a massive amount of telemetry from users of Microsoft Office. This telemetry does not include the specifics of what users type on the screen, of course. It's all engineering data about what parts of the product are being used and error conditions that occur.

Let's call this data signals.

For seven years, on the Microsoft Office team, I worked on ways of using signal data from the field to guide testing in our labs. We called this signals-based testing. In this chapter, I will describe signals-based testing, then show how it fits with the Rapid Software Testing methodology.

Signals for Testing

I define signals-based testing as testing by operating the product in such a way as to produce a specified profile of engineering and operations data (i.e., signals). Signals-based testing is essentially a coverage-based technique, though it also encompasses the analysis of signals to detect failure.

Signals are data emitted by the product that is not intended for consumption or use by its primary users, but rather for the operations and/or product team. This differentiates it from normal product output, such as user data files, UI displayed on a screen, sound emitted, or even the behavior of machinery and electronic devices. Much modern commercial software, whether running locally or hosted online, produces signals that go back to the product team. The data is usually a mixture of diagnostic information ...