Making Software

Credibility and Relevance

Our discussion revolves around two central concepts: credibility and relevance.

Credibility

The degree to which you are (or should be) willing to believe the evidence offered and the claims made about it. Part of credibility is validity, the extent to which a study and the claims made from it accurately represent the phenomenon of interest. Validity has a number of facets, including:

• Whether what you observed was what you wanted to observe and thought you were observing

• Whether measures used actually measure what they are intended to measure

• Whether the account of why something is happening is accurate

• Whether we can generalize from what has been studied to other, conceptually comparable settings

Credibility requires a study to embody not just high validity but also good reporting so readers know how and when to apply the study.

Relevance

The degree to which you are (or ought to be) interested in the evidence and claims. In most cases, you won’t even look at an irrelevant study. The typical exception is when the question is of interest to you but the environment in which the answer was found is very different from yours. Relevance in that case is the degree to which the result can be generalized to your environment—which unfortunately is usually a difficult question.

Some people consider relevance to be a facet of credibility, but we believe it is helpful to keep them apart because a low-credibility statement is hardly more than noise (given the gazillions of things ...