Credibility and Relevance
Our discussion revolves around two central concepts: credibility and relevance.
The degree to which you are (or should be) willing to believe the evidence offered and the claims made about it. Part of credibility is validity, the extent to which a study and the claims made from it accurately represent the phenomenon of interest. Validity has a number of facets, including:
Whether what you observed was what you wanted to observe and thought you were observing
Whether measures used actually measure what they are intended to measure
Whether the account of why something is happening is accurate
Whether we can generalize from what has been studied to other, conceptually comparable settings
Credibility requires a study to embody not just high validity but also good reporting so readers know how and when to apply the study.
The degree to which you are (or ought to be) interested in the evidence and claims. In most cases, you won’t even look at an irrelevant study. The typical exception is when the question is of interest to you but the environment in which the answer was found is very different from yours. Relevance in that case is the degree to which the result can be generalized to your environment—which unfortunately is usually a difficult question.
Some people consider relevance to be a facet of credibility, but we believe it is helpful to keep them apart because a low-credibility statement is hardly more than noise (given the gazillions of things ...