1 Assessing Evidence Quality
Learn how to match evidence to a question, examine its credibility and limitations, and make conclusions that reflect its actual strength and uncertainty.
Three judgments about evidence
Evidence is useful only when it bears on the question being asked and is strong enough to support the conclusion drawn. Assessing it involves three related judgments: relevance (whether it addresses the question), credibility (whether it was produced in a trustworthy way), and strength (how much confidence it justifies). These judgments depend on the question: a well-conducted study may be irrelevant to a particular question, while directly relevant evidence may still be too limited to support a firm conclusion.
Match evidence to the question
Begin by stating the claim or decision clearly. Specify the population or setting, the factor being examined, what it is compared with, and the outcome of interest. Then check whether each piece of evidence matches those elements.
For example, evidence about elementary-school reading programs is only indirectly relevant to whether a particular tutoring program improves middle-school mathematics scores. Evidence about the same program may still be informative if it involves a different student population, comparison group, or outcome, but the difference should be acknowledged.
Also ask whether the evidence is sufficiently current for the question, measures the outcome that matters, and applies to the setting in which a decision will be made. Greater indirectness calls for more cautious conclusions.
Examine credibility
Credibility depends on how evidence was produced, not simply on how polished or authoritative its presentation appears. Examine the source and authorship, including disclosed expertise, affiliations, funding, and possible conflicts. Check whether the methods explain how information was collected, who or what was studied, and how results were analyzed.
Consider whether the measurements reasonably capture the concepts or outcomes being discussed. Check whether the data, methods, assumptions, and references are described sufficiently to evaluate the conclusion. Peer review can help identify problems, but it does not guarantee that a result is correct. It is also useful to consider the work alongside other relevant research.
Identify limitations and uncertainty
Look for limitations that could change a result or its interpretation. is a systematic influence that can distort findings; possible examples include selective recruitment, inaccurate measurement, and missing data. occurs when another factor is related to both the proposed cause and the outcome, making their relationship difficult to interpret.
Observational evidence can reveal associations, but by itself may not establish that one factor caused another. Also consider sample size and representativeness, the duration of follow-up, the size and of the estimated effect, and whether results differ across studies.
A wide confidence interval indicates that an effect estimate is imprecise. A result may be statistically significant yet too small to matter in practice. Conversely, an inconclusive result does not necessarily show that there is no effect: the evidence may simply be too limited to distinguish an effect from no effect.
Judge the strength of support
Do not rank evidence by study label alone. Study design matters, but so do the quality of its execution and its fit to the question. A can provide a broad view by bringing together studies, yet its conclusions remain limited if the available studies are few, inconsistent, or biased. A single study may offer useful evidence but rarely settles a complex question on its own. A lower-ranked study can sometimes provide stronger evidence than a higher-ranked one, depending on the actual evidence and the question.
Five common dimensions help assess a body of evidence:
Risk of : Could flaws in study design or conduct systematically affect the findings?
Consistency: Do results point in similar directions across studies, or do they conflict?
: Do the studies address the population, comparison, and outcome of interest?
: Are estimates precise enough to distinguish among meaningful possibilities?
Publication : Might studies with certain results be more likely to appear in the accessible evidence?
These are the core considerations used to assess certainty across studies. Judge certainty for each important outcome, and make explicit the reasons for lowering or raising it.
Draw a proportionate conclusion
A stronger case usually rests on multiple relevant studies with sound methods and reasonably consistent, precise results, without major unexplained gaps or biases. Weaker support may rely on indirect, selective, imprecise, or conflicting evidence. Stronger evidence does not mean certainty: the appropriate conclusion depends on how much uncertainty remains and whether it matters to the decision.
Suppose one small study finds that students offered tutoring score higher on a mathematics test. The result is relevant if the students and program resemble those in the question. But if students chose whether to participate, motivation could partly explain the difference; this is a possible confounder. A small sample may also leave the estimated benefit imprecise.
If other studies find similar benefits using different methods, confidence may increase. If studies find mixed results, investigate differences in program length, student population, and measurement before deciding what the overall evidence supports. A careful conclusion could say that several studies suggest the program may improve scores, while uncertainty remains because the studies are small and differ in their methods. This makes the direction of the evidence and its limitations visible, rather than claiming either that the program works or that there is no effect when the evidence justifies neither certainty.