4 Evaluating Evidence

What evidence evaluation involves

Evidence is information used to support or challenge a claim. Evaluating it means asking not only whether evidence exists, but also how dependable it is, how closely it bears on the claim, whether there is enough of it, and what uncertainty remains. Confidence in a claim should rise only to the degree justified by the evidence’s quality and connection to that claim.

Check how dependable the evidence is

Assess the Quality of evidence by considering how it was produced and reported.

• Method: Were observations or measurements gathered systematically? Could the method introduce error or bias?

• Source and expertise: Is the source in a position to know, and does it explain where its information came from?

• Data and analysis: Are the sample, measures, and reasoning appropriate to the claim? Are important limitations disclosed?

• Independence: Do multiple sources rely on separate evidence, or are they repeating the same original report?

A source’s reputation can be relevant, but it does not replace checking the evidence behind a specific claim. A firsthand account may be useful for establishing what someone experienced, yet may not establish how common an event is.

Check whether evidence bears on the claim

Evidence is relevant when it supports or weakens the particular claim being assessed. Check whether it addresses the same people, place, time, and conditions as the claim. A study showing that a program helped one small group may be relevant to whether it can help similar groups, but by itself it does not establish that the program will work everywhere.

Distinguish evidence that directly tests a claim from evidence that is merely associated with it. When two things occur together, the connection may matter, but it does not alone show that one caused the other. Consider what alternative explanations could account for the observation.

Judge whether there is enough evidence

Sufficiency depends on the size and importance of the claim. One well-documented case may establish that something can happen, but usually cannot show how often it happens. A large collection of weak or duplicated reports may still provide little support.

Look for enough evidence to address reasonable alternatives and for agreement across independent methods or sources. In research, consider sample size, study design, measurement quality, and whether other studies reach similar results. Consistency matters, but examine disagreement rather than hiding it: studies may differ in population, methods, or context. Judge the overall weight of evidence, not just the number of sources.

Describe uncertainty and calibrate conclusions

Evidence rarely removes all uncertainty. It can arise from limited data, measurement error, differing results, assumptions in an analysis, or difficulty applying findings to a new situation. Separate uncertainty about what was observed from uncertainty about what caused it or what may happen next. Findings may be dependable within a particular context while remaining uncertain elsewhere.

State conclusions in proportion to the evidence. “Supports,” “suggests,” or “is consistent with” may be more accurate than “proves.” If a source gives a probability, range, or confidence interval, consider what it represents and how it was estimated. Verbal labels such as “likely” can be interpreted differently, so numerical information and its context can make uncertainty clearer.

Apply a practical evaluation

For any important claim, work through these questions:

1. What exactly is being claimed?

2. What evidence supports it, and how was that evidence obtained?

3. How directly does it apply to this claim and context?

4. Is it independent, sufficiently broad, and consistent with other findings?

5. What limitations, alternative explanations, or uncertainties remain?

6. How strongly should I believe the claim given all of the above?

For example, a report says a new study strategy improves test scores. Before accepting the claim, check whether the study compared similar groups, used a suitable measure of learning, included enough participants, and reported the size and uncertainty of the effect. Then ask whether the participants and conditions resemble the setting where the strategy is being considered. The study may provide promising evidence without proving that the strategy will help every student.

Independent, appropriate, converging evidence generally supports firmer conclusions than isolated or indirect evidence. Match confidence to the strength of the full body of evidence.

• Relevance