Policy Outcome Evaluation: Questions, Indicators, and Evidence
Learn how to frame answerable policy evaluation questions, select indicators and evidence, assess change and attribution, and draw appropriately cautious conclusions.
Frame answerable evaluation questions
Begin with the policy’s purpose and theory of change: identify the activities or requirements expected to change behavior, services, organizations, or conditions. Use this pathway to identify important outcomes and the decisions the evaluation should inform.
Good evaluation questions are focused, open-ended, and answerable with available evidence. Prioritize them according to their importance to decision-makers and affected communities, the time outcomes need to emerge, and available evaluation resources.
An defines what the evaluation will answer; survey and interview questions collect information used to answer it. A useful question specifies the policy, population, and , and, where relevant, a time period or context. Avoid questions too broad to guide data collection, such as “Did the policy work?”
For a paid-sick-leave policy, questions could ask to what extent workers’ access to paid leave increased; whether access changed similarly for part-time, low-wage, and other worker groups; how employers implemented the policy and what barriers shaped workers’ ability to use leave; and whether the policy had consequences such as reduced work hours or changes in hiring.
Connect results stages to indicators
Outputs, outcomes, and impacts describe different stages in a policy’s results chain. For a paid-sick-leave policy, guidance delivered to employers or inspections conducted are outputs; more eligible workers using paid leave when ill is an ; reduced illness transmission or improved worker health are possible broader, longer-term impacts.
Terminology is not identical across evaluation fields. In particular, evaluation often means a study designed to estimate what changed because of a policy compared with what would likely have happened without it. This causal question differs from simply measuring whether an changed. Define terms consistently at the start of an evaluation.
An is a measurable sign of progress toward a defined , while a specifies how the is calculated or assessed. For example, the question might be whether a policy increased access to paid leave; the is workers having greater access; the is the share of eligible workers reporting that they have and use paid leave; and the is the percentage of surveyed eligible workers reporting use in the past 12 months.
Choose a small set of indicators that directly answers the evaluation questions. Document each ’s definition, population, data source, collection frequency, and calculation. Strong indicators should be valid, representing the rather than an easier indirect substitute; reliable, producing consistent results when the underlying situation has not changed; feasible to gather and analyze with available time, skills, access, and funding; and sensitive to meaningful change, with a clear direction and scale.
Counting employer guidance documents distributed can an , but cannot by itself show that workers gained access to leave. Check data accuracy, completeness, and suitability for the evaluation’s purpose, and avoid collecting more indicators than can be used well.
Match evidence to the question
Match evidence to the . measures amounts, rates, differences, or trends. Administrative records, surveys, audits, or routinely collected statistics may show, for example, whether the percentage of eligible workers using leave changed over time.
captures experiences, explanations, and context. Interviews, focus groups, documents, and observations may reveal why workers did or did not use leave, how employers interpreted requirements, or why implementation differed across places.
Use to estimate how much change occurred and to investigate how change happened and how people experienced it. Combining methods can provide a fuller account, but no single method guarantees credibility. Consider data quality, sampling, missing information, and who may be absent from the evidence.
Assess change and attribution
A describes an before implementation, using the same definition and measurement approach planned for follow-up. Without a suitable , it may be difficult to tell whether an observed level is new or longstanding. When possible, collect data before the policy begins and track outcomes at appropriate intervals afterward.
A before-and-after comparison can show whether an changed, but does not establish that the policy caused the change. Economic conditions, other policies, or changes in the population may also explain the result.
To estimate attribution, compare observed outcomes with a credible estimate of what would have happened without the policy: the . Depending on context and feasibility, evaluators may use a randomized comparison, a well-matched comparison group, or an interrupted time-series design. A comparison is informative only when its limitations and comparability are examined.
If a robust is not feasible, theory-based analysis can test whether the expected causal pathway is supported and consider alternative explanations. State conclusions with appropriate caution.
Examine equity and
Average results can hide important differences. Where appropriate and ethically safe, analyze outcomes by relevant characteristics such as income, geography, age, disability, or employment type. Examine who benefits, who faces barriers, and under what conditions.
Choose group breakdowns with affected communities and protect privacy, especially when groups are small or vulnerable. Also look for , both positive and negative. A policy intended to expand access might also change employers’ scheduling practices or affect workers differently across industries. Open-ended questions and other evidence can surface outcomes not anticipated in the original policy logic.
Interpret evidence carefully
Before analyzing data, decide what patterns would count as meaningful progress and what analysis is appropriate. Then work through the evidence in a way that answers each directly:
Summarize the evidence for each , including and follow-up values where available.
Examine differences over time and between relevant groups, and assess data quality and uncertainty.
Compare quantitative patterns with qualitative accounts and implementation information; investigate where they agree or differ.
Consider alternative explanations, the policy’s timing, and whether enough time has passed for the to emerge.
Answer each directly, distinguishing observed change from evidence of causal effect.
State limitations and avoid claims stronger than the evidence supports.
For example, increased leave use after a policy begins supports the conclusion that use rose during the period studied. Without a credible comparison or other causal evidence, it does not by itself establish that the policy caused the increase. Transparent interpretation makes findings more useful and trustworthy.