The first step in designing an evaluation to measure impact is to define what is being evaluated, for whom, and why. We have outlined some initial questions below, but more detailed questions and tools for formalizing an evaluation plan can be found on the Better Evaluation website, and the U.S. CDC’s programme evaluation website.

What is the intervention or policy change being evaluated?

Defining the intervention of interest is a necessary early step in an impact evaluation. This may include listing activities, setting out the purpose or mission of the intervention or policy, and identifying who is responsible for activities related to the intervention or policy change. 

Moreover, the scale of programme and implementation stage may dictate design: a programme that is early in implementation and has little pre-existing evidence of effectiveness may be randomizable, while a large-scale policy change may require a quasi-experimental design.  

What do you think the intervention will be changing? What are the likely outcomes of the intervention?

A theory of change or programme theory, developed collaboratively with intervention authors and implementers, can aid in identifying important impacts of the intervention or programme change. If the programme being evaluated was developed using a pre-existing social or behavioural theory, such as social cognitive theory, this can also be used to understand likely impacts.  

Whom does this policy or intervention affect? What is the population of interest?

Most impact evaluation designs require quantitative data on populations affected by the intervention, and counterfactual data on populations not affected. Defining these populations, as well as a strategy for sampling within these populations, will be necessary for conducting an impact evaluation. 

More broadly, it may be useful when defining the intervention and developing the theory of change to think broadly about who is affected and how, to help identify unexpected or unintended consequences. 

Who are the intended users of the evaluation’s findings?

The findings of the impact evaluation may be used to make decisions about funding, scale-up of the programme to other locations, or changes to programme design and delivery. The likely uses and users for the evaluation should be identified at the outset, to ensure that the findings serve their needs. 

Preferably, participatory or co-production approaches should be built into the design of the intervention and evaluation from the outset to ensure both reflect the priorities of the community affected. 

What resources are available for conducting the intervention?

Finally, the resources that are available to conduct the evaluation may affect the scale of the evaluation, the methods used, and the outcomes that can be measured. For example, are there funds for primary data collection, or will pre-existing programme data be used? Does the evaluation team need additional capacity to conduct complex analyses?