Choosing when to apply specific designs and methods very much depends on the evaluation context e.g. practical decisions about methodological feasibility, and the research or evaluation questions. In some instances, we may only want or be able to answer some questions relating to process evaluation, such as fidelity, dose and reach, but may not consider acceptability, context or mechanisms of change.

Self-report checklists, questionnaires and using routine data can be a simple and cheap way of collecting data on fidelity, dose and reach, but can be subject to responder bias. Structured observations of intervention activities can reduce the potential for discrepancy between what implementers say they do and what they actually do but can be subject to observation bias and may not be appropriate or feasible in all settings.  

Group interviews or focus groups can be useful for providing insights into consensus and conflict among participants, and for eliciting a wide range of views in a relatively efficient way. However, group dynamics can impact participant responses. One to one interview can be useful to explore more sensitive topics but rely heavily on the skill of the interviewer. Some participants may also feel less comfortable in a one-to-one setting. In depth observation can be useful for understanding implementation and the contextual factors shaping this. 

Process evaluations can take place during feasibility or pilot studies, during the evaluation of the intervention, or post evaluation of the intervention. Process evaluations have often been carried out after implementation has completed, however, it is current good practice to collect data on implementation processes during implementation. Timely data collection provides more information about implementation context and can minimise dependence on potentially unreliable retrospective data collection (Moore et al. 2015). 

Increasingly, conducting process evaluation early in the implementation stages is viewed as beneficial as the findings can be used to improve study design and optimise impact.

Our example of convergent mixed methods study is the case study: Assessing the effectiveness of a comprehensive menstrual health intervention program in Ugandan schools (MENISCUS): process evaluation of a pilot intervention study (Nalugya et al., 2020).

The research team designed an embedded mixed methods process evaluation of the pilot menstrual health and hygiene (MHH) intervention in two schools. The evaluation assessed intervention fidelity, reach and dose, possible mechanisms of action, and contextual factors which influenced implementation and outcomes.

The evaluation aimed to explore:

1. How well the intervention was delivered in terms of fidelity, dose and reach.
2. Whether and how acceptable the intervention was among participants.
3. Contextual influences on implementation and outcomes.
4. Potential mechanisms of change.
5. Implications for scale up and transferability.

The evaluation methods, findings, and summary for this case study are below.

Evaluation methods

The process evaluation was integrated in a pre-post implementation pilot study which followed the UK MRC guidelines for the evaluation of complex interventions.

• Quantitative implementation data including recorded participant attendance lists, structured observations of education sessions, and records of painkiller vouchers, were used to determine reach, dose, and fidelity of the intervention
• Quantitative structured observation data was collected during unannounced visits to assess WASH facilities
• Quantitative self-assessment data was collected from students at baseline and endline to elicit information about students’ knowledge, attitudes, perceptions and the acceptability of the intervention, which could influence receipt of the intervention
• Qualitative data was collected during in-depth interviews with female students, parents and teachers, and during focus group discussions with students stratified by sex

Findings

• Dose and reach were achieved as expected. Fidelity was varied owing largely to time-limitations for delivering some intervention activities such as puberty education.
• Interviews with teachers found that the education sessions were too long for most teachers as the majority were part-time, and as a result, they adapted session duration and content.
• Qualitative data was pivotal in providing contextual information for optimising implementation design to suit the needs of teachers and for raising the question on whether other adaptations are needed in order to achieve intended outcomes.
• Further, qualitative data allowed evaluators to go beyond assessing implementation towards understanding mechanisms of impact - training on menstruation and puberty resulted in, for example, improved knowledge, practices and attitudes toward menstrual health hygiene among girls.

Summary

• The use of mixed-methods process evaluation enabled the research team to assess whether and to what extent the intervention was delivered as intended.
• By using multiple quantitative methods they were able fully assess the feasibility and acceptability of the intervention. Using qualitative data helped gain a more in-depth understanding of implementation and outcomes by generating insights into intervention context and mechanisms of impact.
• The combination of quantitative and qualitative data, provided a robust and detailed account of the intervention to inform future optimisation and scale-up.