Close

Study designs for evidence synthesis

Study designs

There are now different types of evidence syntheses, and some are called by different names. Below we present the main ones. 

'Big Picture' reviews

‘Big picture’ reviews, as named by Campbell et al (2023), also referred to as ‘horizon scanning’ or ‘landscape’ reviews, include scoping reviews, mapping reviews and evidence and gap maps. 

They are descriptive syntheses that examine broad research questions, including multiple interventions, outcomes, contexts and/or types of evidence. This makes 'Big Picture' reviews suitable for answering policy questions. They are often presented as a first step to informing research prioritisation and agenda setting, guiding the design of in-depth syntheses methods, and/or highlighting gaps in primary research.  

Mapping reviews

Also called ‘evidence maps’ these reviews typically include hundreds of studies to address broad research questions relating to the state of knowledge, trends or gaps about a topic. They use high-level predefined codes which are normally analysed quantitatively and presented in tabulations and graphs. Qualitative findings could also be included.

Figure 1 (below) presents an example from an evidence map of 482 publications conducted by Blanchard et al (2024) that shows the number of publications that have evaluated food environment policies by publication year and equity characteristics using the PROGRESS-Plus framework

The graph shows that 50% of publications have assessed at least one equity characteristic, 21% have measured two or more, and which characteristics have been more documented. It also highlights that while the absolute number of publications reporting on equity has increased in the most recent years, their proportion has reduced from 72% (13 out of 18) in 2010 to 40% (28 out of 70) in 2020.

evidence map showing the number of publications by publication year that have assessed the effectiveness of food environment policies by an equity characteristic
Figure 1. Example of a graph from an evidence map showing the number of publications by publication year that have assessed the effectiveness of food environment policies by an equity characteristic.

 

Note: A publication could include more than one health equity characteristic (non-mutually exclusive category). SES: Socioeconomic status. 

Evidence and gap maps (EGMs)

These are similar to mapping reviews but employ a predefined framework to select eligible reports and extract data. They are typically presented as an interactive web-based table which allows the user to access further details on specific studies within it. EGMs can be presented alongside a mapping review or as a stand-alone product.

Figure 2 (below) shows an image of an interactive EGM of 1945 studies conducted by Sparling et al (2022) that links food security and nutrition with mental health. The framework (rows and columns) was defined a priori, and cells open when clicked to a bibliography of research at that intersection. The size of the bubbles is proportional to the number of studies in a cell, and the colour of the bubbles represent groups of studies with different populations of interest. 

This example shows that a high number of studies linked anthropometry (especially BMI) and diets with depression. EGMs can also be filtered by any coded characteristic of interest.

Example of an Evidence and Gap Map of research linking food security and nutrition with mental health.
Figure 2: Example of an Evidence and Gap Map of research linking food security and nutrition with mental health.

 

Scoping reviews

These reviews are used to document the nature, range and extent of evidence, including the concepts used and their characteristics. They allow for more exploratory and interpretive methods than other types of Big Picture reviews as data can be extracted using both pre-defined codes and text or themes. They are often used to characterize a wide literature base in the interest of developing clearer parameters and guidance for further research.

Quantitative systematic reviews

Quantitative systematic reviews assess data from quantitative primary studies, typically to answer research questions about the effectiveness, efficacy or safety of an intervention. 

By contrast with Big Picture reviews, systematic reviews analyse the specific results of each included study and tend to focus on narrower research questions about fewer interventions and outcomes. They can be conducted with or without a meta-analysis. 

With a meta-analysis

A meta-analysis is a set of statistical techniques that allows the pooling of results from all studies together for a given outcome in order to determine an overall estimate of effect. The results are presented in a forest plot. 

For a meta-analysis to be meaningful, heterogeneity in the studies needs to be low, i.e., there needs to be little variation in the results due to different populations, interventions, comparators, outcome measures, risk of bias, or study methods or context. 

Figure 3 shows an example of a forest plot conducted by Majorin et al (2019) for a Cochrane review on interventions to improve disposal of child faeces for preventing diarrhoea and soil‐transmitted helminth infection. The pooled results (i.e., the last line with the black diamond) are from case‐control studies and indicate that disposal of faeces in a latrine decreased the odds of diarrhoea and for children aged less than five years (OR 0.72, 95% CI 0.61 to 0.85; 20 comparisons).

For a higher quality version of the figure, please visit page 178 of the original publication.

Picture of forest plot
Figure 3. Example of a forest plot. Each row represents a different study including the number of participants or events, the weight that the study contributes to the analysis, the effect measure, an horizontal line representing the confidence interval and a box which is proportional to the study’s weight. The final lines (including the black diamond) describe the results for all the studies combined.

 

Without a meta-analysis

When a meta-analysis is not feasible or appropriate, for instance when effect estimates from some studies are missing or when heterogeneity is too high, a synthesis without a meta-analysis can be conducted. This consists of a narrative synthesis of the results. Campbell et al (2019) provide guidance for such an approach.

Qualitative systematic reviews

Qualitative systematic reviews assess primary qualitative studies to address review questions aiming to understanding the 'how’ and ‘why’ of the phenomenon of interest. Qualitative reviews require a different approach from quantitative systematic reviews, from searching the literature to synthesizing the data. 

We present below the three most common types of qualitative evidence syntheses. These can also be integrated in a mixed-methods systematic review (see ‘Other types of syntheses’).

Thematic analysis

This is the most common and accessible synthesis approach. It allows the identification of descriptive or analytical themes across the primary studies whether the latter present detailed results (‘thick’ data) or not (‘thin’ data). 

However, sometimes this leads to the production of simplistic descriptions of frequency of themes. The generation of interpretative explanations, constructs or hypotheses beyond the themes from the primary studies depends on the quality and depth of the data available.

Framework synthesis and best fit framework synthesis

These approaches analyse the data using an established framework, which can be useful when the intervention is theory-based, or there is a general agreement regarding how an intervention works and its desired impacts. 

The identification of a suitable framework should be done at the outset of the review (and certainly prior to data extraction). There is a risk to simplify the data and force it into the framework.

Meta-ethnography

This approach applies to small samples of studies that present ‘thick’ data. It allows in-depth interpretive analyses and the generation of new knowledge and theory beyond those presented in the primary studies. While the findings can be rich, meta-ethnography requires an experienced team and is not suitable for large bodies of literature.

Other types of evidence syntheses

There are several other methods to synthesize evidence, including: 

  • mixed-methods systematic reviews (they include both a quantitative and a qualitative component)
  • diagnostic test accuracy reviews
  • economics/cost-effectiveness reviews
  • overviews of reviews (also called umbrella reviews or systematic reviews of systematic reviews)
  • document reviews (syntheses of other types of documents than studies, such as policy documents and the media) 

While there is guidance to produce the former types of reviews (e.g., by Cochrane), it is currently lacking for document reviews.