The COSMIC project uses complexity science methods to better understand how health systems function and respond to health programmes, such as payment for performance schemes.
The COSMIC project is a 4.5-year project which began in June 2018. It is a partnership between the LSHTM, the Ifakara Health Institute in Tanzania, the University of Zambia and Makerere University. The COSMIC project is funded by a Health Systems Research Initiative grant which is jointly funded by the Foreign, Commonwealth and Development Office (FCDO), the Medical Research Council (MRC) and Wellcome, in collaboration with the Economic and Social Research Council (ESRC).
The COSMIC project has six main objectives:
- To develop a system dynamic model (SDM) of the Tanzanian health system and its response to a Pay for Performance (P4P) programme;
- To develop an agent based model (ABM) to study the changes in provider and patient behaviour resulting from P4P;
- To synthesise findings from the complementary methods of SDM and ABM;
- To explore the generalisability of the SDM and ABM;
- To generate policy relevant recommendations and translate them to decision makers to optimise the design of P4P;
- To build capacity among decision makers and researchers on SDM and ABM in low and middle income countries (LMIC).
Payment for performance (P4P), or the provision of financial rewards based on the achievement of pre-specified performance targets, is gaining popularity as a mechanism to improve the availability, quality and utilisation of essential health services. Despite a growing number of P4P schemes globally we still understand little about how these programmes affect health systems. The schemes often vary in their design across settings and we need to better understand how to optimise the design of these programmes for maximum impact.
Existing studies have mainly focused on how a single element within the health system is affected, e.g. drugs or staff, rather than looking at all these factors in an interconnected, comprehensive manner, recognising the complexity inherent within health systems.
Two complexity science methods, system dynamic models (SDM) and agent-based models (ABM) are particularly relevant to evaluating health system strengthening initiatives such as P4P. SDM represents a top-down approach modelling the complex macro behaviour of the system. ABM is a bottom-up approach to model emergent macro behaviour from micro-level interactions between agents within the system (e.g. health workers, health care managers and patients). These methods have been applied in high-income settings but there has been much less application in LMIC.
There are 3 main advantages to using these methods:
- they shed light on the complex interplay between different elements of the health system;
- they can be used to identify potential adverse events and triggers for these; and
- they can serve as "virtual laboratories" to test the effects of changes to programme design and context.
This project will build a system dynamic model (SDM) and an agent based model (ABM) of the Tanzanian health system and its response to P4P. These models will be developed based on data from the impact and process evaluations of P4P in each country, and in-depth interviews with key stakeholders. The generalisability of the models to Zambia will be explored. The COSMIC project will build capacity among decision makers and researchers on SDM and ABM to study health systems in LMIC.
This research will contribute:
- methodological knowledge for researchers evaluating P4P and other complex health systems interventions;
- to our understanding of the use of SDM and ABM to study health systems; and
- P4P policy recommendations for programme implementers and policy makers. We will also examine how these methods can be adapted for better use by international donors and decision makers in LMIC for optimising their health systems investments.
The COSMIC project comprises the following six main objectives (click on each one for more information):
- 1. To develop a system dynamic model (SDM) of the Tanzanian health system and its response to a P4P programme
The SDM will help us understand the holistic impact of P4P on the health system. To create the model, we started by developing a casual loop diagram (CLD) showing how the health system responds to the P4P programme. The diagram uses arrows to map out relationships between different variables in the health system. An example of a simple CLD representing how P4P is expected to impact the behaviour of health workers is given below – the promise of bonus payments motivates health workers to deliver more incentivised services to women and children, resulting in further bonus pay outs.
The CLD has been developed using secondary data collected in a previous evaluation on programme impact, with the diagram recently validated through stakeholder interviews. The causal loop diagram will be used as a blueprint to develop the quantitative SDM, reflecting how health system variables behave and change over time in response to P4P.
- 2. To develop an agent based model (ABM) to study the changes in provider and patient behaviour resulting from P4P
In the ABM, we have two groups of agents: patients and health workers. Patients and health workers are modelled to have specific characteristics and behaviours based on the data, whilst also allowing for the fact that not all patients will behave the same in response to the same health worker in the same health facility.
We started by drawing a conceptual map to connect the different characteristics and behaviours of the agents within a health facility. For example, we consider how characteristics and behaviours of health workers (such as kindness), facility characteristics (such as availability of drugs) and patient characteristics (such as socio-economic status) affect the demand for and supply of services.
- 3. To synthesise findings from the complementary methods of SDM and ABM
One of the methodological innovations of this research is to synthesize the findings from ABM and SDM in characterising the P4P scheme in Tanzania. This is the first time that both methods are combined in health systems research to offer a rich picture of the interactions between processes and between agents, and how they influence maternal and child health outcomes. Both the SDM and ADM models will generate outputs of the same stock variables at the top level (e.g. staffing levels, drug availability, cost of care) and the same outcomes (service utilisation and quality), enabling us to analyse what additional or alternative insight each model provides.
- 4. To explore the generalisability of the SDM and ABM
In addition to P4P, two other results-based programmes have been employed in Tanzania with aim to improve maternal and child health (MCH) services; Results-Based Financing (RBF), a scaled-up version of the P4P programme, and Direct Health Facility Financing (DHFF), which transfers donor funding directly to health facilities. We will use the developed SDM model to compare the impact of each of these programmes and identify which components of each of the three programmes is key to improving MCH services in Tanzania.
We will also test the generalisability of our SDM to another setting, Zambia. Using expert interviews and data collected from a previous evaluation of the RBF programme in Zambia, we will alter the structure and behaviour of our Tanzania model to generate policy relevant recommendations regarding RBF in Zambia.
For exploring generalisability of the ABM, we will compare the data from Zambia to that used for calibration of the agent and environmental characteristics in the ABM for Tanzania, and adjust as necessary. Qualitative interviews with providers and managers and/or observations of providers and patients in both Tanzania and in Zambia will also be undertaken to provide a more detailed understanding of behaviours and how they differ by context, to guide a sensitivity analysis to explore the generalisability of the decision rules within the ABM from Tanzania to Zambia.
- 5. To generate policy relevant recommendations and translate them to decision makers to optimise the design of P4P
Once the models have been validated, the models will be used to generate policy relevant recommendations regarding P4P in each of the study countries and share the results with policy makers through a series of discussions and meetings. To this end, we will run a series of simulations (or ‘what if’ scenarios) and optimisation experiments. The simulations will involve assessing the effects of feasible changes to the design of the P4P. We can also run optimisation scenarios that would set outputs/outcomes at a given desired level and examine what changes to the programme design would be needed to reach the desired output/outcome level.
We will develop a user-friendly interface to enable the use of the models by policy makers, so that they can query the model and run their own simulations in the future. This will be achieved through discussions and targeted testing and dialog with decision makers.
- 6. To build capacity among decision makers and researchers on SDM and ABM in low and middle income countries (LMIC)
Policy makers will be engaged throughout the process of model construction and in defining simulation scenarios, and in this way will gain understanding of modelling and its application for decision makers. During the course of the project, a set of training materials will be developed and training sessions held for researchers as well as policy makers. The project will also develop easy-to-use communications tools to help present the findings and the models.