London School of Hygiene & Tropical Medicine

Using machine learning to reduce the effort of screening has been discussed for decades, and the arrival of generative AI has promised even greater work savings. However, the systematic review community has not arrived at a common understanding of how work savings can be achieved while maintaining rigorous methodological standards.

The most common paradigm for AI-assisted screening is prioritised screening, where AI is used to predict the relevance of unseen documents, which are then screened by hand in descending order. This results in the identification of all eligible studies long before all studies have been screened. However, this only saves labour if screening is stopped early, which always carries a risk that eligible studies are missed. Stopping criteria are commonly used to balance the costs of screening manually with the risks of missing eligible studies.

In anticipation of the possible update of guidance in the Cochrane handbook on AI-assisted screening and stopping criteria, Max will discuss what makes a stopping criterion compatible with the maintenance of rigorous methodological standards. After presenting evidence from a recent evaluation of the reliability of stopping criteria, Max will set out considerations for how these could and should be implemented in practice.

Speaker

Max Callaghan

Max Callaghan is a scientist at the Potsdam Institute for Climate Impacts Research. He works on the development of methods to use AI responsibly to assist evidence synthesis. He is an associate convenor of the AI methods group, a joint Methods Group between Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence (CEE), and a methods editor for AI for Campbell Systematic Reviews.

Event notices

• Please note that you can join this event in person or you can join the session remotely
• Please note that the recording link will be listed on this page when available