Uncertainty and its consequences in social policy evaluation and evidence-based decision making
The methodologies of RCTs and systematic reviews imply a high standard for the level of rigour in evidence-based decision making. When these standards are not met, how should decisionmakers act? When a clear body of evidence is not available there is a risk that action is delayed while further research is conducted or that action is taken without optimal use of the evidence that does exist. In fact, all evidence-based decisions involve a degree of uncertainty. The question we address in this paper is: What level of certainty is required for which kinds of decisions? Scientific skepticism demands a high degree of certainty for sure and steady advances in knowledge. Medical interventions with a risk of death require a high degree of certainty. But what about decisions in social policy? We argue that decisions should be made based on a consideration of both the uncertainty and consequences of all possible outcomes. Put simply, if severe negative consequences can be ruled out, we can tolerate greater uncertainty in positive outcomes. We present a framework for making decisions on partial evidence. The framework has implications for the generation of evidence too. Social policy evaluations should systematically consider potential negative outcomes. Sources of uncertainty - including assumptions, methods, generalizability of findings as well as statistical uncertainty - should be analyzed, quantified where possible and reported. Investment should be made in reducing uncertainty in outcomes with the biggest consequences. Uncertainty can be managed by placing small bets to achieve large goals. Overall, more systematic analysis of uncertainty and its consequences can improve approaches to decision-making and to the generation of evidence.
Matthew Jukes is a Fellow and Senior Education Evaluation Specialist at RTI International. He has two decades of academic and professional experience in evaluating education projects, particularly in early-grade literacy interventions and the promotion of learning through better health.lshtm
Dr. Jukes’ research addresses culturally relevant approaches to assessment of social and emotional competencies in Tanzania; improving pedagogy through an understanding of the cultural basis of teacher-child interactions; frameworks to improve evidence-based decision-making; and methods to set reading proficiency benchmarks. He is contributing to projects in Malawi and Tanzania aimed at improving the quality of pre-primary and primary education in those countries. Dr. Jukes came to RTI in 2016 from Room to Read, where he was Senior Director of Global Research, Monitoring and Evaluation. Prior to joining Room to Read in 2012, he was an associate professor of International Education at the Harvard Graduate School of Education, where he taught courses on evidence-based decision making and on the roles of both culture and health in effective education. Dr. Jukes has also applied his research to work with the World Bank, UNAIDS, UNESCO, USAID and Save the Children.