This post is written by Alicia Phillips Mandaville, Research Associate at Overseas Development Institute and Vice President of Amida Technology
All humans make mistakes and those who publish data ranking poor countries’ levels of state fragility are no different. But when potential errors affect millions of dollars and thousands of lives, for the sake of the citizens of those affected countries it’s imperative that adequate checks and balances are in place.
Last weekend – thanks to the diligent work of Thomas Leo Scherer, a a senior research specialist at the U.S. Institute of Peace who writes for The Washington Post’s Monkey Cage– it came to light that something inside the Organization for Economic Cooperation and Development (OECD’s) 2015 ‘States of Fragility’ ranking was deeply flawed. The ranking and analysis is intended to offer OECD’s recommendations on pursuing global development goals in the context of state fragility – ie: how can the world best marshal resources to improve lives in countries with recognized challenges to stability.
However, after considerable effort to replicate the OECD’s results (a standard approach to testing the logic and methodology of other researchers), Scherer concluded that, “the OECD misclassified a large number of states.” Pushing farther, he found that no matter how one reads and applies OECD’s self-described methods, “only 30 of the 70 listed states are classified correctly.” So as best we can tell, the global ranking designed to influence resource allocation for poverty reduction in fragile states is at least half wrong.
For the last several years, I ran a similarly complex, public, data system that ranked developing and emerging countries on behalf of the US Government’s Millennium Challenge Corporation (MCC). I can tell you personally that this is a nightmare outcome. It’s a nightmare because this is the kind of error that reverberatesthrough economic decision making for potentially billions of dollars in public and private investments. But more than that, it is a nightmare because it could have been avoided.
If you wish to do no harm and you are in the business of ranking poor countries on any dimension, you can build checks into the structure of your system. There are three fundamental ways to do this. None are resource intensive, most are quiet and unobtrusive, and all contribute to accuracy. From least to most visible this includes:
- Inverse functions and internal parallels: For every calculation, there is an inverse calculation. Running a system where at least one team member is responsible for decomposing the ranking helps catch early errors even within the team.
- Clarity and civic partnerships: When you invite others in to the methodology you’ve used to calculate a ranking, they can check your work at speeds that allow course corrections. There are numerous non-profit and research institutions deeply committed to expanding knowledge about the cause and consequence of state fragility; inviting them to check the data before it is public is mutually beneficial.
- Transparency: Share your work. This means publishing raw data, data notes, and methodologies in sufficient detail that everyone knows exactly how you calculated an outcome. The process of publishing one’s work not only invites review by others, but it aligns internal incentives to make sure the ranking system is free from human error over the long run.
An ideally responsible international ranking system combines all three of these. I took the belt and suspenders approach deeply seriously when the investment reputations of low income countries were at stake, and over the years MCC’s scorecards grew ever more transparent and never less accurate. These self-check tools are reliable, free, and a definitional part of being responsible in use of data for international development.
The OECD makes fantastic contributions to cross nationally comparable indices and I have the utmost respect for my colleagues there. Now they have an opportunity to demonstrate how a stellar international data team addresses the reality of being human.