Digital revolutionaries are all human — but we can compensate for that

This post is written by Alicia Phillips Mandaville, Research Associate at Overseas Development Institute and Vice President of Amida Technology

All humans make mistakes and those who publish data ranking poor countries’ levels of state fragility are no different.  But when potential errors affect millions of dollars and thousands of lives, for the sake of the citizens of those affected countries  it’s imperative that adequate checks and balances are in place.

Last weekend – thanks to the diligent work of Thomas Leo Scherer, a a senior research specialist at the U.S. Institute of Peace who writes for The Washington Post’s Monkey Cage– it came to light that something inside the Organization for Economic Cooperation and Development (OECD’s) 2015 ‘States of Fragility’ ranking was deeply flawed.  The ranking and analysis is intended to offer OECD’s recommendations on pursuing global development goals in the context of state fragility – ie: how can the world best marshal resources to improve lives in countries with recognized challenges to stability.

However, after considerable effort to replicate the OECD’s results (a standard approach to testing the logic and methodology of other researchers), Scherer concluded that, “the OECD misclassified a large number of states.” Pushing farther, he found that no matter how one reads and applies OECD’s self-described methods, “only 30 of the 70 listed states are classified correctly.” So as best we can tell, the global ranking designed to influence resource allocation for poverty reduction in fragile states is at least half wrong.

For the last several years, I ran a similarly complex, public, data system that ranked developing and emerging countries on behalf of the US Government’s Millennium Challenge Corporation (MCC). I can tell you personally that this is a nightmare outcome. It’s a nightmare because this is the kind of error that reverberatesthrough economic decision making for potentially billions of dollars in public and private investments. But more than that, it is a nightmare because it could have been avoided.

If you wish to do no harm and you are in the business of ranking poor countries on any dimension, you can build checks into the structure of your system. There are three fundamental ways to do this.  None are resource intensive, most are quiet and unobtrusive, and all contribute to accuracy.  From least to most visible this includes:

  • Inverse functions and internal parallels: For every calculation, there is an inverse calculation. Running a system where at least one team member is responsible for decomposing the ranking helps catch early errors even within the team.
  • Clarity and civic partnerships: When you invite others in to the methodology you’ve used to calculate a ranking, they can check your work at speeds that allow course corrections. There are numerous non-profit and research institutions deeply committed to expanding knowledge about the cause and consequence of state fragility; inviting them to check the data before it is public is mutually beneficial.
  • Transparency: Share your work.  This means publishing raw data, data notes, and methodologies in sufficient detail that everyone knows exactly how you calculated an outcome.  The process of publishing one’s work not only invites review by others, but it aligns internal incentives to make sure the ranking system is free from human error over the long run.

An ideally responsible international ranking system combines all three of these. I took the belt and suspenders approach deeply seriously when the investment reputations of low income countries were at stake, and over the years MCC’s scorecards grew ever more transparent and never less accurate. These self-check tools are reliable, free, and a definitional part of being responsible in use of data for international development.

The OECD makes fantastic contributions to cross nationally comparable indices and I have the utmost respect for my colleagues there.  Now they have an opportunity to demonstrate how a stellar international data team addresses the reality of being human.

1 Comment on "Digital revolutionaries are all human — but we can compensate for that"

  1. JessieHenshaw | May 22, 2015 at 12:42 pm | Reply

    It’s a great message, but in sending it as a Tweet saw a need to qualify:
    “Digital revolutionaries are all human — but [if we are deeply truthful] we can [begin to] compensate for that”

    I think it’s really our deeply conflicted and dangerously overreaching world, that pushes us to make serious mistakes, and also makes opportunity for being deeply truthful to produce new insight into the amazingly unsustainable circumstances we find ourselves in. It seems to have been an accumulation of “little cheats” that allowed us to inherit it, is what many people notice.

    Even when hearts are in the right place the model of prosperity we inherited can lead our minds far over the edge. I got the heart of why money is such a conflicted “universal good” that it does “universal harm” at the same time, simplified into a Tweet today too:
    “What’s #money, but #Giving #People #Support for #being #more #productive #than #others, to then #take #support #away #from, #to #multiply??” …

    As usual, the heart of our wicked problem has long been “hidden in sight”, and as much as it’s a bit shocking, the reward is in being patient … and not running away.

Leave a comment

Your email address will not be published.