Toward a Big Data Revolution for Sustainable Development

Written by Robert Kirkpatrick, Director, UN Global Pulse.

This is the latest post in our blog series on ‘What kind of ‘data revolution’ do we need for post-2015?

In choosing the term ‘revolution’, it is clear that Secretary-General Ban Ki-moon’s High Level Panel sought to convey the magnitude of the data-driven transformation required for effective development in a post-2015 world. Yet in doing so they also issued a call to action that seems deliberately open-ended, perhaps to provoke constructive debate.

If such was their intent, they have succeeded: witness the tremendous range of interpretations pointed out by Margot Cointreau of PARIS21. Coalitions of the willing are being built to improve the quality of official statistics, improve efficiency by complementing existing methods with mobile surveys, m-health or crowdsourcing, and open up access by pushing hard on governments, UN agencies and NGOs to share more of their data in standardized machine-readable formats.

I believe that there can be no data revolution without all of these ingredients.  Yet I am also convinced that even tremendous progress on all of these fronts will not suffice to bring it about.  If we continue to rely upon the same types of information to make development decisions that we always have, we cannot move beyond linear organizational processes we have today. Per the literal meaning of the word, a revolution must “overturn” previous ways of working.  The Industrial Revolution forever changed not only manufacturing, but nearly every aspect of daily life.   What would it take to bring about an analogous revolution in sustainable development, and how will we know when we have arrived?

Disruption, not incremental improvement

The term ‘’data revolution” implies a transformation in measurement.  The discovery of x-rays allowed a radiologist to see inside a living human body to diagnose injury and disease. The discovery of the genetic code eventually allowed physicians not only to predict risk of disease, but to design treatment customized for each patient. These were true data revolutions in which new tools and data led to more effective interventions.

I think the revolution “test” for development should be this: can development practitioners “see” or measure something that they could not before — behavior consistent with a disease outbreak, an unaccounted for refugee population on the move, or indications that a sudden food price hike is causing unexpected hardship?  If so, does this information allow practitioners to do something useful that they could not do before, or do it earlier?

I believe that a true data revolution allows development decisions to be based – at least in part – on entirely new insights into human nature, and on a newfound ability to observe trends in behavior on a massive scale and in real-time. Imagine being able to track population movement, communication impact or financial instability as it happens. More importantly, the data revolution must enable development organizations to discover and adopt new ways of working more suited to today’s volatile landscape: agile, adaptive, learning-driven processes, made possible through a combination of the wisdom of the crowd, the instinct of the expert, and the power of computation.

A Revolution Already Well Underway

Citizens today – in both developing economies and industrialized ones – are generating a growing ocean of digital data, every minute of every day, just by going about their daily lives.  As we use mobile devices to communicate, buy and sell goods, transfer money, search for information on the Internet, and share our lives publicly on social networks, we leave digital trails that private sector firms are mining to understand the needs of their customers, track emerging market trends, and monitor their own operations in real-time.

The fact is – this data revolution has been underway in the private sector for over a decade, fundamentally altering how companies make decisions and leading to the emergence of countless new business models. With 2016 just around the corner, this is good news for us, because it means we don’t have to start a data revolution from scratch. The challenge – and the opportunity – before us is to adapt the innovative tools and methods of the Big Data Revolution to our own needs and learn how to stay ahead of the curve in a fast-changing post-2015 world.

Toward A Post-2015 Data Revolution

At Global Pulse, we see big data as a raw public good with the potential to generate a real-time understanding of human well-being.  For the past 4 years, we have been researching, innovating and advocating around many of the principal challenges related to transforming it into better outcomes for the poor, discovering new approaches, building tools, and working to demonstrate ways to overcome barriers to adoption and scale. There are a number of areas of big data innovation that UN Global Pulse is leveraging with partners to develop solutions. Examples include examining parental attitudes to vaccinating their children by analysing social media posts, online searches for disease symptoms used to predict disease outbreaks and movement patterns of anonymized mobile phone subscribers used to model the spread of malaria, or identify populations displaced by natural disasters.

Big data is a new resource that may be leveraged to strengthen policy decisions, enable faster responses to emerging crises, and provide real-time evidence of impact. In order to harness its potential we need to develop systems to protect individual privacy so that it can be shared in ways that provide insights relevant for development while protecting market competitiveness.

What does this mean for the global development community?  It’s time to get serious about big data — to move beyond the excessive hype, acknowledge current limitations, and to start figuring out what is relevant for our work.  This isn’t about laying the groundwork for a distant future: there are applications of big data, real-time analytics and data science that should already be part of how we all work.  A number of governments are forging ahead, at least on the domestic front.

As many others have commented, there is a definite need to build skills and capacity around big data since there is a dearth of these skills even in the tech sector.  We need to learn how to form partnerships with potential data providers in the private sector, grounded in the right combination of philanthropic incentives and enlightened self-interest. We need to take data protection and privacy norms, policies, and techniques to a new level to mitigate potential for misuse. Most of all, we each need to make sure organizations are open to begin learning through experimentation. This is critical to learning how to integrate new types of real-time digital information into our processes, policies and culture.

I welcome your thoughts and comments.

3 Comments on "Toward a Big Data Revolution for Sustainable Development"

  1. Big Statisticians are also very sympathetic to the Big data cause: see UN Statistical Commission Big data and modernization of statistical systems at or UNECE statistics at

    The potential obviously exits, however, the actual production using big data sources is still marginal, except on some niches as shown when browsing the complete range of official statistics generally produced as shown by the The Classification of Statistical Activities (CSA)

    As a not very young statistician, I have witnessed several revolutions with enthusiasm: producing statistics with a manual calculator then with a mini-computer, with stand alone micro-computers, with networks of pcs, then internet came, and now the cloud computing. Big data sounds like a natural evolution from previous revolutions that made possible the creation and the processing of large volumes of digitalised information. With each revolution we have been able to produce new informations.

    Will the next revolution be the computerisation of the decision-making process?

  2. Reblogged this on Moderna Museet and commented:
    Datarevolusjonen er allerede i gang:
    “Innbyggere i både utviklingsland og industrialiserte land, genererer hver dag et voksende hav med data, bare ved å gjøre dagligdagse rutiner:
    Vi bruker våre mobiltelefoner, vi kjøper og selger, vi overfører penger, vi søker etter informasjon på internett, vi deler våre liv på sosiale medier.
    Vi etterlater oss digitale spor som private selskaper analyserer for å forstå sine kunder, oppdage markedstrender, og overvåke egne aktiviteter.”

    Her kan du lese hva direktøren for UN Global Pulse – Robert Kirkpatrick – skriver om nye tusenårsmål – og behovet for kunnskap om Big Data.

  3. Big data is obviously the big wave not only for the public sector but the private sectors as well. As Big data becomes a daily phrase for many organizations, I believe there are 3 important factors for this big data revolution: availability of data, data security, and the mindset of analyzing the data. A recent book I came across points out that human tends to reason their way through a problem; however, with big data, that will not be the case. The vast amount of information sometime will point to something that seems completely unrevelant to the problem in hand. To fully appreciate the advantage big data brings, and for us to truly move into an age of big data, I believe a transformation of how we preceive data will come.

Leave a comment

Your email address will not be published.