Last Friday I came home from Edinburgh, head buzzing with ideas after a 2-day meeting on evidence in development and global health. ‘How is evidence defined?’ the speakers from universities and NGOs asked. ‘How is it generated and used?’

Or should we say, ‘How is it fabricated and abused?’ On the face of it, gathering evidence that development programmes have achieved what they set out to do is a good thing – keeping systems functioning, keeping people honest. But, as presenters at the conference showed, it can look quite different from the perspectives of the people actually engaged in the evidence-gathering, or those from whom evidence is being extracted. Development is big business, and the money involved, together with the remoteness of many projects’ aims from local concerns, sets up perverse incentives.

Did you know, for instance, that there are twenty-five thousand NGOs in Nepal alone – almost one for every thousand people? That’s a nice factoid, and now it’s out here on the internet it may go and take on a life of its own. Deepak Thapa of Social Science Baha, from whom I got it, was actually careful to point out that the 25,000 figure includes all registered community-based organizations, from chess clubs to major charities. Stripped of the caveat, though, it makes a nice talking point. It resonates with preconceived ideas we may have about NGOs gone wild, the aid gravy train, etc. This is the fate of many statistics that, relieved of such niceties as standard errors or confidence intervals, colonize our brains. Zombie statistics, some people call them.

Or is it we who are the Zombies? Normative ideals (also called schemas, frames, or cultural models) are part of our make-up; and it’s these that make dispassionate evaluation of evidence in development so difficult. If you feel about anything, “Wouldn’t it be nice if…?” then you have them. We all have them. What varies (what determines our degree of zombiness) is how accessible they are to conscious reflection. If you really think about it – if you’re honest – encounters with “facts” are always partial. As George Lakoff has written: “If the facts don’t fit the frames in your brain, the frames in your brain stay and the facts are ignored or challenged or belittled.” [1]

The topic of evidence in the context of development is especially thorny because international development is a moral as well as a technical project: It involves imposing values and ideals on people whose sense of what matters is often different. A case in point: Tim Allen told the story of how, in the early 2000s, deworming went viral in the Economics and Development communities. Within a few years, giving deworming pills to children had come to be seen as the most effective development intervention out there. This was despite a paucity of evidence; the single academic paper in which the case was made was an anomaly (and, it turns out, based on a flawed analysis). But it fit a frame – the idea that getting rid of parasites in early life should have long-lasting effects – so it stayed. Thousands of children were given deworming pills without their parents’ permission, and without clear understanding of what it was supposed to achieve. [2]

One response to this state of affairs is to say we’d be better off without all this so-called evidence. Another is to ask how evidence gathering could be done smarter and better. One obvious way is to involve the presumed beneficiaries in deciding what to measure, and take account of their priorities are in terms of problems to address. This is not a new idea; it’s the central idea behind so-called participatory approaches that go back to the 1960s. [3] Unfortunately, it runs counter to much global health and development practice these days.

Turning the juggernaut around

My feeling is that meetings like the one in Edinburgh are vitally important. I’m grateful to the organisers for making it happen, and including me. But the voices that were so clearly audible at this conference are hard to hear over the din outside from so many organizations and vested interests. To have impact, people concerned about these issues need to find allies, not only with established NGOs and governments, but with movements – with organized labour, with opposition parties, with students – with allies of all shapes and sizes.

This is easier said than done, of course. Ethiopia can serve as an example: a place where, under adverse circumstances, a movement has emerged in the last few years to address inequity in access to the benefits of development. But how do advocates work with movements like this – without leaders, without clear organizational structure? Given the violent response of the Ethiopian authorities to demonstrators and to critics in general, solidarity is risky. Perhaps these risks come with the territory, wherever an established system of power is challenged.

Some movements may be easier to work with than others, though. In the last session of the workshop, Marlee Tichenor told the story of health workers in Senegal who imposed a boycott on data-collection in their clinics. Between 2010 and 2013, clinicians refused to record routine patient data, which they saw as a distraction from their core responsibility to care for people. Because international donor funds were in jeopardy, the government of Senegal was forced to pay attention. [4] In this case the boycott was organized by health workers’ unions, which have a structure of the sort that that researchers and NGO workers are familiar with.

On the train home to London I reflected on the lessons of the conference: To ask, when examining any development project: who set its goals, and who’s benefitting? To bear in mind that we think in terms of frames, and we evaluate evidence in relation to them, on the fly. And to remember that not everything that counts can be counted, and not everything that can be counted counts.

***

The conference on ‘Evidence and Organisations in Development’ was part of the Spaces of Evidence project.

[1] George Lakoff (2014). The ALL NEW don’t think of an elephant! Know your values and frame the debate. White River Junction: Chelsea Green. P. xiv.

[2] The story is told at greater length by Duncan Green & Mohga Kamal-Yanni in  ‘Deworming delusions’.

[3] Participatory Learning and Action is a good source on these approaches. Sadly it’s no longer published.

[4] See Marlee Tichenor. (2016). The power of data: Global malaria governance and the Senegalese data retention strike. In V. Adams (Ed.), Metrics: What counts in global health (pp. 105–124). Durham: Duke University Press.