Data-driven decisionmaking has a lovely alliterative sound. It also makes a lot of sense in the international development world - shouldn’t we have good, solid information to help shape the choice of program activities?
Easier said than done, regrettably. Our team has been mulling about how we can use concepts from randomized control trials - RCTs - to get information on what works and what doesn’t in NDI’s tech4dem work. It is particularly important with new technologies that we’re often pushing because often there’s not enough of a track record for these shiny new tools or approaches to determine if they are effective.
In a a randomized control trial, you need five basic things:
- A research question. Something in the form "Is X or Y better at Z?";
- Something to randomize - this could be wards, governorates, individuals, towns, households, radio markets, etc.;
- A controlled way to reach people where we determine who does or doesn't get the "treatment" - that is information, or tool or whatever it is we want to investigate as to its effectiveness;
- A quantifiable number for results to determine whether your output is changing;
- Representation or scale. There needs to be a certain representation or number of comparison points.
Recently I had the opportunity to attend a political geeks’ conference that was all about RCTs. A year out from 2012’s epic electoral slugfest between US presidential candidates Obama and Romney, a lot of very useful research on the efficacy of different campaign tools is coming out. Real-world research on the effects of different political techniques is hard (if you’re interested, I highly recommend the engaging overview in the book Victory Lab.)
In the world of international development this concept is even harder. Working without detailed citizen profiles - or even basic polling data - impact can be hard to measure. A properly implemented RCT can be colossally expensive; while donors might want you to only do things in your programs that work, funding the research to see what works is harder. (Next time you’re talking to your friendly donor, make sure you push the idea of money for trials.)
Ethically RCTs can be a challenge as well. Because you need a control group, there’s is some population that isn’t getting whatever Awesome Democracy Sauce(tm) your program is providing. While NDI isn’t in the business of dispensing, say, life-saving medicines, one has to think about the implications of doing something in one area and not in another. Unlike a university or federal agency, we are not required to submit our project to an IRB that would review our projects according to human subjects ethical guidelines so have to self-monitor any RCTs we consider.
The most confusing thing for me is the core question of segmenting groups: to be able to evaluate a trial, there needs to be some people who receive your program effects, and others that do not. That could be by electoral constituency, by radio market, or in some cases by individual or household, but they need to be very similar groups.
We’re wrapping up a small-scale RCT over the recent Georgia elections; I’ll look forward to talking about it as soon as we get the data processed.
Any experience out there with using RCTs in dmocracy and governance work in international development? We’d love to learn from you on what worked and what didn’t.