Crowdsourcing is Hard Part 1: Incentives

The idea that everyone anywhere will contribute to your world-improving project is a powerful concept. The tantalizing vision of an army of unpaid enthusiasts doing all the work for you makes it sound like crowdsourcing will make your job easy, but successful execution of such a project has proven to be very hard.

I told you a bit about the long-term election observation work we are doing with ISFED, GYLA and TI in Georgia, and the fact that even their extensive networks won't have eyes and ears everywhere. Enter crowdsourcing. We're going to try to take the strengths of trained observer election monitoring and meld with crowdsourced citizen reporting to combine the best of both worlds.

The focus of my trip was to get this project rolling. We assembled the leadership of all three organizations together at a gorgeous hotel at the foot of the Caucuses, about two miles from the border with Chechnya. There's plenty of posts to be written about the excercise in cat-herding that is pulling together a partner coalition, but today I'm going to focus just on our discussion of how to integrate crowdsourcing. It made for an intense couple hours.

Crowdsourcing is the inverse Field of Dreams problem: if you build it, they may not come. There's a host of elements that need to be in alignment to pull off a successful crowdsourcing project, and technology is the least of your problems. Thanks to the clever folks at Ushahidi's CrowdMap project anyone can set up their own basic dots-on-a-map site in about 5 minutes.

The picture shows you the part that's hard. For those who can't read my flipchart scrawl, these are the main categories of challenges that we had to solve to make a system work.

• Incentives
• Knowledge
• Barriers to Participation
• Verification of Reports
• Privacy
• Bias
• Attacks
• Response
• Submission Mechanisms

There's a lot there, but basically it all comes down to thinking about why people would participate, what might stop them, and what to do with the information. There's too much to cover in just one blog post, so I'm going to break this out into 3 over the next few days.

Incentives are key. Haiti was the initial crowdsourcing win par excellence - when your city has been turned into rubble and your brother is trapped in a building, you have a strong reason to participate. In a typical election, that's not so easy - even civic-minded citizens have to reach the bar of seeing something they feel is wrong and taking the time (and small amount of money) to send an email or SMS about it.

Patriotism and a wish to see change is probably the biggest and most important reason that people get involved - in our case, a desire to make Georgia a better place. However, a country has to be in a place where things are not so good that individuals are blase about their situation, nor so bad that they feel hopeless to make change.

Some incentives are a bit counter-intuitive. In typical election monitoring missions we intentionally pick organizations that are seen as non-partisan, but in fact bias can be a good thing for citizen reporting. If you're leaning to one side you might have the reason to blow the whistle on chicanary from your rivals and send that SMS.

There's other possible incentives out there - elements of gamification or even financial payouts in the form of SMS credits could encourage people to participate. However, the quality of the information gathered through those mechanisms might then be more suspect.

One of the most significant incentives for participate isn't positive: making mischief. There's some hilarious examples of people taking advantage of a crowdsourcing system to create outcomes that the system organizers didn't, um, anticipate. In a citizen reporting system one has to consider what would happen if there was an organized set of malevolent actors attempting to game the system. The promise of crowdsourcing is that volume of legitimate reports would swamp the bad apples, but that's only true if there is a lot of genuine submissions.

Next post I'll talk about barriers to participation, which brings with it a whole new set of challenges.