Quality results paid for at a fair price are the goals of both work requestors and workers in the crowdsourcing realm. However, the downdraft on quality by unscrupulous workers creates a corresponding plunge in pay rates.
A high school principal friend of mine lamented recently that 90% of his time and energy is spent on 2% of the student body’s troublemakers and the handful of underperforming teachers. He wondered how much good could be generated if that 90% were instead channeled into the mass of productive kids.
Alas, paid crowdsourcing follows a similar profile. Troublemakers consume a disproportionate amount of time and cost which imposes an unfortunate tax on submitted work. Once again, Panos Ipeirotis elegantly outlines this challenging landscape in an insightful post.
“…The requesters already pay minimum wages for work that is worth minimum wage. How is that possible given the effective hourly rate of $2/hour?
The basic problem: Spammers. Given that many large tasks attract spammers, most requesters rely on redundancy to ensure quality. So instead of having a single worker to do a task, they get 5 workers to work on it. This increases the effective rate from $2/hr to $10/hr.
Effectively, what Amazon Mechanical Turk is today is a market for lemons, following the terminology of Akerlof's famous paper, for which he got the 2001 Nobel prize...”
I’d highly recommend reading his entire post. He makes some very good suggestions for the Amazon Mechanical Turk service. Other paid crowdsourcing vendors like Elance and Crowdflower have implemented some creative technical and review oriented processes that significantly address the quality conundrum. And, we’ve written at length on different techniques to improve quality and discourage scammers. Despite, the downward forces on work quality and fair pay are still worthy adversaries.
- Brent
Comments