« Crowdsourcing is an Internet Intern for PR Professionals | Main | Paid Crowdsourcing Its March toward Mainstream Business »

08/20/2009

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Brent,

Nice - Can I be your intern on the next list? But seriously, I can't tell if you actually paid twice for this work? Did you do it twice and use one to check the quality of the other? How do you know that for 3 cents that you are getting good results.

Thanks.

JC

John,

Good question. Our internal intern performed the task on enough companies to give us a confident estimate of how long it would take to do the full list should she actually have to do it. (Bookmark that work for later use in quality testing the Smartsourced results accuracy.)

Our confidence in the quality of the crowdsourced answers comes from the parts of the process not specified in the post because they would have added extra complexity to understanding the basic point of the article - cost and earnings comparisons.

The statistics of the 4 tasks per company are a roll-up summary of the actual task process. Given the nature of the data being asked for, some parts were easier than other parts. "Does this company offer a GL module?" was dead simple and fast, whereas "Give an average price paid for the product" had varying degrees of difficulty in coming to an answer. So, in some of the task categories, we asked more than one worker to answer the same question and averaged the answers. The price paid per task ranged from $0.01 to $0.05 depending on the difficulty. Therefore, the overall average time to complete and cost per task are a rollup of what was actually varied task prices and more than 600 tasks.

We compared the early results with those of the intern to gauge quality, and then at the end inspected it all in the same fashion you might review an employee's work. Spot checks and eyeballing.

I don't get your math. If it takes 10 seconds per task * 600 tasks it should take 100 minutes. Why did the intern take 720 minutes? Forgive me but it seems like the tasks should take longer if random people are doing bits of them instead of one person (no learning or task adaptation can happen). If this is the case, your basically just paying people really little.

The comments to this entry are closed.