Artificial artificial intelligence – our experience with Mechanical Turk

So Amazon Web Services (AWS) has a pretty neat service called the Mechanical Turk. The name of the service is inspired by this story where a human was hidden inside a chess playing “robot” which impressed crowds during the late 18th century. Amazon’s service doesn’t hide the fact that its powered by humans but the end result is the same – humans performing tasks in lieu of a computer.
The task we needed “turked” was the transcription of various text labels that were on top of overlays on a particular large image. The goal of all of this was a Google Maps product so we all ready had the overlays loaded into a Google Maps UI.  To recap:

• We had a Google Maps application which loaded an image layer with about 3000 discrete overlays.
• We had lat/lng coordinates for all of these overlays.
• Each of these images had a text label that uniquely described it.
• We wanted the Turks to transcribe these labels.

Next, we began to investigate how the Mechanical Turk service actually works. The process is delightfully simple. The idea is to break the tasks out into discrete and reproducible actions called human intelligence tasks (HITs). This pattern aligned well with our problem because all of the overlays were independent and we had coordinates for each of them. Amazon displays each of these HITs inside a template that the “requester” constructs.

The templates are HTML pages (Amazon allows javascript) which plug-in variables for each HIT when the tasks go live. They also capture the data that you want to save from the task – in our case the labels. Designing templates is pretty straightforward and javascript is a nice touch.

For our task, we embeded Google Maps with our image overlaid and used HIT variables to display 3 markers per task. This is where we hit our first and only snag. Because of directory restrictions on Google Maps API keys our HIT template kept generating javascript errors because it was being executed on un-predictable domain names. We didn’t think this was a huge deal so we plowed ahead as planned.
We ran the first set of HITs with a 2 cent/HIT payment and a minimum approval rating of 95%.

The results were less than stellar – within an hour or so we received this email:

A strange error message pops up for your/these HITS.
“The Google Maps API key used on this web site was registered for a different web site. You can generate a new key for this web site at http://code.google.com/apis/maps/.”
Also, the mouseover that displays at screen bottom is “javascript: void (0)”…and no label appears below the marker either.  I do NOT have any javascript blockers operational, so no problem there.
Also, I would appreciate it if you NOT deny me credit for this HIT, as the technical error message was not my fault and not explained or warned in the instructions.
I would like to do several, possibly many of these HITS, if you can tell me what to do to overcome this problem.
Thanks,
[REDACTED]

People were obviously not getting the instructions and were scared away by the JS error and fear of HIT rejection. One of the problems with the service is that you can’t modify your HIT template once a set has been published. You have to cancel the batch and re-start after your edits. At this point, we decided to cancel the run and modify the instructions. With the new instructions people seemed to “get it” and were generally more willing to forgive the JS error.

At the end of the run, the accuracy of the Turks was around 90% or so on transcribing the labels. We got about 25 man hours of work accomplished in about 42 hours of actual time at a cost of about $25. All told we were really impressed with the service as well as the Turks themselves. We definitely recommend the service for any discrete and repeatable tasks.

Things we learned:

• Be EXTREMELY clear in your instructions – try to be as un-ambiguity as possible.
• The Turks live and die by their approval rating so be nice (we just accepted every task).
• Unexpected popups and JavaScript errors probably scare people away so try to avoid them.
• Obviously higher reward rates are going to attract more Turks – something to take into account.

We’ve also recently become fascinated by other things that could be played out or experimented with the Turks. Notably it seems like an ideal venue to experiment with some game theory topics.