Amazon鈥檚 Mechanical Turk (MTurk) is a crowd-working platform that pays people small sums to take part in menial tasks, such as tagging photos or filling out forms. Essentially it is a way to get humans to perform robotic jobs that machines can鈥檛 yet manage 鈥� but now the bots are getting their revenge by taking on the tasks themselves.
That is a problem, because MTurk is widely used by scientists as a cheap way to carry out research. University of Minnesota social psychologist Hui Bai was using it to collect data on the perception of far-right movements when he noticed a massive spike in support for groups including the KKK and the Nazi party 鈥� and it seemed to be coming from an army of bots.
, he discovered a number of responses to his survey weren鈥檛 answering a detailed question. Instead, they simply said 鈥淰ery good鈥� or 鈥淰ery nice鈥�.
Advertisement
Bai also discovered that around half of his sample of 578 responders had the same GPS location as someone else. Around 50 were supposedly logging on from a statue in Buffalo, New York. A handful of others appeared to have taken the survey in the middle of a lake in Kansas.
These strange locations are a tell-tale sign of bots taking the survey, says Bai. 鈥淚 was wondering, what is going on?鈥�
Bot or not?
He鈥檚 not the only one. Erin Buchanan and John Scofield at Missouri State University and the University of Missouri identified bots completing MTurk tasks around two percent of the time in a .
We already know that gathering scientific data with MTurk can cause problems. indicates that between 14 and 18 percent of responses to MTurk surveys are fraudulent in some way. 鈥淚t makes it harder to see what actually happens in your data,鈥� says Buchanan. Adding bots to the mix will only make things worse.
Bai has set up an online survey for fellow researchers to report any anomalies in their data that may be the work of bots. He has received around two dozen replies so far, suggesting that the use of MTurk bots has grown.
鈥淭hree months ago, no more than five or 10 percent of the total subject base were suspected bots,鈥� Bai says. 鈥淣ow half my participants aren鈥檛 human.鈥�
Kurt Gray at the University of North Carolina, Chapel Hill is editor of a psychology journal, where he estimates at least half of the papers he reviews include data from MTurk.
鈥淚t鈥檚 worrying you鈥檙e not observing real people making real decisions,鈥� he says. 鈥淲e鈥檙e in the business of determining how and why people act, and if we鈥檙e not looking at people, we鈥檙e not doing our jobs.鈥�
Amazon did not respond to a request for comment from New 杏吧原创.
