People seem to be answering research survey questions randomly on Amazon鈥檚 crowdsourcing website. The findings could mean that many academic studies are wrong.
Michael Chmielewski at Southern Methodist University in Texas and Sarah Kucker at Oklahoma State University recently revisited data they had collected on Amazon鈥檚 Mechanical Turk (MTurk) platform, a virtual labour marketplace where people are paid to perform short tasks. MTurk is often used to gather survey responses for social science research.
Since 2015, Chmielewski and Kucker had used MTurk to collect data on how a child鈥檚 language skills developed depending on their parents鈥� personalities. When New 杏吧原创 published an article in 2018 claiming automated bots were targeting the site and ruining academic studies, the pair revisited their data and found inconsistencies. But rather than bots ruining their data, it seems humans racing through possible survey answers and not reading the questions were causing the problems.
Advertisement
By performing a statistical analysis on their results, the team found that the responses just weren鈥檛 right. 鈥淭he conclusions were just massively wrong,鈥� says Chmielewski. 鈥淲ell-established links between neuroticism and depression weren鈥檛 there. We were seeing links in the wrong directions. Things that should have been negatively related were now positively related.鈥�
Read more: Genetic privacy attack could reveal DNA secrets from genealogy sites
Chmielewski believes that the low quality of results is down to participants rushing through the answers and not reading them. 鈥淵ou need to be much more careful using MTurk now than people were before,鈥� he says. After screening for inconsistent answers, the pair removed 60 per cent of survey responses in one portion of their study and 38 per cent in another.
鈥淎s a regular MTurk user I also had the perception that the data quality was deteriorating in recent times,鈥� says Florian Kunze at the University of Konstanz, Germany. Amazon didn鈥檛 respond to a request for comment for this story.
Kristy Milland who does work offered on MTurk wasn鈥檛 surprised by the results. 鈥淢Turk is a labour platform, not a participant pool,鈥� she says. 鈥淲e signed up to make money, not to help science.鈥�
Social Psychological and Personality Science
