People can only tell apart artificial intelligences from humans around 60 per cent of the time, according to a test taken by more than 1.5 million people. The results raise questions about whether the new generation of AIs should have to identify themselves in conversation, say researchers.
Computer scientist Alan Turing first proposed a test for machine intelligence in 1950. In its original form, a person talks via text with both another person and a machine and has to guess which is which 鈥� if they can鈥檛 differentiate, then the machine has passed the Turing test.
at AI21 Labs in Tel Aviv, Israel, and his colleagues devised an inspired by the Turing test. In their version, a player can swap messages with either a large language model, like ChatGPT鈥檚 GPT-4, or another player. They then have 2 minutes to work out who or what they are interacting with.
Advertisement
To help the AIs avoid giving themselves away when asked, Jannai and his team used three different language models, sometimes switching between them within a single conversation, and gave the AIs random prompts to act as people with specific intentions and objectives. The team also incorporated a typing delay for all players to mask the rapid computation time of the models.
The results of 1.5 million conversations show that people could only tell if they were dealing with聽 another human or an AI 68 per cent of the time. Focusing only on the conversations with AIs, this shrank to 60 per cent, meaning that 40 per cent of people mistakenly identified an AI as human 鈥� not much better than a coin flip.
Jannai and his team were surprised at the ways in which people managed to detect AIs or signal to each other that they were human, such as using local slang or asking the AIs to do things they are forbidden to, like provide instructions for making a bomb.
at the University of Surrey, UK, says that as the tests only lasted 2 minutes, longer interactions with AIs in the real world might make it easier to identify them.
There are also long-standing questions as to how useful the Turing test is for assessing machine intelligence, he says. 鈥淲hen you ask, 鈥業s an AI entity intelligent?鈥�, it鈥檚 ill-defined. You can ask, 鈥業s an AI good at maths?鈥� Or 鈥業s an AI good at finding solutions to conflicts when they happen in the workplace?鈥� Those are slightly better defined, and there might be more concrete answers. In that sense, I find the Turing test less helpful.鈥�
But even if some people aren鈥檛 fooled by AIs, enough are for us to be concerned, says Jannai. 鈥淕iven that, at least in some cases, people can鈥檛 tell the difference, what interactions do we want, and should we want, people to experience when they鈥檙e interacting with AI bots?鈥� he says. 鈥淪hould people be informed about whether they鈥檙e talking to a person or not, or is it OK for a company to let people talk to customer service bots without acknowledging the fact they鈥檙e robots?鈥�
arXiv
