A clever use of capital letters could be an easy way to flummox artificial intelligences like ChatGPT, letting people distinguish them from humans in conversation.
The idea is reminiscent of the Turing test, first proposed by computer scientist Alan Turing in 1950. He said that an AI would be considered truly intelligent once we couldn鈥檛 distinguish its answers from a human鈥檚. But now that large language models (LLMs) like ChatGPT can sound convincingly human, at the University of California, Santa Barbara (UCSB), and his colleagues wanted to find other ways to weed out the machines.
After considering a number of options, the team settled on two tests. One asks the LLM to identify what is depicted in images created using text characters, known as ASCII art. The other asks questions obscured by capital letters that change the meaning of the words or produce nonsense: for instance, 鈥渋sCURIOSITY waterARCANE wetTURBULENT orILLUSION drySAUNA?鈥� (is water wet or dry?), for which the expected answer is 鈥渨et鈥�.
Advertisement
The team tested five LLMs, including OpenAI鈥檚 GPT-3 and ChatGPT, and Meta鈥檚 LLaMA. All of them failed the capital letter test, and only ChatGPT managed to score on the ASCII test, with a paltry 8 per cent accuracy. By comparison, when the team asked 10 people to take the same tests, they achieved 100 per cent accuracy on the capital letters test and 94 per cent on the ASCII test.
Wang says it isn鈥檛 surprising that people can easily pass the capital letter test. 鈥淗umans like to find and recognise patterns,鈥� he says. 鈥淔or bots, those uppercase and lowercase letters are a single word, and they don鈥檛 know how to exclude part of them.鈥�
at the University of Sheffield, UK, who wasn鈥檛 involved in the work, says that, although LLMs fail the tests now, they might be able to pass them with further training, making the tests useless. 鈥淚 don鈥檛 think the proposed solutions can be a definite way of identifying bot from human,鈥� she says.
But team member , also at UCSB, believes that LLMs won鈥檛 be able to pass the capital letter test because of the way they break text up into chunks to process the data, a process known as tokenisation. 鈥淗umans understand the test at a word level,鈥� he says. 鈥淚f you split the word into different tokens, the machine struggles.鈥�
arXiv
