John is in the playground. Bob is in the office. Where is John? If you know the answer, you鈥檙e either a human, or software taking its first steps towards full artificial intelligence. Researchers at Facebook鈥檚 AI lab in New York say an exam of simple questions like this could help in designing machines that think like people.
Test yourself on some of the questions here.
Computing pioneer Alan Turing famously set his own test for AI, in which a human tries to sort other humans from machines by conversing with both. However, this approach has a downside.
Advertisement
鈥淭he Turing test requires us to teach the machine skills that are not actually useful for us,鈥� says , an AI researcher at Microsoft. For example, to pass the test an AI must learn to lie about its true nature and pretend not to know facts a human wouldn鈥檛.
These skills are no use to Facebook, which is looking for more sophisticated ways to filter your news feed. 鈥淧eople have a limited amount of time to spend on Facebook, so we have to curate that somehow,鈥� says , Facebook鈥檚 director of AI research. 鈥淔or that you need to understand content and you need to understand people.鈥�
AI plays 20 questions
In the longer term, Facebook also wants to create a digital assistant that can handle a real dialogue with humans, unlike the scripted conversations possible with the likes of Apple鈥檚 Siri.
Similar goals are driving AI researchers everywhere to develop more comprehensive exams to challenge their machines. Facebook itself has created 20 tasks, which get progressively harder 鈥� the example at the top of this article is of the easiest type. The team says any potential AI must pass all of them if it is ever to develop true intelligence.
Each task involves short descriptions followed by some questions, a bit like a reading comprehension quiz. Harder examples include figuring out whether one object could fit inside another, or why a person might act a certain way. 鈥淲e wanted tasks that any human who can read can answer,鈥� says Facebook鈥檚 , who led the research.
Having a range of questions challenges the AI in different ways, meaning systems that have a single strength fall short.
The Facebook team used its exam to test a number of learning algorithms, and found that none managed full marks. The best performance was by a variant of a neural network with access to an external memory, an approach that Google鈥檚 AI subsidiary DeepMind is also investigating. But even this fell down on tasks like counting objects in a question or spatial reasoning.
Richardson has also developed a test of AI reading comprehension, called MCTest. But the questions in MCTest are written by hand, whereas Facebook鈥檚 are automatically generated.
The details for Facebook鈥檚 tasks are plucked from a simulation of a simple world, a little like an old-school text adventure, where characters move around and pick up objects. Weston says this is key to keeping questions fresh for repeated testing and learning.
But such testing has its problems, says Peter Clark of the in Seattle, because the AI doesn鈥檛 need to understand what real-world objects the words relate to. 鈥淵ou can substitute a dummy word like 鈥榝oobar鈥� for 鈥榗ake鈥� and still be able to answer the question,鈥� he says. His own approach, Aristo, attempts to quiz AI with questions taken from school science exams.
Whatever the best approach, it鈥檚 clear that tech companies like Facebook and Microsoft are betting big on human-level AI. Should we be worried? Recently the likes of Stephen Hawking, Elon Musk and even Bill Gates have warned that AI researchers must tread carefully.
LeCun acknowledges people鈥檚 fears, but says that the research is still at an early stage, and is conducted in the open. 鈥淎ll machines are still very dumb and we are still very much in control,鈥� he says. 鈥淚t鈥檚 not like some company is going to come out with the solution to AI all of a sudden and we鈥檙e going to have super-intelligent machines running around the internet.鈥�
Reference:
