杏吧原创

What are the big challenges ahead for you?
The big challenge is unsupervised learning: the ability of machines to acquire common sense by just observing the world. And we don鈥檛 have the algorithms for this yet.

Why should AI researchers be concerned about common sense and unsupervised learning?
Because that鈥檚 the type of learning that humans and animals do mostly. Almost all of our learning is unsupervised. We learn about how the world works by observing it and living in it without other people telling us the name of everything. So how do we get machines to learn like in an unsupervised way like animals and humans?

This week, Facebook demonstrated a system that can answer simple questions about what鈥檚 happening in a picture. Is that trained by annotations made by humans?
It鈥檚 a combination of human annotation and artificially generated questions and answers. The images already have either lists of objects they contain or descriptions of themselves. From those lists or descriptions, we can generate questions and answers about the objects that are in the picture, and then train a system to use the answer when you ask the question. That鈥檚 pretty much how it鈥檚 trained.

Are there certain types of questions your AI system struggles with?
Yes. If you ask things that are conceptual then it鈥檚 not going to be able to do a good job. It is trained on certain types of questions like the presence or absence of objects, or the relationship between objects, but there鈥檚 a lot of things it cannot do. So it鈥檚 not a perfect system.

Is this system something that could be used for Facebook or Instagram to automatically caption pictures?
Captioning uses a slightly different method, but it鈥檚 similar. Of course, this is very useful for the visually impaired who use Facebook. Or, say you鈥檙e driving around and someone sends you a picture and you don鈥檛 want to look at your phone, so you could ask 鈥淲hat鈥檚 in the picture?鈥�

Right now the system just tells you the type of image it is聽 鈥� if it鈥檚 outdoors or indoors, if there鈥檚 a sunset or whatever. It then gives you a list of the things that鈥檚 found in it, but it鈥檚 not like full sentences. It鈥檚 just a list of words.

It doesn鈥檛 know the relationships between these things?
Right, and so the next generation that we have working in the lab is more like prose.

What other potential uses do you envisage for such artificial neural networks?
In biology and genomics, there is a lot of interesting work. For example, at the University of Toronto has shown that you can train a deep-learning system to emulate the biochemical machinery that reads the DNA and produces proteins. With that you can figure out the relationship between multiple particular changes in the genome and particular diseases, which are not really traceable to a single mutation but can be an assembly of things. There is going to be a lot of progress in medicine because of this kind of stuff.

Are there problems that you think deep learning or the image-sensing convolutional neural nets you use can鈥檛 solve?
There are things that we cannot do today, but who knows? For example, if you had asked me like 10 years ago, 鈥淪hould we use convolutional nets or deep learning for face recognition?鈥�, I would have said there鈥檚 no way it鈥檚 going to work. And it actually works really well.

Why did you think that neural nets weren鈥檛 capable of this?
At that time, neural nets were really good at recognising general categories. So here鈥檚 a car, it doesn鈥檛 matter what car it is or what position it is. Or there鈥檚 a chair, there are lots of different possible chairs and those networks are good at extracting the 鈥渃hair-ness鈥� or the 鈥渃ar-ness鈥�, independently of the particular instance and the pose.

But for things like recognising species of birds or breeds of dogs or plants or faces, you need fine-grained recognition, where you might have thousands or millions of categories, and the differences between the different categories is very minute. I would have thought deep learning was not the best approach for this聽 鈥� that something else would work better. I was wrong. I underestimated the power of my own technique. There鈥檚 a lot of things that now I might think are difficult, but, once we scale up, are going to work.

Facebook recently unveiled an experiment in which engineers gave a computer a passage from Lord of the Rings and then asked it to answer questions about the story. Is this an example of Facebook鈥檚 new intelligence test for machines?
It鈥檚 a follow-up of that work, using the same techniques that underlie it. The group that鈥檚 working on this has come up with a series of questions that a machine should be able to answer. Here is a story, answer questions about this story. Some of them are just a simple fact. If I say 鈥淎ri picks up his phone鈥� and then asked the question where is Ari鈥檚 phone? The system should say that it鈥檚 in Ari鈥檚 hands.

But what about a whole story where people move around? I can ask, 鈥淎re those two people in the same place?鈥� and you have to know what the physical world looks like if you want to be able to answer these questions. If you want to be able to answer questions, like 鈥淗ow many people are in the room now?鈥�, for example, you have to remember how many people came into this room from all the sentences. To answer those questions, you require reasoning.

Do we need to teach machines common sense before we can get them to predict the future?
No, we can do this at the same time. If we can train a system for prediction, it can essentially infer the structure of the world it鈥檚 looking at by doing this prediction. A particular embodiment of this that鈥檚 cool is this thing called . It鈥檚 a neural net that you feed random numbers and it produces natural-looking images at the other end. You can tell it to draw an airplane or a church tower, and for things that it鈥檚 been trained on, it can generate images that look sort of convincing.

So that鈥檚 a piece of puzzle, to be able to generate images聽 鈥� because if you want to predict what happens next in videos, you must first have a model that can generate images.

What kind of things could a model predict?
If you show a video to a system and ask, 鈥淲hat鈥檚 the next frame in the video going to look like?鈥� it鈥檚 not that complicated. There are several things that can happen, but moving objects are probably going to keep moving in the same direction. But if you ask what the video will look like a second from now, there are a lot of things that can happen that you just can鈥檛 predict, so there the system will have a hard time making a good prediction. That鈥檚 the problem we鈥檙e facing that we don鈥檛 know how to handle properly.

And what if you鈥檙e watching a Hitchcock movie and I ask, 鈥�15 minutes from now, what is it going to look like in the movie?鈥� You have to figure out who the murderer is. Solving this problem completely will require knowing everything about the world and human nature. That鈥檚 what鈥檚 interesting about it.

Five years from now, how will deep learning have changed our lives?
One of the things we鈥檙e exploring is the idea of the personal butler, the digital butler. There isn鈥檛 really a name for this, but at Facebook it鈥檚 called Project M. A digital butler is the long-term sci-fi version of M 鈥� like in the movie Her.

Image credit: Denis Allard/Agence R茅a