杏吧原创

鈥淚 expect that after many months of using the VEST, people who are deaf will have a direct perceptual experience of hearing that is the equivalent of how you or I hear鈥� (Image: Dan Winters)

You have described the brain as 鈥渓ocked in a vault of silence and darkness鈥�, so how does it create such a rich reality for us to experience?
That鈥檚 one of the great mysteries of neuroscience: how do electrochemical messages in your brain get turned into your subjective experience of the world? What we know is that the brain is good at extracting patterns from our environment and assigning meaning to them. I鈥檓 interested in how we can plug alternative patterns into the brain and experience additional aspects of reality.

What new realities could we perceive?
We only pick up on a small fraction of signals that are going on in the world: those for which we have developed specialised sensors. There are many other signals out there 鈥� X-rays and gamma rays among others 鈥� but we鈥檙e completely blind to them. No matter how hard we try, we鈥檒l never see that part of the spectrum naturally.

But the brain is really flexible about what it can incorporate into its reality. It receives information in the form of electrochemical signals from our eyes, our nose, our skin, and works out meaning from them. Crucially, it doesn鈥檛 care where these signals are coming from; it just figures out how to use them.

I think of the brain as a general-purpose computer. Our senses are just plug-and-play devices that we have inherited through evolution. And if that鈥檚 the case, we should be able to interface any data stream into the brain and it will figure out how to deal with it.

鈥淥ur senses are just plug-and-play devices inherited through evolution鈥�

So how do you plan to patch a new data source into the brain?
We鈥檙e experimenting with what we call the versatile extrasensory transducer, or . It鈥檚 a wearable device covered with vibratory motors (pictured). When you wear the VEST, at first it just feels like strange patterns of vibrations on your torso, but the brain is really good at unlocking sensory patterns and figuring out what the input means.

(Image: James Duncan Davidson/TED)

How does a vibrating jacket allow us to experience a different reality?
Well, for example, we are trialling it with deaf participants at the moment. We capture sound from their environment and translate it into different patterns of vibrations. After a week or so our volunteers are able to figure out what鈥檚 being said using the vibrations alone. They can understand the auditory world through their skin.

Do these people actually start experiencing hearing in the same way non-deaf people do?
That鈥檚 a question we鈥檙e working on right now. My expectation is that after many months of using the VEST, people who are deaf will have a direct perceptual experience of hearing that is the equivalent of how you or I hear.

We don鈥檛 know if it will be exactly the same experience 鈥� it鈥檚 hard to communicate what a sense feels like, because our language is only capable of sharing things that we have in common. Nonetheless, the meaning of what someone is saying is directly evident to the deaf participant. It might be like learning a foreign language 鈥� at first you read it and translate it in your head to get the meaning, but with practice you just read it and understand the meaning directly.

But the brain is specialised to hear different frequencies. Can it really figure out speech from vibrations on the skin?
It seems crazy to hear via a moving pattern of touch on the skin, but ultimately, this is just translated into electrochemical signals coursing round the brain 鈥� which is all that regular hearing ever is. Traditionally, these signals come via the auditory nerve, but here they come via nerves in the skin.

We already know that the brain can figure out meaning from arbitrary signals. For example, when a blind person passes their fingers over Braille the meaning is directly evident to them. And when you read words in a New 杏吧原创 article, you don鈥檛 have to think about the details of the squiggles 鈥� the meaning simply flows off the page. In the same way, we鈥檙e demonstrating that a deaf person can extract the meaning of words coming to them by vibratory patterns.

What else might we sense using the VEST?
Here鈥檚 an example we鈥檙e working on now. We stream 5 seconds of real-time data from the internet to a person wearing the VEST. Then two buttons appear on a screen, and the person has to make choice. A second later they get a smiley face or frowny face telling them whether their choice was the right one. The person has no idea that what they鈥檙e feeling is real-time stock market data, and that the buttons represent buy or sell decisions.

What could this experiment demonstrate?
We鈥檙e seeing whether participants get better at making trade decisions without having any idea what they are doing. People often come with a lot of assumptions about the market, and we want to see whether we can circumvent that by having the brain decrypt patterns without any pollution from prior knowledge. Then it鈥檚 simply a pattern-recognition problem for the brain. Eventually we鈥檒l tell the participants what is really going on, and we are interested to know what the experience will be like for a person who wears this stock market VEST for long time. Are they suddenly going to feel a tightening in their stomach and think, 鈥淥h gosh, the oil price is about to crash鈥�?

Would they be able to describe in words what the different patterns meant to them?
Historically, people can figure out meaning from quite subtle data without ever knowing exactly what they鈥檙e doing. Take British plane-spotters for example. During the second world war, some were really good at distinguishing between British and German planes from a distance. The British government tried to get them to teach others to do it, but they were unable to. It was ineffable knowledge 鈥� they were picking out very subtle patterns without knowing exactly how.

The vibrating jacket has so many potential applications. What else have you in mind?
Yes, it鈥檚 hard to decide which possibilities to test first. We are playing with feeding the jacket a real-time sentiment analysis on Twitter, as filtered by a hashtag. Let鈥檚 say you鈥檙e a presidential candidate giving a speech. You could wear the jacket and feel how the Twittersphere is reacting as you鈥檙e going along. One of our other experiments involves working with pilots and feeding them cockpit data or drone information via the VEST.

鈥淎 politician using the jacket could feel the Twittersphere reacting to a speech鈥�

That鈥檚 intriguing. Do the pilots end up feeling like they鈥檝e become the drone, say?
Yes, it鈥檚 like extending your skin to the plane or drone, so you feel the pitch, yaw and roll. It鈥檚 a new perceptual experience that we believe will allow someone to pilot better. We鈥檙e thinking about astronauts too. They spend a lot of time looking at hundreds of monitors, so wouldn鈥檛 it be great if they could directly feel the state of the space station and know when things were changing or shifting?

Our whole lives are spent looking at little screens. In my view it鈥檚 better to experience the data rather than simply look at it.

Are there limits to how many extra senses you might acquire using wearables?
You mean, could you have a Twitter jacket and stock-market jeans? I don鈥檛 see why not. We don鈥檛 know if there are any limits to how many different things you could sense. My intuition is that the limits are distant. We鈥檝e got an enormous amount of real estate in the brain. If you lose one sense, the area of the brain responsible for it gets taken over by other senses. The brain is great at redistributing space for what鈥檚 needed and there鈥檚 plenty of room to share real estate without noticing any diminished effects elsewhere.

Does the VEST system have limitations?
The thing that defines what is possible is what we can build sensors for. If we have a good sensor, it鈥檚 trivial to convert the information it captures into vibrations in the jacket.

Could this technology could help us monitor our health?
That鈥檚 a good example of waiting for the right sensors to come along. If you want to measure blood glucose you currently do it with a spot of blood, but now people are starting to develop sensors that can image glucose through the skin. When we have this sensor, it鈥檒l be easy for us to use it to feed data to the VEST. Then you鈥檇 have a direct way of experiencing your blood sugar level.

It鈥檚 like when you sit still for too long in one position and then shift your body to get blood flowing to where it needs to go: you do that subconsciously using a stream of information from your muscles. It would be the same thing with the VEST. You鈥檙e taking something that is normally invisible and making it visible. You would directly experience blood sugar readings and think, 鈥淚 need to eat鈥�.

You mentioned in a recent TED talk that you might be able to give us 360-degree vision. How would that work?
I鈥檓 keeping that one under wraps for now, but the basic idea is that you鈥檇 wear a 360-degree camera and translate the data from that into the VEST. My intuition is that it wouldn鈥檛 end up feeling like vision 鈥� it would be more like a sixth sense, like 鈥淚 can feel someone coming up behind me鈥�.

If you could choose only one extra sense to have, what would it be?
That鈥檚 an interesting question. Right now, everything about our society is engineered around the senses that we currently have. If I were suddenly able to have ultrasonic hearing, I would hear animal calls that no one else could hear. As a nature lover that would be amazing, but I don鈥檛 know if it would be lonely in that extrasensory space if no other human joins me there.

I鈥檇 also like to explore whether the VEST can allow us to better connect with other people. Perhaps if my wife and I both wore a VEST, and used it to somehow experience each other鈥檚 emotions, that might bring us to a new level of closeness. Or perhaps it would be detrimental [laughs] 鈥� we just don鈥檛 know until we try.

When can I buy one of these things?
In about nine months. We鈥檙e working with engineers and fashion designers, and making everything open source so that anyone can input whatever data stream they like. We anticipate that the world鈥檚 hive mind will come up with some great ways to use it.

This article appeared in print under the headline 鈥淭ry my extrasensory jacket on for size鈥�