ERIK WEIHENMAYER lost his sight when he was 13. Twenty years on, in Paul Bach-y-Rita鈥檚 lab at the University of Wisconsin Medical School, he caught a rolling ball, played a game of rock, paper, scissors, walked through a doorway and watched a flickering candle flame. Nothing had changed with his eyes. Instead he 鈥渟aw鈥� with his tongue.
A camera mounted on Weihenmayer鈥檚 forehead fed a signal into an electronic device that turned the pattern of light and dark into electrical pulses. The pulses stimulated an array of 144 electrodes on a grid about the size of a postage stamp, which zapped the coded image onto his tongue. At first he described the sensation as being like candy pop rocks exploding, but later he experienced something more 鈥渙ut there鈥� in the world 鈥� a sense of space, depth and shape.
Cheryl Schiltz danced for the first time in seven years after just 20 minutes wearing the same device. Normally she is unable to stand upright without holding onto something and concentrating on the stable things she can see around her, because her sense of balance has been destroyed by an antibiotic. If she tilts her head, the world spins and she stumbles.
Advertisement
But when Bach-y-Rita鈥檚 device translated signals from a sort of spirit-level-in-a-hat into patterns of pulses on her tongue, she quickly learned how to read them to substitute for her missing sense of balance. The effect persisted even when she took the device off, and lingered longer each time she tried. 鈥淚 was normal,鈥� she says, still emotional when she remembered the first time. 鈥淚 was completely normal, and I had forgotten what it felt like.鈥�
These 鈥渟ensory substitution鈥� devices are based on a technology called the BrainPort, which Bach-y-Rita describes as a kind of USB connector into the brain. They are close to commercialisation, initially for 鈥渨obblers鈥� like Schiltz, but eventually as devices for blind people too. The US military is interested in developing the system to guide pilots and divers through dark skies or murky waters, and others are looking at more frivolous uses in virtual reality and games. But in neuroscience and philosophy circles sensory substitution has attracted a great deal of interest because of what it is revealing about the brain and our senses.
The fact that Weihenmayer sensed something 鈥渙ut there鈥� and forgot the tingling on his tongue, and the way Schiltz felt completely normal, even although she was working with a different sort of sensory feedback to keep her balance, suggests that the traditional separation of the senses 鈥� sight, hearing, touch and so on 鈥� has little bearing on how we experience the world. The sense organ that picks up the information, and the way it is delivered to the brain, seem less important than the nature of the information itself.
鈥淎t first it felt like candy pop rocks exploding on his tongue. But later he felt a strange sense of depth鈥�
Some see this merely as a dramatic demonstration of the brain鈥檚 flexibility. In other words, deprived of a primary source of information such as vision, the brain turns to a less prominent source, say touch, and extracts useful information from that sense. Others, however, have taken the findings much further, going to so far as to suggest that the traditional view of how the senses work is completely wrong.
The orthodox view of sensory perception is all about building internal pictures of the world. Sensory systems extract information from outside and channel it into the brain, which builds up a representation of our environment. Sensing is therefore the passive process of picking up signals; perception is the active process of turning the signals into useful information.
This model certainly chimes with our everyday experience: we talk about our mind鈥檚 eye, of mental images, and so on. And there is some scientific evidence that it is correct. Brain imaging reveals that when people see, hear, feel, smell or taste, specialised parts of their brains respond, and the timing of the response coincides with the moment of conscious perception.
Another line of evidence comes from our ability to imagine things. Even in the absence of sensory information, we can generate images and sounds in our heads, and most researchers believe that the process of imagining mimics real sensory perception. When people imagine seeing something, their visual cortex lights up. Moreover, using a technique called TMS or transcranial magnetic stimulation, which temporarily knocks out activity in the brain regions it is targeting, Harvard neuroscientists Steven Kosslyn and Alvaro Pascual-Leone have shown that people can鈥檛 imagine things if their visual circuits are switched off.
That鈥檚 all very well, says Kevin O鈥橰egan, a psychologist with the CNRS, France鈥檚 national research centre at Ren茅 Descartes University in Paris. But he is not convinced that this proves there is a representation of the world inside your head. In fact, he argues for a profoundly different view of sensory perception, and claims that Bach-y-Rita鈥檚 sensory substitution studies support it.
O鈥橰egan鈥檚 starting point came some years back, when his interest lay mostly with eyes. He was curious as to how the world around us could feel completely stable in the face of our almost continuous eye movements, particularly large, jerky movements called saccades. O鈥橰egan reasoned that saccades must be reported back to the brain so it could compensate for them as it built up its internal image of the world.
But he could find no evidence that this actually happens. There are neural signals associated with eye movements, but they didn鈥檛 seem to be involved in building up successive visual snapshots into one big picture. Since then others have tried, and failed, to find evidence that these signals are used to compensate for eye movements in this way. But if the brain doesn鈥檛 compensate for huge shifts in eye position, how can it create a stable image of the world?
Another puzzle came from a famous experiment by Dan Simons and Christopher Chabris of Harvard University. They asked volunteers to watch a recording of a basketball game and count the passes made by one of the teams. Early on in the game a man in a gorilla suit walked slowly across the court. Despite the fact that he was visible for about 45 seconds, around 40 per cent of the viewers failed to notice him. Yet when asked to watch the game with no task in mind, they all saw it immediately (New 杏吧原创, 18 November 2000, p 28). To Simons, this is strong evidence that, despite the impression we have of seeing a complete and detailed image of the world, there鈥檚 a lot missing. We rely on the brain to fill in the blanks.
O鈥橰egan goes one step further. He suggests that the mental image is not only incomplete, it is completely absent. We don鈥檛 reconstruct the world in our mind, we merely glimpse it in fleeting fragments. 鈥淭here is no internal picture,鈥� he says. Where most researchers would argue that seeing is all about building up an internal image, O鈥橰egan has us flitting from one visual element to the next, only becoming aware of things when we need information. In this, O鈥橰egan departs radically from the traditional view of sensory perception: sensing becomes an active rather than a passive process, with potentially profound ramifications.
O鈥橰egan had been struck by an earlier version of Bach-y-Rita鈥檚 device, in which blind or blindfolded volunteers wore a larger array of electrodes taped onto the skin of their back or abdomen. One volunteer, with the array on his front, was 鈥渧iewing鈥� objects using a camera mounted on a tripod, but not getting anywhere. Out of frustration he grabbed the camera and started waving it around. When he started actively manipulating the camera in this way, something dramatic happened. He very quickly moved from merely feeling a tingling on his stomach to sensing the presence of external objects. Another volunteer, using a head-mounted camera, also suddenly felt the outside world become very real when he grabbed the zoom control 鈥� and almost fell over backwards as objects surged towards him.
What this suggests is that substituting touch information for visual information can produce a vision-like experience, but only when people actively control the camera in some way. Weihenmayer, for example, could almost see objects with his tongue. He didn鈥檛 taste them, and after a short while he didn鈥檛 feel them either. But this sensation only happened when the camera was mounted on his head, so he could move it as if he were scanning with his eyes. Similarly, blind people tapping with a cane experience open space at the end of their stick, not vibrations on their hands. They, too, are seeking information about the space around them.
Results like these have convinced O鈥橰egan that sensory perception is not about passively collecting information but actively seeking it, and noticing how the information responds to our actions. We sense the world not by soaking up information, but by taking physical actions to interrogate it. 鈥淚f the story is right, sensations are not generated in the brain,鈥� says O鈥橰egan. 鈥淭hey are things we do.鈥� Sensory substitution works because it matters less to our brain where information comes from than the manner in which we gather it.
If it is right, O鈥橰egan鈥檚 theory doesn鈥檛 just explain sensory substitution, it has philosophical implications too. In particular it suggests a solution to one of the 鈥渉ard problems鈥� of consciousness: why does seeing something feel different from touching it? The answer certainly doesn鈥檛 seem to lie in the electrical activity of the brain. Whatever sensory stimulus triggers the activity, be it touch, taste, sight or sound, the information is translated into electrical pulses. And no one has ever been able to find anything unique about these pulses, or where they are sent to in the brain, to explain why they produce different sensations.
That spongy feeling
O鈥橰egan believes that his 鈥渟ensorimotor鈥� theory might provide an answer. Maybe, he says, touch, taste, sight and sound feel different because we have to perform different actions to collect the information.
Take, for example, the softness of a sponge. Where does the feeling of softness come from? No one has ever found a neural mechanism or specific part of the brain that exclusively lights up when you feel something soft. That, says O鈥橰egan, is because there isn鈥檛 one. Working with philosopher Erik Myin at the University of Antwerp in Belgium, he has proposed that the feeling of softness comes from how you go about seeking information about the sponge. When you press the surface, it gives way. This is a different action from touching a sharp or hard surface, or a liquid.
While the theory seems to make sense for touch, or the difference between seeing and touching, what about the hardest 鈥渉ard problem鈥� of all, the sensation of different colours? How can we explain 鈥渞edness鈥� or 鈥済reenness鈥� in terms of different actions? To complete his theory, O鈥橰egan needed to find unique actions or activities that are associated with perceiving different colours.
It seemed an impossible task, but O鈥橰egan and colleague David Philipona from the Sony Computer Science Laboratory in Paris were in for a surprise. When they looked at the physical properties of coloured surfaces they found fundamental differences in the way the different colours interact with light. In classical models, reflections from surfaces are the sum of two sources: one that behaves like the reflection from a matt surface, and another that behaves like the reflection from a sheet of glass laid over the matt surface. As we move our eyes, both types of reflection change their spectral composition, and the relationship differs according to colour you are looking at.
O鈥橰egan suggests that as we move our eyes over a coloured surface, we detect something of this change in relationship. And by that we experience colour. The key point as far as perception goes is what happens when we probe the environment: it鈥檚 not the brain activation itself that gives the colour. The researchers have found that primary colours produce particularly distinctive changes, which may explain why they are universally recognised as special.
鈥淲hy does seeing something feel different from touching it? It doesn鈥檛 seem to be down to the electrical activity of the brain鈥�
Already, O鈥橰egan鈥檚 ideas have doubters. Bach-y-Rita thinks the explanation for sensory substitution lies with the remarkable flexibility of the brain. There are multiple pathways from all the senses to all the different sensory areas in the brain, he says. If you lose the main input from the eyes to the visual cortex, say, weaker pathways from the skin, ears, tongue and so on take over. This is what happens in the brains of Braille readers, who recruit their visual cortex when feeling the forms of the letters.
It may not be long before we know who is right, as O鈥橰egan and colleagues are busy thinking up testable predictions of their sensory substitution theory. One such prediction is that it should be possible to make the substitution feel more convincing by making the information-gathering action 鈥渕imic鈥� the original as closely as possible in the new medium.
To that end, O鈥橰egan and colleague Malika Auvray have rigged up a video camera that represents the visual world in sound. Brighter objects become louder sounds, objects high up in the visual field are represented by high pitches and object low down by low pitches, while lateral position is represented with stereo sound. It is a little hard to imagine, but say the camera was looking at a light bulb in the centre of the field of view, you would hear powerful noise made up of a limited range of pitches centred in space. With a horizontal strip light you鈥檇 hear a smaller range of pitches over a wider space. As you move the camera, the sound changes.
In preliminary tests with a similar device designed by engineer Peter Meijer from Eindhoven in the Netherlands, the signals took a little getting used to. But after a couple of hours of feedback either from touching or being told what they were viewing, people were able to recognise objects by their sound. They could tell plants from statues and crosses from circles. But they weren鈥檛 fooled into thinking they were seeing. O鈥橰egan鈥檚 prediction is that the more they can make the sound information follow the rules of visual images, the more like seeing it will feel. For example, Meijer鈥檚 system has a delay between moving the camera and hearing the sound. Another simple tweak would be to cut off the sound each time the subjects blink, which is exactly what happens to our visual world, though we scarcely notice it.
Perhaps one day blind people will play rock, paper, scissors in stereo surround sound. And if O鈥橰egan is right, they could feel almost as if they are seeing. Now that really would be a sensation.
