Read more: 鈥�Mind readers: Eavesdropping on your inner voice鈥�
Editorial: 鈥�Brain-eavesdropping tech can鈥檛 steal your thoughts鈥�
When you read this sentence to yourself, it鈥檚 likely that you hear the words in your head. Now, in what amounts to technological telepathy, others are on the verge of being able to hear your inner dialogue too. By peering inside the brain, it is possible to reconstruct speech from the activity that takes place when we hear someone talking.
Advertisement
Because this brain activity is thought to be similar whether we hear a sentence or think the same sentence, the discovery brings us a step closer to broadcasting our inner thoughts to the world without speaking. The implications are enormous 鈥� people made mute through paralysis or locked-in syndrome could regain their voice. It might even be possible to read someone鈥檚 mind.
Imagine a musician watching a piano being played with no sound, says at the University of California, Berkeley. 鈥淚f a pianist were watching a piano being played on TV with the sound off, they would still be able to work out what the music sounded like because they know what key plays what note,鈥� Pasley says. His team has done something analogous with brain waves, matching neural areas to their corresponding noises.
How the brain converts speech into meaningful information is a bit of a puzzle. The basic idea is that sound activates sensory neurons, which then pass this information to different areas of the brain where various aspects of the sound are extracted and eventually perceived as language. Pasley and colleagues wondered whether they could identify where some of the most vital aspects of speech are extracted by the brain.
The team presented spoken words and sentences to 15 people having surgery for epilepsy or a brain tumour. Electrodes recorded neural activity from the surface of the superior and middle temporal gyri 鈥� an area of the brain near the ear that is involved in processing sound. From these recordings, Pasley鈥檚 team set about decoding which aspects of speech were related to what kind of brain activity.
Sound is made up of different frequencies which are separated in the brain and processed in different areas. 鈥淪imply put, one spot [of neurons] might only care about a frequency range of 1000 hertz and doesn鈥檛 care about anything else. Another spot might care about a frequency of 5000 hertz,鈥� says Pasley. 鈥淲e can look at their activity and identify what frequency they care about. From that we can assume that when that spot鈥檚 activity is increasing there was a sound that had that frequency in it.鈥�
Frequency isn鈥檛 the only information you can extract. Other aspects of speech, such as the rhythm of syllables and fluctuations of frequencies are also important for understanding language, says Pasley.
鈥淭he area of the brain that they are recording from is a pathway somewhere between the area that processes sound and the area that allows you to interpret it and formulate a response,鈥� says , an auditory researcher at the University of Oxford. 鈥淭he features they can get out of this area are the ones that are really important to understanding speech.鈥�
Pasley鈥檚 team were able to correlate many of these aspects of speech to the neural activity happening at the same time. They then trained an algorithm to interpret the neural activity and create a spectrogram from it (see diagram). This is a graphical representation of sound that plots how much of what frequency is occurring over a period of time. They tested the algorithm by comparing spectrograms reconstructed solely from neural activity with a spectrogram created from the original sound.
They also used a second program to convert the reconstructed spectrogram into audible speech. 鈥淧eople listening to the audio replays may be able to pull out coarse similarities between the real word and the constructed words,鈥� says Pasley. When New 杏吧原创 listened to the words, they could just about make out 鈥淲aldo鈥� and 鈥渟tructure鈥�. However, its fidelity was sufficient for the team to identify individual words using computer analysis.
Crucial to future applications of this research is evidence that thinking of words promotes activity in the brain that resembles hearing those words spoken aloud.
鈥淲e know that for much of our sensory processing, mental imagery activates very similar networks,鈥� says at the University of Li猫ge, Belgium. We need to be able to show that just thinking about the words is enough, which would be useful in a medical setting, especially for locked-in patients, he says.
鈥淚t鈥檚 something we鈥檇 like to pursue,鈥� says Pasley. His isn鈥檛 the only team that is hoping to produce sound from thoughts. at Boston University, Massachusetts, has interpreted brain signals that control the shape of the mouth, lips and larynx during speech to work out what shape a person is attempting to form with their vocal tract and hence what speech they are trying to make. They have tried out their software on Erik Ramsey, who is paralysed and has had an electrode implanted in his speech-motor cortex. At present the software is good enough to produce a few vowel sounds but not more complex sounds.
Laureys has also been working on ways to distinguish brain activity corresponding with 鈥測es鈥� and 鈥渘o鈥� answers in people who cannot speak. Other neuroscientists have been devising similar ways to communicate with patients in vegetative states, by monitoring brain activity using fMRI.
鈥淥f course, it would be much better if one could decode their answers, their words and thoughts,鈥� he says.
Auditory information is processed in a similar way in all of us so in that sense the new model can be applied to everyone, says Pasley, but the settings certainly need to be tuned to the individual because of anatomical differences across brains. He says the training process is short and involves listening to sounds. After the model is trained it can be used to predict speech 鈥� even words it has not heard before.
Pasley says the technology is available to turn this idea into a reality. 鈥淭he implants transmit the recorded signals to a decoder that converts the signals into movement commands, or in our case, speech.鈥� He wants to develop safe, wireless, implantable interfaces for long-term use.
No one will be reading our thoughts any time soon, says Jan Schnupp, also at Oxford, as only a small number of people having essential brain surgery will have these devices implanted.
But for those in need of a voice the work is a positive step. 鈥淚t adds to the fascinating literature of decoding thoughts which is getting more and more precise,鈥� says Laureys. 鈥淲e have reason to be optimistic.鈥�
We don鈥檛 know if speech from thoughts is possible yet, says Pasley 鈥� but he鈥檚 upbeat. 鈥淚t鈥檚 certainly the hope.鈥�
Journal reference: PLoS Biology, DOI: 10.1371/journal.pbio.1001251
