Reading the mind
The end for eye-spy?
A mind-reading machine that could decipher what someone is seeing might soon be possible. Such a device would enable images represented in the brain to be shown on a television screen, and would perhaps even be able to take pictures from dreams.
Imagine a general brain-reading device that could reconstruct a picture of a person's visual experience at any moment in time," says Dr Jack Gallant, head of the research team and assistant professor in the Department of Psychology at the University of California, Berkeley.
In March, the Gallant Lab in Berkeley released groundbreaking research, taking a giant scientific and technological leap forward with their study on interpretation of brain activity. A team of four researchers produced an analytical tool that can accurately identify images from brain activity patterns.
The study, which has created a buzz in the scientific community, presents a new way of doing functional magnetic resonance imaging (fMRI) research, and a new way of decoding the brain. And, while researchers can't currently reconstruct a photograph seen by test subjects without knowing the images that the photograph is from, they expect to in the future.
"Our data suggest that there might potentially be enough information in brain activity signals, measured using fMRI, to do this in the future," says Kendrick Kay, a graduate student in Berkeley's psychology PhD program, one of the study's test subjects and first author of the study. "But given that we can't reconstruct a picture that the person saw, it should be obvious that we also can't reconstruct dreams or visual imagery at this time."
However, since neuro-scientists generally assume that all mental processes have a concrete neurobiological basis, as long as they have good measurements of brain activity and good computational models of the brain, it should be possible to decode the visual content of mental processes like dreams, memory, and imagery.
"The computational models in our study provide a functional account of visual perception," Kay says. "It is currently unknown whether processes like dreaming and imagination are realised in the brain in a way that is functionally similar to perception. If they are, then the techniques developed in our study should be directly applicable."
Mechanisms of vision
The research was carried out at the University of California by scientists Kay, Ryan Prenger, Thomas Naselaris and Jack Gallant. Gallant's lab has studied the neural mechanisms underlying vision and visual perception, particularly perception of natural scenes and selective attention, for more than ten years. The team uses four approaches to address these topics: fMRI; studies of neuro-logical patients with specific brain lesions; neurophysiology; and quantitative computational modeling.
The Gallant Lab focuses on several specific aspects of visual perception. First, they examine the role of an important human extrastriate visual area, V4, in scene analysis. These experiments involve studies of neurological V4 lesion patients and functional neuroimaging.
"We also use neurophysiological methods to investigate the function of early and intermediate cortical visual areas during natural vision," Gallant says. "These studies address both how the visual system responds to complex scenes during natural vision, and how it is affected by top-down processes, such as scene segmentation and grouping."
Another research method uses quantitative nonlinear system identification methods to objectively characterise the way natural scenes and complex objects are represented in intermediate visual areas, and to determine how these representations are modulated by attention. These experiments involve functional neuroimaging, neuro-physiology, and computational modeling of visual processing.
Brain activity measurement
The two-year study, which began in November 2005, was originally intended as an effort to build computational models of processing in the visual system of the brain. Predictive models are the gold standard of computational neuroscience and are critical for the long-term advancement of brain science and medicine.
It was during this process that researchers realised that this kind of modeling is closely related to efforts to decode the meaning of brain activity patterns. The study began with two of the scientists viewing a total of 1,750 images, chosen at random, while undergoing an fMRI brain scan. Each subject participated in five MR scanning sessions in which they were required to lie still in an MR scanner while viewing a series of photographs. Each session lasted about two hours.
The data obtained were used to construct mathematical descriptions of voxels in the brain - three-dimensional pixels that represent the blood flow response in a small region of the brain. The encoding model was implemented using computer code. Fitting the model to the data was computationally intensive since there were thousands of individual voxels.
"The computational model can predict the pattern of brain activity that would be elicited by any arbitrary photograph, including photos that were not in the initial set used to build the model," Gallant explains. Then each of the research subjects viewed a new set of 120 images - which were random and completely distinct from the first set of photos - while having their brains scanned. Combining the new data with the computational model created earlier, the scientists were able to identify, from brain activity alone, which images had been seen at each point in time.
Researchers picked out which of 120 images two volunteers had been viewing with 92 per cent and 72 per cent accuracy, compared with 0.8 per cent accuracy when left to chance.
"We simply look through the list of predicted activities to see which one is most similar to the observed activity, and that's our guess," Gallant says. "We were all pretty amazed at how well this works. It implies that there should be enough information in these brain activity signals to do much more."
Previous research has shown that fMRI can pick out brain activity associated with viewing different images, but only at a basic level (the category of the image, like 'face' or 'house') and only if brain activity associated with those had been previously measured. Now, it has been shown that brain imaging can uncover complex, arbitrary images.
To test how feasible it might be for a machine to respond to brain activity triggered by just one event, the experiment was conducted again using data from single brain scans. The results - 51 per cent and 32 per cent accuracy for each of the volunteers - were good enough to suggest that it may be possible to decode perceptual experiences in real-time.
According to Kay, the accuracy levels are much higher than what other researchers might have expected to achieve using fMRI. Researchers ran testing on a third subject not long after data collection for the first two. Results for the third subject were similar in performance to the second one.
Expanding the decoding system out to more images would lower the accuracy level but still produce a relatively high success rate. The team estimates that if they used one billion images (roughly the number on Google), it would have a success rate of about 20 per cent. With that many images, Gallant says, the software is close to doing true image reconstruction, working out what you are seeing from scratch.
The model isn't perfect, so the decoder does make mistakes. When the decoder selects the wrong image, it tends to choose one that is structurally similar to what the person actually saw. For example, if the person actually saw a picture of a man standing in front of a house, the decoder might choose a picture of a statue placed in front of a museum.
"In general, as the set of potential images grows larger, more of them will be structurally similar to one another, so there are more chances for mistakes," Kay says. But researchers say that with enhanced computational models of the visual system and improved measurements of brain activity (such as higher signal-to-noise ratio and higher spatial resolution), accuracy will improve.
"There is no reason we shouldn't be able to solve this problem," Gallant says. "That's what we are working on now."
Some useful applications
This groundbreaking research opens the door to many future applications. In addition to their value as a basic research tool, brain decoders could be used to aid in diagnosis of diseases such as stroke and dementia, assess the effects of therapeutic interventions including drug and stem cell therapy, or used as the computational heart of a neural prosthesis. They could also be used to build a brain-machine interface (BMI).
"Decoding visual content is conceptually related to the neural-motor prosthesis BMI work that many labs are doing these days," Gallant says. "The main goal in that work is to build a decoder that can be used to drive a prosthetic arm or other device from brain activity."
As Gallant pointed out, there are significant differences between sensory and motor systems that impact the way a BMI system would be implemented. But ultimately, the statistical frameworks used for decoding in the sensory and motor domains are very similar, suggesting that a visual BMI might be feasible.
"This is an ongoing effort which will require many more studies," Gallant says.
Can Big Brother read your mind?
While this new research may have extensive benefits for the medical field, it also raises questions about a person's brain being interrogated against his will.
The Berkeley study suggests that fMRI-based measurements of brain activity contain much more information about underlying neural processes than might have been expected. In fact, as previously noted, so much information is available in these signals that one day it may even be possible to reconstruct the visual contents of dreams or visual imagery.
However, the scientists say that currently their techniques have only been applied to visual images and, to date, the experiments rely on cumbersome MRI scanning equipment. Also, the software decoder itself has to be adapted to each individual during hours of training while in the scanner.
"The general idea of being able to interpret brain activity has much potential," Gallant says. "However, we believe strongly that no one should be subjected to any form of brain-reading process involuntarily, covertly, or without complete informed consent." So, what is the likelihood of future legal systems using a mind-reading machine?
According to the Berkeley researchers, using this technique on citizens brings with it most of the problems that are already known regarding eyewitness testimony, including poor or biased memories, inadvertent coaching, and evidence rehearsal.
"Any brain-reading device that aims to decode stored memories will inevitably be limited not only by the technology itself, but also by the quality of the stored information. An accurate read-out of a faulty memory only provides a misleading picture," Gallant says.
"Therefore, any future application of this technology in the legal system will have to be considered with extreme caution." The researchers warn about potential privacy issues when scanning techniques improve.
"It is possible that decoding brain activity could have serious ethical and privacy implications downstream in, say, the 30- to 50-year timeframe," Gallant says.
He points out that brain decoding, like any technology, has potential for use and misuse. One can make an analogy with the field of genetics: an understanding of the genetic code is critical for progress in medicine and disease treatment, but it can also be used to develop biological weapons.
"Better understanding of brain function would likely also have a wide range of potential technical applications, most positive, but a few potentially negative," he says.
"I believe strongly that society must develop safeguards to minimise the negative potential effects of all technologies."
A silent cell phone?
The National Aeronautics and Space Administration (NASA) is now able to know what people are going to say... before they say it.
Dr Chuck Jorgensen, chief scientist for neuro engineering at NASA's Ames Research Center, has led research on computerised human, silent, subvocal speech. He is an expert on subvocal speech and emerging trends in bioelectronic device interfaces.
The subvocal-speech process works when small button-sized sensors on the neck are able to capture electromyographic (EMG) signals before the actual words are spoken. The sensors gather nerve signals, send them to a processor, and then to a computer program that translates them into words.
"Biological signals arise when reading or speaking to oneself with or without actual lip or facial movement," Jorgensen says.
Subvocal speech is a form of human communication based on neural impulses that control the vocal tract rather than sounds. These EMG signals arise from commands sent by the brain's speech center to articulators that enable production of audible words. The subvocal speech technology intercepts EMG signals before a sound is produced and then infers corresponding words or actions to those captured signals.
Basically, a person using the subvocal system thinks of phrases and talks to himself so quietly it cannot be heard, but the tongue and vocal cords do receive speech signals from the brain.
In NASA's first experiment, scientists trained special software to recognise six words (stop, go, left, right, alpha, and omega) and ten digits (zero through nine) that the researchers repeated subvocally. Initial word recognition results were an average of 92 per cent accurate. Silently speaking these words, scientists conducted simple searches on the Internet by using a number chart representing the alphabet to control a Web browser program.
"We took the alphabet and put it into a matrix — like a calendar. We numbered the columns and rows, and we could identify each letter with a pair of single-digit numbers," Jorgensen says. "So we silently spelled out 'NASA' and then submitted it to a well-known Web search engine. We electronically numbered the Web pages that came up as search results. We used the numbers again to choose Web pages to examine. This proved we could browse the Web without touching a keyboard."
A second demonstration successfully controlled a mechanical device using a simple set of commands. Jorgensen 'drove' a simulated Mars rover in real-time, using sub vocal commands.
This technology is being considered for a number of different applications in both the military and civilian world, including for soldiers in a warzone, first respondants to a fire, and astronauts in space. NASA has demonstrated subvocal communication in pressurised fire suits and diving equipment.
"We have done remote robot control subvocally, most recently for the show 'Modern Marvels' and later this summer for Discovery Channel's 'Future World' show," Jorgensen says. "We have also demonstrated analog robot control rather than discrete word control, and there is a patent submitted on this work called 'Clench Codes.'
NASA is also exploring application to wheelchairs, as well as a number of military programs.
"A civilian application of this technology would be a silent cell phone," Jorgensen says. "No longer would we hear people talking loudly on their phones in a restaurant."
The Computing, Information, and Communications Technology Program, part of NASA's Office of Exploration Systems, funds the subvocal word-recognition research.
|To start a discussion topic about this article, please log in or register.|
"Africa is abundant with engineering opportunity. We look at some of the projects and the problems."
- DECC-EDF makes yet another attempt to fund 3rd Generation Nuclear at any cost [12:04 pm 25/05/13]
- UK just six hours from running out of gas in March [09:02 pm 24/05/13]
- Ideas for a final year university project [05:55 pm 24/05/13]
- Fourth Generation Nuclear: Molten Salt Reactors [10:39 am 24/05/13]
- LED bulb efficiency - its all about the drivers not the LEDs? [09:52 am 24/05/13]
Tune into our latest podcast