Imagine what it would be like if the computer could show you what you want.
Sound too far away? Recently, however, four scientists from Kyoto university in Japan have made such a vision a step closer. Shen Guohua, Tomoyasu Horikawa, Kei Majima and Yukiyasu Kamitani on BioRxiv published their research results of using AI to decode human thinking.
Machine learning has previously been used to study brain scans (MRI). To show humans some simple images, such as black and white letters and simple geometric shapes, AI can restore the image seen by the human eye according to the signal of brain activity. The research was published in the journal annals. But this time, the scientists in Kyoto developed a new technique that USES the deep neural networks of artificial intelligence to "decode" ideas. The new technology allows scientists to decode more complex layered images, with multiple layers of color and structure (such as a bird, or a man in a cowboy hat). "We've been looking at ways to reconstruct or reconstruct images by looking at human brain activity." One of the scientists Kamitani said, "before we assume that the image is made up of pixels or simple shapes, but it is well known that the brain processes visual information when layer to extract the characteristics of the different level of complexity of different or other information."
The new AI research will allow computers to detect objects, not just binary pixels. "These neural networks constitute an AI model that can be used to represent the hierarchical structure of the human brain," says Kamitani. In the 10-month study, the researchers presented three subjects with natural images of a period of time (such as a bird or a person's photo), abstract geometric shapes or letters.
The first row is the picture that the subjects saw, and the next few rows are the reconstructed images generated by the AI based on the different images of the three subjects.
In one part of the test, the researchers measured the brain activity of the participants when they were watching one of 25 images. In another part of the test, the recording of brain activity was performed after the participants recalled the images. After the brain activity was measured, the computer reverse-engineered the information it had collected and generated images that were in the minds of the participants.
The flowchart shown below is produced by a team of researchers from the Kamitani laboratory at Kyoto university, and is a step by step to break down how the visual image is encoded.
The following two charts show the results of computer reconstruction of brain activity when subjects viewed images of natural images or letters.
Reconstruction of other natural landscape images generated by DGN technology. The image of the black box and gray frame is the original image, and the image is reconstructed with the DNN network based on the VC activity data. The three reconstructed images are from the three subjects.
All the reconstruction results of the alphabetic sequence.
In another group of subjects, scientists had a new breakthrough when they looked at the brain-wave measurements that were carried out during the recall process. "It's not the same as before," says Kamitani. "we can reconstruct the blurry images of people in the brain in the process of remembering." As the chart below shows, when trying to decode the brain signals that people recall from images, the AI system is not as good as it was. That's because, compared to natural images or letters, it's harder for humans to remember exactly the image of a cheetah or fish.
Reconstruction of some images with complex images. The image in the lower right corner, as a blank control, is reconstructed from the brain signals that were generated during the test process that was not shown during the test.
"The result is that the brain is less activated at that time." Kamitani explained that he also noted that as the accuracy of the technology was increasing, the potential for potential applications of the research was limitless. People can draw pictures or make artistic creations simply by imagination. Your dreams can be drawn by the computer; It can help improve their care by visualizing the hallucinations of the mentally ill. And the brain-machine interface may one day be able to achieve direct communication between people and images in their minds, rather than just language.