Scientists have reconstructed short videos using only brain activity recorded from mice, effectively recreating what the animals were seeing.
The study, led by researchers at University College London and published in eLife, offers a new way to investigate how the brain turns visual input into an internal representation of the world, and may eventually make it possible to compare how different species perceive the same surroundings.
Previous human work has shown people images and films while recording brain activity with fMRI, then used those measurements to reconstruct visual information down to the level of individual pixels. The new work takes a different route, using recordings from individual brain cells rather than broader imaging signals, which can give a more detailed picture of how visual information is represented.
"We wanted to have a better way of investigating how the brain interprets what we see," said lead author Dr Joel Bauer of the Sainsbury Wellcome Centre at UCL. "The current methods of understanding what specific groups of neurons are representing are not very generalizable to situations which haven't been specifically tested for."
To rebuild the films, the team used a dynamic neural encoding model originally developed by another group for the 2023 Sensorium Competition, which predicts how individual neurons will respond while mice watch movies and takes account of other factors including the animals' movements and changes in pupil diameter.
The researchers first calculated how the neurons would be expected to behave if the mouse were looking at a blank screen, and compared that with their actual activity while it watched a film. Neural activity was measured with a microscopic imaging method that identifies active cells by detecting localised increases in calcium. Using the difference between predicted and measured activity, an algorithm gradually adjusted the pixels of an initially blank film until the reconstruction resembled the one the mouse had been shown.
The harder test came afterwards. The team recorded a mouse's brain activity while it watched a video that had never been included in the model's training data, and reconstructed a 10-second film from that activity alone. The result suggests the method was not simply recalling previously shown clips.
"Using this approach, we were able to achieve high-quality reconstructions of 10-second video clips," Bauer said. "The accuracy of the reconstructions improved with the inclusion of data from more individual neurons, demonstrating the importance of comprehensive neural data."
The reconstructions were assessed using pixel correlation, which compares corresponding pixels in the original and reconstructed films. The analysis showed only small differences in timing, though the researchers say there is considerable room to improve image resolution and the proportion of the scene that can be reconstructed.
Vision, as the team points out, is not a camera-like recording process. The brain continuously interprets, filters and modifies incoming sensory information, and the differences between the world and its internal representation are precisely what the technique is designed to expose.
It is here that the achievement needs restating in plain terms. A film was supplied to a mouse. A different film came back.
Bauer is explicit about the status of the discrepancy. "The visual processing pipeline skews and warps our representation in a way that modifies information," he said. "This deviation between reality and representations in the brain is not necessarily an error but a feature."
The Questionable Times accepts that characterisation and draws the obvious conclusion. The changes are not faults. They are decisions. The mouse was handed ten seconds of material, made adjustments to the framing, the timing and the level of detail it considered necessary, and returned a version it preferred.
The Department of Unnecessary Research notes that the animal exercised these rights over a clip it had never been shown before, without training, supervision or a contract, and that no screening has been arranged. There would be no point. Every member of the audience would hand back a different film.