From imagination to canvas: AI 'reads' the brain to recreate images

A dark mountain overlooking red plains, surrounded by a strange blue shape: three French artists have presented AFP with an image generated by artificial intelligence from one of their brain activity. “A first,” the group says.

• Read also: Google announces new Gemini updates

• Read also: Someone asked AI to imagine the Simpsons in real life in the 1950s and the result was amazing

“In my head, I thought strongly about the volcano,” says Pierre Fautrel, one of the Obvious team members who participated in the experiment.

During an hour spent in an MRI machine at the Brain Institute of the Pitie-Salpetriere Hospital in Paris, he imagined several scenes, each inspired by a short description. During this process, his brain activity was recorded and then processed by a specially trained artificial intelligence.

If Pierre Fautrel admits that the work produced was not exactly what he had in mind, “it retained the semantic elements: a burning mountain with flowing lava with a landscape on a light background.”

For almost a year, he, Hugo Caselles-Dupré and Gauthier Vernier have devoted all their energy to the “From Mind to Image” project, a project that sounds like science fiction: copying an artist’s imagination thanks to generative artificial intelligence.

“Artistic ingenuity”

To achieve this, the thirty-year-olds started with an existing “open source” model, MindEye, which links visual images to brain activity, and then trained their AI in several stages.

First, by feeding an artist’s portraits and landscapes into an MRI to monitor which brain regions are activated, and using that data to feed the AI.

This attempts to reconstruct the original images from brain data alone, without seeing them during training.

The process is repeated several times for about ten hours “to create a database,” explains Hugo Caselles-Dupré.

Second, they repeated the exercise based on memories of these images only, before trying the experiment on pure imagination by reading “prompts,” which are descriptive texts written before entering the MRI.

“We have known for about ten years that it is possible to reconstruct the image seen through the activity of this visual cortex,” explains Alize Lopez-Bersem, a researcher at the Brain Institute and Inserm. “But an ‘imagined’ image, no. That is a challenge.”

It takes dozens of hours to sort through the information collected in the MRI before presenting it to the AI.

But, once this work is done, the generation of images “is done more or less instantaneously, on very powerful computers,” as Hugo Caselles-Dupré summarizes.

“Two years ago, I would never have believed this could exist,” says Charles Millerio, a neuroradiologist who is involved in the trio’s research project.

For him, this “technical prowess” is linked to a double advance in recent years: advances in medical imaging, which have made a leap forward “in terms of precision and accuracy,” and the dazzling development of generative artificial intelligence.

Surrealism of artificial intelligence

The trio of artists also drew inspiration from the Surrealist movement, which celebrates its centenary in 2024, to give their algorithmic creations a special character.

“It is a movement in which we try to move as quickly as possible between the mental image and the plastic perception,” notes Pierre Fautrel, who sees their experiment as a way to “reinterpret Surrealism.”

“For us, there are already links between art and science,” adds Hugo Caselles-Dupré, recognizing that this type of technology “can be very scary if used in the wrong way.”

In the future, the three artists hope to expand their experiments to other formats, by reconstructing audio or video. Until then, they will present their various creations in October at the Danysz Gallery in Paris.

In 2018, it stunned the art world by selling for more than €400,000 at Christie's in New York, a work presented as the first work produced by the artificial intelligence program “Edmond de Belamy”, a fictional portrait with blurred features printed on canvas.