Reconstructing and Recording Images the Mind Sees

Anthony D’OnofrioBME 281 Section 2, Tuesdays

October 23, 2012

What are DreamsBiggest mystery• Scientists do not know– No purpose at all

• Mind’s just bored– Making connections loosely (Letting our mind wander)

• Powered by emotion– Fright, Sadness, Happiness, Freedom

• Dreams have been seen as a connection to the unconscious.– Controlled by imagination and previous memories as basis

of dreams• Dreaming occurs mainly during REM sleep

REM sleep• Rapid Eye Movement

– Includes eye rapidly moving but also low muscle tone and a rapid, low-voltage EEG (electroencephalogram)• Recording of electrical activity along the scalp. • EEG measures voltage fluctuations resulting from current within

the neurons of the brain.• REM sleep in adult humans typically occupies 20–25% of

total sleep– 1–2 hours of a night's sleep. – 4-5 periods

• Occurs close to morning.

– Short at the beginning of the night and longer toward the end.• It was identified and defined by Nathaniel

Kleitman, Eugene Aserinsky, and Jon Birtwell in the early 1950s.

What if We Could Record Images?

• The Gallant Lab at UC Berkeley– Dr. Gallant and Dr. Nishimoto– Main goal is to understand what portions of the brain

become stimulated by a certain image – Develop a computer model that records those brain

activities evoked and reconstructs them.– Whatever image we see in our mind, the computer

model will read it, memorize it, and reconstruct it.

• Prototype– It works

fMRI machine

• Functional Magnetic Resonance Imaging• Instead of using EEG to read the brain waves– More accurate, computational representation

• Procedure uses blood flow to detect connections between brain activation and task subject performs during the scan.– Specific to analyzing brain activity

• Watch several hours of different movie clips

Computer Model: Looking at the Brain

• Reads brain’s visual cortex• Determines the function of each brain section of

the cortex• The computer will look at brain activity and

notice certain patterns (shape, color, motion) that are similar in each movie– Example border color

• Certain part of brain gets stimulated, when blue border appears

• Computer reads that

Computer Model: The Data Collecting

• Each neuron acts like a filter • Takes a visual stimulus as input, and produces a response

as output.

• Computer memorized all the outputs and data collected from brain activity

• Builds a dictionary from that– Dictionaries create definitions for each neural output

transmitted

The Experimental Process

• Have same subject go into fMRI – Watch new set of movie clips • Computer does not know

• Computer reads the brain activity• Deciphers the brain activity using the

dictionaries created • Matches similar clips randomly downloaded

from YouTube.

Recap and Conclusion • Record brain activity while subject watches several hours of movie

trailers.• Build dictionaries to translate between shapes, edges and motion in

the movies and measure brain activity. • Record brain activity to a new set of movie trailers that will be used

to test the quality of the dictionaries and reconstructions.• Build a random library of 18,000,000 seconds of video downloaded

at random from YouTube (that have no overlap with movies subjects saw in fMRI).

• Put each of these clips through the dictionaries to generate predictions of brain activity. – Select the 100 clips whose predicted activity is most similar to the

observed brain activity. – Average those clips together. – This is the reconstruction.

• Surreal

https://www.youtube.com/watch?feature=player_embedded&v=nsjDnYxJ0bo

Future• More accurate

– Mapping and understanding more about which neurons are stimulated

• Record dreams– Psychologically analyze people’s

dreams• Schizophrenia• Post Traumatic Stress Syndrome

• Put images into the head– Outputs would turn into inputs

• Criminal Court rooms – Unaccurate– Ethical Concerns

• Seeing memories• Help people in coma respond,

stroke patients, dementia– People who can’t talk, talk

through their images in head

Work Cited Page• Anwar, Yasmin. "Scientists Use Brain Imaging to Reveal the Movies in

Our Mind." UC Berkeley News Center. Media Relations, 2011. Web. 18 Oct. 2012. <http://newscenter.berkeley.edu/2011/09/22/brain- movies/>.

• "Brain Scanner Can Record Your Dreams on Video." DVICE (Syfy). The SyFy Online Network, 2011. Web. 18 Oct. 2012. <http://dvice.com/archives/2011/09/brain-imaging-c.php>.

• "Dream Recording Device Around the Corner?" Neurogadget.com. N.p., 2011. Web. 18 Oct. 2012.

<http://neurogadget.com/2011/09/28/dream-recording-device-around-the-corner/2708>.

• Gallant, Jack L., and Shinji Nishimoto. "Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies -

GallantLabUCB." Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies -GallantLabUCB. Google Sites, 2011. Web. 18 Oct. 2012. <https://sites.google.com/site/gallantlabucb/publications/nishimoto-et-al-2011>.

Work Cited Page• Gallant, Jack L., and Shinji Nishimoto. "Reconstructing Visual Experiences

from Brain Activity Evoked by Natural Movies." Current Biology 21 (2011): 1641-646. Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies. Elsevier, 2011. Web. 18 Oct. 2012

• Ghosh, Pallab. "Dream Recording Device 'possible' Researcher Claims." BBC News. BBC, 27 Oct. 2010. Web. 18 Oct. 2012. <http://www.bbc.co.uk/news/science-environment-11635625>.

• Moisse, Katie. "UC Berkeley Scientists 'See' Movies in the Mind." ABC News. ABC News Network, 22 Sept. 2011. Web. 18 Oct. 2012. <http://abcnews.go.com/Health/MindMoodNews/scientists- youtube-videos-mind/story?id=14573442>.

• "Recording Dreams Science World." Recording Dreams « Science World. Science World, 2011. Web. 18 Oct. 2012. <http://blogs.voanews.com/science-world/2011/09/22/recording-dream/>.