SYNESTHESI A Multimodal Communication 2008 Ingvar Lind sight taste smell hearing touch.

SYNESTHESI AMultimodal Communication 2008

Ingvar Lind

sight

taste

smell

hearing

touch

Synesthesia - Multimodal Communication 2008 Ingvar Lind

Muldimodality in communication• Different modalities are used simultaneously in

multimodal communication.• The modalities interact in certain ways.• There are production and reception modalities.• The receiving modalities are received through the

senses, which translate physical properties in the world to mental modules.

• The mental modules are both experiences and unconcious simulation processes that together combine to the world we know and feel.

• The modalities are combined in an indirect fashion in multimodal communication. In synesthesia, the combination of modalities is direct and unchanging.


What is Synesthesia?


Definitions of Synesthesia:


Definitions of Synesthesia:• A condition in which stimulation of one sense also

evokes another sensory pathway in the brain.



evokes another sensory pathway in the brain.• It is involuntary.



evokes another sensory pathway in the brain.• It is involuntary.• It is automatic (consistent and permanent).



evokes another sensory pathway in the brain.• It is involuntary.• It is automatic (consistent and permanent).• It can be cross-modal and inter-modal (=cross-sub-

modal).




modal).• It is highly memorable.




modal).• It is highly memorable.• It is paired with emotion/mood.




modal).• It is highly memorable.• It is paired with emotion/mood.• It is most often unique to one person, but similarities

exist between the synesthetic types.





exist between the synesthetic types.• The synesthetic phenomena is an extension of

perception rather than a limitation, as in color blindness.





exist between the synesthetic types.• The synesthetic phenomena is an extension of

perception rather than a limitation, as in color blindness.• It is directional, from one modality to another.


Examples of Synesthesia:• numbers -> shapes/colors/textures/movements

Extraordinary people - Daniel TammetPart 2: Beginning 3 min.

http://www.youtube.com/watch?v=ydiBnm1ejhM&NR

Part 3: Beginning 3 min.

http://www.youtube.com/watch?v=4BxhYPKXXnQ


Examples of Synesthesia:• graphemes (letters) -> sounds

Sounds of the Alphabet, 1 min.

http://www.youtube.com/watch?v=XOuU0ppCcn8


Examples of Synesthesia:• drugs -> synesthesia

Synesthesia caused by drugs, 2 min.

http://www.youtube.com/watch?v=XZ7jqgEYSLU


Examples of Synesthesia:• music -> colors• graphemes (numbers and letters) -> colors• taste -> shape• speech sound -> taste• time units -> positions in space

Seeing Life in Colors: Crosswired Senses, ABC NEWS 3 min.http://www.youtube.com/watch?v=KApieSGlyBk


Examples of simulated Synesthesia:• sound -> moving shapes• music -> moving lines

Synthesizer Synesthesia, 1.22 min.

http://www.youtube.com/watch?v=l4_T5VYMHqc

Toccata and Fugue in D minor, 8.34 min.

http://www.youtube.com/watch?v=ipzR9bhei_o


Colors

Tastes

Smells

Touch

Temperature

Vibration, skin

Shape in 3D

Movements

Pain

Positions

Synesthesia Modalities and Modules

Concepts (language)

Concepts (mathematics)

Emotions

Music

Sound

Time units

Graphemes

Phonemes

Faces

Bodies

etc.


Categorical perception and Synesthesia

Categorical perception is perception of a distinct number of kinds, opposed to a perception of continuous dimensions in experience. The hue circle is an example of a continuous dimension that perception labels with a small number of hue categories: red, orange, yellow, green, blue, purple.

Categorical perception is a necessare function of perception to scale down the complexity of information from the world into a relatively small number of labels.

Synesthesia are often connections between categories of modalities rather than between continuous dimensions. This suggests that language, or other more abstract brain functions has a major role in synesthesia.


Synesthesia Questions

Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)?




1. Cross-sensory metaphores (blue music, blue feelings)





2. Geometrical properties are common between the modalities (2D structures, 3D-structures (places, movements, textures and directions in a room in vision, auditory and tactile modalities)






3. The Bouba-Kiki effect






3. The Bouba-Kiki effect (unknown words suggest certain shapes)

4. Temperature and color (blue feels cold, red feels warm)



The mind-body-dualism




1. The mind studied with introspection, interviews and psychophysics. Only the mental functions are considered.




1. The mind studied with introspection, interviews and psychophysics. Only the mental functions are considered.

2. The body as controlled by a computerlike brain, with neural network models explaining what goes on between input and output.

Can the two perspectives be fused into one? Can synesthesia show us how the mind works?



Theories of synesthesia

1. Module theory

Cognitive psychology: modules have separate functions in the brain. Some sensory modules are linked in synesthesia.



Theories of synesthesia

1. Module theory

Cognitive psychology: modules have separate functions in the brain. Some sensory modules are linked in synesthesia.

2. Matching theory

Cross-modal matching, one modality can process information patterns that another modality process: size, shape, intensity, texture, quality, time, space.

”Synesthesia”-model for a connection between color and sound

Vertical direction: Brightness

Central vertical arrow: neutral colors (black, gray, white, shiny)

Horizontal circle: Hues (red, orange, yellow, green, blue, purple, red)

Vertical direction: mental Volume of sound (or maybe Pitch)

Central vertical arrow: neutral vowels (central in the formant space)

Horizontal circle: Vowel sounds (timbre) (a, e, i, y, u, o, a)

1-1 mapping

Dotted line: strength of Hue or strength of Vowel


An experiment in Synesthesia

Make a synesthetic one-to-one connection model between the 2D vowel space and the 2D hue space, using the computer programs Praat and ColorSpace. What is the most natural connection between the two? Explain why.

Vowel space:Praat (Macintosh, Pc, Linux, SGI)http://www.fon.hum.uva.nl/praat/Praat objects – New – Sound – Create Sound from Voweleditior

Hue space:ColorSpace (Windows program, similar to Macintosh Color Picker) http://www.codeguru.com/cpp/w-d/dislog/commondialogs/article.php/c1861View – Color Space (Color Picker)


References

Sean Day, “Trends in colored letter synesthesia”, research on colored letters:http://web.mit.edu/synesthesia/www/trends.htmlSensequence, a huge webpage on synesthesia:http://www.sensequence.de/indexen.html"Art and Synesthesia: in search of the synesthetic experience":http://www.doctorhugo.org/synesthesia/art/index.html” The Phenomenology of Synaesthesia”, V.S. Ramachandran and E.M. Hubbard:http://www.imprint.co.uk/pdf/R_H-follow-up.pdf“The perceptual reality of synesthetic colors”:http://www.psy.vanderbilt.edu/faculty/blake/PDFs/PalmeriEtAl_PNAS_2002.pdfCognitive, Affective, & Behavioral Neuroscience 2004, 4 (3), 335-34,” Not all synesthetes

are created equal: Projector versus associator synesthetes”http://watarts.uwaterloo.ca/~src/papers/Dixon.CABN.2004.pdfConsciousness and Cognition 16 (2007) 913–931“Varieties of grapheme-colour synesthesia: A new theory of phenomenological and

behavioural differences”http://home.comcast.net/~sean.day/Ward%20et%20al%202007.pdfThe neuropsychology of synesthesia:http://scienceblogs.com/neurophilosophy/2007/09/the_neuropsychology_of_synest.php


References, books

Richard E. Cytowic

Synesthesia, A Union of the Senses, 1989

Richard E. Cytowic

The Man Who Tasted Shapes, 1993

Monica Vester

En värld av nyanser, 2004

SYNESTHESI A Multimodal Communication 2008 Ingvar Lind sight taste smell hearing touch.

Documents

Transcript of SYNESTHESI A Multimodal Communication 2008 Ingvar Lind sight taste smell hearing touch.