What is What is multimodality?multimodality?

Bella DicksBella Dicks

School of Social SciencesSchool of Social Sciences

Cardiff UniversityCardiff University

‘all the modes employed contribute to meaning…’

science centre exhibition hallscience centre exhibition hall

modes and mediamodes and media

ModesModes - abstract, non-material - abstract, non-material resourcesresources of meaning-making of meaning-making (e.g. the (e.g. the ‘languages’ of image, sound, gesture, colour, ‘languages’ of image, sound, gesture, colour,

texture, narrative)texture, narrative)

MediaMedia - specific material forms in - specific material forms in which modes are realised which modes are realised (e.g. a (e.g. a photograph, a computer screen, an audio photograph, a computer screen, an audio recording, paint, a wall, clothing, a chisel, a poem, recording, paint, a wall, clothing, a chisel, a poem, a block of wood)a block of wood)

Kress, G.and Van Leeuwen, T. (2001). Multi-modal Discourse. London: Arnold.

multimodalitymultimodality

– peoplepeople communicate through: communicate through:speech,speech, accentaccent, , gesture, facial expression, gesture, facial expression, gaze-direction, shape, weight, size, body gaze-direction, shape, weight, size, body posture, position & movement, dress (colour, posture, position & movement, dress (colour, shape, texture, etc.), hairstyle, etc. shape, texture, etc.), hairstyle, etc.

– environments/objectsenvironments/objects communicate communicate through:through:colourcolour, , texturetexture, , shapeshape, , position, position, opaqueness/light, weight, movement, sound, opaqueness/light, weight, movement, sound, etc.etc.

multimodality in the research process

1. observations/elicitations of the field (=data)

2. recording of data 3. analysis of data records4. representation/dissemination of

research findings

N.B. are data recorded, represented, reproduced, reduced, transformed?

1. Multimedia in the field1. Multimedia in the field

Material environments, e.g.Material environments, e.g.– Relative position of objectsRelative position of objects– Walls and corridorsWalls and corridors– Colours, shapes and sizesColours, shapes and sizes

Social interaction, e.g.Social interaction, e.g.– Relations of proximityRelations of proximity– Sequencing of eventsSequencing of events– Turn-taking in conversationTurn-taking in conversation

modalities of the field setting

Exhibits Hall Science Theatre

framing: a multimodal principle

framing works across diverse media: e.g. in interior design; in sound compositions, in paintings.

introduces breaks and discontinuities; contrasts and continuities; uses framelines and empty space

framingframing In In writingwriting; e.g. punctuations, prepositions ; e.g. punctuations, prepositions

and paragraphsand paragraphs In In imagesimages; e.g. lines and borders; e.g. lines and borders In In soundsound; e.g. pacing, intonation and ; e.g. pacing, intonation and

rhythmrhythm In In material environmentsmaterial environments; e.g. object ; e.g. object

positioning, corridors and wallspositioning, corridors and walls In In movementmovement; e.g. jumps, gestures and ; e.g. jumps, gestures and

flowsflows In In web-pagesweb-pages; e.g. pop-up boxes, graphic ; e.g. pop-up boxes, graphic

layout and hyperlinkslayout and hyperlinks

‘‘Framing parcels up the conceptual Framing parcels up the conceptual world (…) It marks off, but in doing world (…) It marks off, but in doing so it establishes, at the same time, so it establishes, at the same time, the elements which may be joined. the elements which may be joined. (…) “Without framing, no meaning”, (…) “Without framing, no meaning”, we might say.’we might say.’Gunther Kress (2003) Literacy in the new media age, p. 123/4/5.Gunther Kress (2003) Literacy in the new media age, p. 123/4/5.

audio recordings: framing what’s audio recordings: framing what’s saidsaid

PausesPauses

HesitationsHesitations

Ordering of itemsOrdering of items

SoundscapesSoundscapes

2. recording multimodality2. recording multimodality

What are the What are the affordancesaffordances of different of different recording media?recording media?

– photographs v. writingphotographs v. writing

– photographs v. videophotographs v. video

– writing v. soundwriting v. sound

multimodal data-recordsmultimodal data-records still imagesstill images deploy: deploy:

colourcolour shape shape light light directionalitydirectionality

perspectiveperspective position position borders borders sizesize

voice recordingsvoice recordings deploy: deploy:accentaccent expression expression tonetone pitchpitch

languagelanguage volumevolume cadencecadence rhythmrhythm

recording mediarecording media

E.g. cameras embody their own framing E.g. cameras embody their own framing devices:devices:– edge v. centre of photographedge v. centre of photograph– composition of framecomposition of frame– angle and position of cameraangle and position of camera– choice of what to recordchoice of what to record– decision about when to press shutterdecision about when to press shutter– Etc.etc.Etc.etc.

data recordsdata records1.1. photographsphotographs

• record spatial framing e.g. boundaries & record spatial framing e.g. boundaries & transitstransits

2.2. audio recordingsaudio recordings– record temporal framing e.g. sequencing of record temporal framing e.g. sequencing of

items; pauses, etc.items; pauses, etc.

3.3. video recordingsvideo recordings– record spatial and temporal framingsrecord spatial and temporal framings

4.4. fieldnotesfieldnotes– record spatial and temporal framingsrecord spatial and temporal framings

fieldnotesfieldnotes‘‘When you enter [the Science Centre], you see a When you enter [the Science Centre], you see a huge, well-lit hall huge, well-lit hall in front of youin front of you with high with high ceilings and a ceilings and a gallery abovegallery above. It is a white space . It is a white space with big white pillars and vast windows, but with big white pillars and vast windows, but there's also lots of colour, noise and movement. there's also lots of colour, noise and movement. On entering, immediately On entering, immediately ahead of youahead of you is a is a plastic ball, suspended seemingly by magic in plastic ball, suspended seemingly by magic in front of a bright yellow solid pyramid. front of a bright yellow solid pyramid. BeyondBeyond are more yellows and reds and greens and are more yellows and reds and greens and blues. blues. All aroundAll around are a variety of different brightly- are a variety of different brightly-coloured devices or machines, housed in brightly coloured devices or machines, housed in brightly coloured casings and consoles. Many of them coloured casings and consoles. Many of them move, produce sounds, create visual effects, and move, produce sounds, create visual effects, and so forth when activated by a user… so forth when activated by a user… ……There are There are lots of children moving aroundlots of children moving around excitedly, flitting from exhibit to exhibit, excitedly, flitting from exhibit to exhibit, sometimes bumping into each other.sometimes bumping into each other.