Multimodal Methods for Analyzing Communication and Learning with Digital

Technology

Summary● Multimodal Social Semiotics & Meaning Making

● Multimodal Social Semiotics: some definitions

● Multimodal Social Semiotics & Research Methods : Interaction analysis

● Multimodal Social Semiotics & Research Methods : video analysis & transcripts

● Applications:○ Learner / User generated content○ Embodiment & Technology

■ Embodiment & Technology: a case study

Social Semiotics Semiotics:

● Study of signs and signification processes● Interpretative semiotics (double formation of meaning / Semiosis:

processes of signification and interpretation (Eco))

Differences between traditional semiotics and social semiotics● Arbitrariness (signifier and signified):

- traditional: arbitrary rules that can not be modified (De Saussure)- social: the "codes" of language and communication are formed by social processes (= social dimension of meaning) :→ rules made by people that can be modified (through institutional or symbolic power)→ motivated sign: instead of fixed and unchanging "codes", signs are considered to be resources that people use and adapt to make meaning accordingly to the interests of the producer/interpreter and the characteristics of the objects

MEANING

MultimodalityMultimodality & Social Semiotics:People use differents modes to make meaning. Going being linguistics approach → framework for the analysis of visual, aural, embodied and spatial aspects of the interaction and the environment

● Each mode has different affordances that are the product of the social and historical use. The form should be particularly apt for conveying a certain meaning

● People orchestrate meaning through their selection and configuration of modes

meaning potential of an artifact

social & cultural environment

resources, interests and knowledge of the person

Multimodal Social Semiotics: some definitionsMode: a set of resources for making meaning and organizing principles, a channel of communication

Semiotic resources (Halliday) : a resource for making meanings → action, material and artefacts that we use to communicate (difference with sign). They have semiotic potential and semiotic affordances. Ej.: written text, image, walking

Multimodal Social Semiotics: some definitionsSemiotic potential: each semiotic resource has semiotic potentials (rather than a specific meaning) that are culturally and historically shaped

Affordance: material, social and cultural (not just perception!)

Semiotic affordance: each semiotic resource has semiotic potentials that are new meanings that have not been recognized yet and are latent in it

Modal affordance: potentialities and constraints of different modes; what is possible to express and communicate easily with the resources of a mode and what is less straightforward or even impossible. They are historically and socio-culturally shaped.

Multimodal Social Semiotics: some definitionsMotivated sign: agency of the sign maker, draws the attention on the interests and intentions that motivate a person’s choice of a given semiotic resources

multimodal ensemble: bring together different mode and look at how different mode bring different meaning

simultaneity: all the semiotic resources mutually elaborate each other. Each one by itself is incomplete. together they create a whole that is bigger than its part

coherence & cohesion: how different modes work together to produce coherence and make a complete “text” (are the used modes coherents between them or they contradict?) / cohesion: devices to construct coherence

Van Leeuwen, Theo. Introducing social semiotics. Psychology Press, 2005.Jewitt, C (2012) Multimodal methods for researching digital technologies. In: The Sage Handbook of Digital Technology Research. Sage, London.

Application in our research

● Design: theoretical perspective

Design: theoretical perspective

Full Body Interaction Environment as Multimodal Ensemble● what modes are present?

- which are their semiotic resources? semiotic potentials and modal affordances? what are the potentials and constraints of Full Body Interaction? What is possible to communicate and express easily with this mode? What is less straightforward or even impossible? (inventories, socio-cultural references)● How the different modes relate together?

Multimodal Ensemble

Semiotic Resources

Semiotic potentials

Modal affordances

Multimodal Social Semiotics & Research Methods

Applications: Interaction analysis

HISTORY:

‘30/’60: Interaction Sociology (Goffmann, Garfinkel) → bottom- up approach to study society

‘60: Conversation Analysis

‘70: Conversation Analysis & Embodiment (Goodwin)

‘90: Workplaces studies (Goodwin, Mondada)

- role of body and action in achieving collaboration

- embodiment in conversations

CONCEPTS:- Adjacency pair (invitation to the other to respond) & paired action (adjacency but through action)

- Reading other bodies (Hindmarsh and Pilnick, 2002): people read each other bodies; you see somebody doing something and if you know the context you will know what to do next and what is expected from you (ej. study in anesthesia: they don’t speak so much but they coordinate each other through action & body)

Multimodal Social Semiotics & Research Methods : video analysis & transcriptsFrom video to data:- at the end of the day do a little description of each video to describe what is going on

- vignette, summary

- coding categories

- multimodal transcription - visualization

Multimodal transcription :- Define a methodological framework

- Define the purpose and focus of the transcript (ej. timescale, modes)

- Design the transcript

Multimodal Transcription: the quality of transcript is not how much it is a representation of the reality but the point to which it allows research your topic

- the modes selected for transcription depend on the focus of the research

Multimodal Social Semiotics & Research Methods : video analysis & transcripts

Bezemer, J. (2012). How to transcribe multimodal interaction? In Texts, Images and Interaction: A Reader in Multimodality. Mouton de Gruyter.Mavers, D. (2012). Transcribing video. NCRM working paper.

Applications: Learner / User generated content→ Motivated Sign / Form as meaning

● What kind of semiotic resources do they use to communicate something? Which cultural references it relate to?

● What are the resources available? Which one are selected and how they are re-shape? (which other selection could have been possible)

● How the different resources are orchestrated between them?

● Is it coherent? How this coherence have been constructed?

● What she wants to express with it?

Applications: Embodiment & Technologymultimodality / ethnography / psychology

DESIGN● understand the potential that different modes can bring to interaction

(modal affordance; which mode can be apt for this purpose?)● possibilities of identifying novel semiotic resources● ...

ANALYSIS● emphasis on situated meaning making: how people make meaning

in a specific situation● analyze which modes are used in the environment and which

resources are available ● examine and compare the effect of using different modes in learning

context● ...

Embodiment & Technology: case study

goal:METHOD:1) Logging Video Data: descriptive account (time, content, events, modal configuration): vignette, sketches, screen-capture, maps..2) Raw analysis: watch the video in different way (sound, slow/fast motion) → start to generate criteria for sampling3) Sampling: select episodes / phenomena / etc. // category definition4) Transcription 5) Modal analysis:

- break down the modes (ej: position, gaze, hand-action)- look for intermodal relation (ej: pace)

6) Incorporate the context into the analysis

Light Table

Application in our research

Application in our research

● Design: participatory design

● Analysis

Design: participatory design - children production

● What kind of semiotic resources do they use to communicate something? Which cultural references it relate to?

● What are the resources available? Which one are selected and how they are re-shape? (which other selection could have been possible)

● How the different resources are orchestrated between them?

● What she wants to express with it?

Semiotic resources

Semiotic potentials

Motivated Sign

Analysis- Methodological framework

● situated meaning making● which modes are used in the environment and

which resources are available ● examine and compare the effect of using different

modes

- Go beyond discourse analysis

- Sense-making through the body and the space