Interaction, Media and Communication Research Group Drawing Things Together: Integrating Modalities...

Interaction, Media and Communication Research Group

Drawing Things Together:Integrating Modalities in Dialogue.

MAGIC: Multimodality and Graphics in Interactive Communication.

EPSRC/ESRC PACCIT Initiative: People at the Centre of C&IT.

Pat Healey, James King, Charlie Peters.

Information, Media and Communication Research Group, QMUL.

John Lee, Jon Oberlander.

Human Communication Research Centre, University of Edinburgh.

Simon Garrod, Nick Fay.

Department of Psychology, University of Glasgow.

Ichiro Umata, Yasuhiro Katagiri,

ATR Media Integration and Communications Laboratories, Kyoto.


Outline

1. Drawing-in-interaction: ethnographic observations• turn-taking and topic management

2. Graphical Language Games: Musical ‘Pictionary’• Experiment 1: Community-specific Graphical Languages • Experiment 2: Mechanisms of Interaction

3. The mechanisms of interaction available to people directly constrain the form and organisation of shared symbol systems – not just individual cognitive-computational abilities

4. Augmented Human Interaction.


Design for Human-Human Interaction


Design for Human-Human Interaction?‘Single User’ Baton Design:

‘Multi-User’ Baton Design:

designed for exchange:• symmetric

• smooth• centrally balanced

Example: Architectural Design


Ethnography of Architectural Design• 6 staff working on a design competition

– mixed expertise and responsibilities• Prepare 4 A2 presentation boards addressing:

– use of site, environmental concerns, building use, open-space use • 58 project interactions video taped over 4 weeks

– 40 two-party interactions – 13 three-party interactions – 5 interactions > 3-party

• One 12 minute, 3 party interaction transcribed for analysis– coded for overlaps, pauses, run-throughs, stress etc.– coded for gesture and drawing activity


Basic Observations• Complex variety of drawing spaces (in 12 minutes)

– 6 physical drawing spaces • pieces of paper or regions of a piece

– 3 gestural drawing spaces • drawing in the air with pen or finger

– verbal ‘spaces’


016 R: jus shoa few people lukkin ova the: edge and e:mm a cuppla people R: RIGHT-HAND BY SIDE: MOVES HAND FORWARD AND POINTS ON 1. KEEPS FINGER ON 1, MOVES IT FROM L-R

wandri n up ( 1. 0) en / / meb be: : .R: (CO NT’D.)

.R: POINTS AT 1.

017 J : bu’ i ’ * if i tsa f ree form .J : BEGINS TO DRAW IN THE AIR ABOVE DRAWING BOARD

obj ect l i ke that // i ts bounta have a rai l round i t / / o r some .J : (CO NT’D.)

R: Yeah Yeah

ki ndav (post u m) so I thought shoul d I j us= .J :(CO NT’D)

.J : AND STARTS TO DRAW ON 1.

018 R: =J UST put the rai l en nothin el se .R: BRINGS HAND FORWARD ABOVE 1,

.FLICKS S-HAND ABOVE 1:

.J : (CO NT’D.)

Excerpt 1


Turn TakingDrawing activity does not automatically claim the floor:

1. Drawing activities continue across turn and speaker changes– overlapping speech is avoided (e.g., Levinson, 1983)– gestures are used to bid for and maintain floor control (Bavelas,

et. al. 1995) - but see Furuyama.

2. Where competition for the floor occurs during drawing:

drawing is suspended

gesture & speech used to compete for floor


Excerpt 2

143 J: =dimension R: Draws (cont’d…)

( 3. 0)R: Draws (cont’d…)

144 J : [so thas145 R: [SO I thi nk* j ust do a hal f a fence ri gh j us doa hal f as though

.R: RIGHT-HAND

FLICKS UP FROM

DRAWING

.R: RESUMES DRAWING ON 5. (CONT’D…)


Topic ManagementDrawing activity is used to manage topic changes: ceases or shifts

location if topic changes• In excerpt 1 topic changes from rail to space.

– J moves hand to draw in air above the board– reference to ‘space’ is harder to resolve

• When topic returns to rail drawing resumes on the page

(Earlier in interaction J draws same ‘space’ over the drawing)


ObservationsDrawing, gesture and speech are integrated into composite

communicative signals (cf. Neilson and Lee, 1994)• often treated as separate channels (e.g., Netmeeting)

Drawing activities often serve interactional functions• not only representation of domain or computational aid. • use of space to manage topics

• cf. topographic and referential gesture spaces


Graphical Dialogue

50% of everyday drawings are produced as part of interaction

(van Sommers, 1994)

Examples of Graphical Dialogue: • Routine interactions

– e.g., sketch maps, explanatory diagrams, games • Auxiliary mode of Communication

– e.g., cross-linguistic communication, aphasia• Specialised interactions

– e.g., design interactions

Does interaction affect representation?


Graphical Language Games: Musical ‘Pictionary’What do we do if we don’t share a symbol system?

Criteria for Task: • communication task• structured, regular, domain• few (or no) established representational conventions• exclusively graphical interaction

Typical Set-up:• Subjects seated in separate (soundproof) rooms

– communication via shared whiteboard application– 30 sec piano piece each

• Task: draw picture of target: no letters or numbers– SAME or DIFFERENT


Room A Room B

• One piano piece each: same or different?• Communicate by drawing: no letters or numbers


Music Drawing Types:

1. Abstract: – Graph-like representation of domain structure e.g., pitch, intensity,

rhythm

2. Figurative: – Ad hoc associations: faces, figures, objects or situations

3. Composite:– Mixture of Abstract and Figurative

(independent classification by 2 judges: Kappa = 0.9, N =287, k= 2)

Sequence of ‘Figurative’ Trials:

Sequence of ‘Abstract’ Trials:


Abstract / Figurative Contrast:

Abstract drawings provide a more complex representational system for the task. Specifically:

1. Systematicity: support direct comparison within and between items

2. Proto-Compositionality: distinct parts of the drawing refer to distinct parts of the music

Figurative drawings are more holistic and more ad hoc.


What affects use of Abstract or Figurative?

Previous (non) findings:

1. No effect of target on drawing type:– genre / tempo / mode

2. No effect of medium on drawing type– stylus vs. mouse

3. No difference in drawing effort– same average quantity of lines and ink


Effects of Interaction: 1. Repetition promotes abbreviation -but only if participants can signal

understanding– c.f. grounding in dialogue

2. Dialogue partners tend to use drawings of the same type.– c.f. accommodation/entrainment

3. Level of communicative interaction:• Concurrent Drawing

– 60% Abstract• Alternate Drawing:

– 60% Figurative


Hypotheses: 1. AUTONOMOUS CO-ORDINATION (truth wins): • Participants independently migrate toward the representational

scheme that is most efficient for the Concurrent task.– co-ordination of time-based axis? more comparative?– co-ordination emerges as aggregate individual experience

• (e.g., Clark, Lewis…)

2. COLLABORATIVE CO-ORDINATION• Participants use the opportunities afforded by concurrent interaction

to establish a co-ordinated 'sub-language'.– collaborative revision and refinement of conventions – co-ordination emerges through local histories of interaction


Experiment 1: Community Sub-Languages? Does interaction contribute anything, in addition to individual expertise, to

the co-ordinated use of the Abstract drawings?

Experimental Design:

Phase 1: develop several communities with equivalent task experience but different interaction histories

Phase 2: compare interaction within and between communities.

Note: community membership is hidden from participants.


Subject 2

Subject 1 Subject 6

Subject 5

Subject 3 Subject 4

Round 1 = Round 2 =Round 3 = Round 4 =

Phase1: Community Development


Phase 1:• 10 ‘communities’ (66 people)

– seated round edge of large PC lab• Music Task: one piece each, same or different?

– 4 rounds of 12 trials – different partner on each round

During Phase 1:• common ‘interaction history’ accumulates

– reliable increase in speed: 53 sec. to 43 sec. – reliable increase in accuracy 37% to 52%


Between Group =

Within Group =

Phase 2: Experimental Manipulation


Results: Chi2(2) =19.0, p=0.00

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Within Community Between Community

Proportion

Abstract

Figurative

Composite


Results:


Conclusions: Experiment 1Cross-group interaction de-stabilises use of Abstract drawings• independently of

– indivdual expertise– what is being represented– individual cognitive-computational abilities– explicit knowledge of community membership

• (cf. Healey 1997).

Support for Collaborative Co-ordination Hypothesis:• localised patterns of interaction lead to community-specific (graphical)

‘dialects’

Why?


Why does interaction matter?

Graphical Interaction Mechanisms:•localisation •alignment.

• Abstract drawings are ‘proto-compositional’: interaction allows participants to co-ordinate meaningful elements of each other’s drawings


Hypotheses:3. ‘MUTUAL MODIFICATION: coordinated use of Abstract drawings

depends on ability to use interaction devices to annotate and modify elements of each other's drawings.• (e.g., circling, underlining, and arrows)

Experiment 2: interfere with use of interaction devices (localisation and alignment) and assess effects on communication.


Manipulation 1: Block Localisation

Room AScreen

Room BScreen

Annotation of other’s drawing

blocked.


Manipulation 2: Block Alignment

Room AScreen

Room BScreen

Transpose


Experimental Design:

• Pairs seated in separate soundproof rooms • Communicate via shared whiteboard- no letters or numbers

Subject A’s View Subject A’s ViewSubject B’s View Subject B’s View

+ BLOCKING- BLOCKING

- T

RA

NS

NP

OS

ITO

N+

TR

AN

SP

OS

ITIO

N

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

+B +T + T - B -T + B -T -B

Level of Interaction

Proportion of Drawings

Figurative

Abstract

(Blocking: Chi2(3) = 96.70, p =0.00, Transposition: Chi2(3) = 81.61, p =0.00)

Effects of Interference with Interaction:


Conclusions Experiment 2: • Blocking and Transposition cause change in graphical conventions

– independently of what is being represented– Independently of individual cognitive-computational abilities

• Complexity of emergent symbol system depends on mutual modification (localisation, alignment)– co-ordinated manipulation of external representations– participants’ ability to modify each other’s representations

• not editing / annotation / revision per se


General Conclusions:Significant parallels between verbal and graphical dialogue

– grounding, accommodation, turn-taking, – modality independent, interactional constraints on representation

How do people co-ordinate despite differences in interpretation?• Local, surface-based, manipulations of external representations.• Verbal and graphical repair mechanisms

– Localisation• specficity, reprise fragment clarifications

– Alignment• sequential relevance, embedded repair


General Conclusions:Drawing supports transactional and interactional functions• not just a representational medium

Not explained by physical / perceptual / computational properties per se • alternate vs. concurrent drawing

Modalities combine to form composite communicative signals applications often treat them as separate channels (e.g., net-meeting)

Generic interaction mechanisms are apparent across modalities• turn-taking, topic management, repair, grounding, accommodation

Augmented interaction mechanisms richer more expressive languages?


Augmented Human Interaction:Beyond Face-to-Face

1. Capture: gesture, expression, attention, engagement, understanding…

2. Provide augmented cues for interaction

3. Enable richer, more robust, forms of human communication– gestural languages, musical languages, dance …


Augmented Human Interaction Lab

• multi-user, real-time, full body, motion capture.• integrated 3D audio display


Augmented Human Interaction Lab

Interaction, Media and Communication Research Group Drawing Things Together: Integrating Modalities...

Documents

Transcript of Interaction, Media and Communication Research Group Drawing Things Together: Integrating Modalities...