Readers routinely represent implied object rotation: The role of visual experience

Wassenberg & Zwaan, in press, QJEP

Brennan PaynePsych 525

10.27.10

Theories of discourse comprehension

•Construction-Integration (Kintsch & van Dijk, 1978; Kintsch, 1998)

•Structure-Building (Gernsbacher, 1990, 1997)

•Event-Indexing Model (Zwaan et al., 1995)

•Resonance Model (O’Brien et al., 1995, 1998)

•Shared assumption that discourse comprehension can be modeled as the integration of abstract and amodal representations.

Theories of discourse comprehension

•Construction-Integration (Kintsch & van Dijk, 1978; Kintsch, 1998)

•Structure-Building (Gernsbacher, 1990, 1997)

•Event-Indexing Model (Zwaan et al., 1995)

•Resonance Model (O’Brien et al., 1993, 1995, 1998)

•Shared assumption that discourse comprehension can be modeled as the integration of abstract and amodal representations.

Construction-Integration• Computational model

• Different levels of representation: • Surface form of the text• Text base: propositional information from the text• Situation model: representation of situation implied with the text;

derived from propositional text base• Proposition: “idea unit”; smallest unit of knowledge. Follows predicate

argument form: PREDICATE(ARGUMENT1, ARGUMENT2)

2a. The carpenter pounded the nail into the wall.

2b. The carpenter pounded the nail into the floor.[POUNDED(CARPENTER, NAIL)], [INTO (NAIL, WALL)]

[POUNDED(CARPENTER, NAIL)], [INTO(NAIL, FLOOR)]

1a. The ranger saw the eagle in the sky.[SAW(RANGER, EAGLE)], [IN (EAGLE, SKY)]

1b. The ranger saw the eagle in its nest.[SAW(RANGER, EAGLE)], [IN (EAGLE, NEST)]

Alternative Account1. The carpenter pounded the nail into the wall.

2. The carpenter pounded the nail into the floor.

[POUNDED (CARPENTER, NAIL)], [IN (NAIL, WALL)]

[POUNDED (CARPENTER, NAIL)], [IN (NAIL, FLOOR)]

Proposition Account: Highly Identical, only difference is N specifying orientation

Perceptual Symbol Account (Barsalou, 1999a,b; Stanfield & Zwaan, 2001; Zwaan et al., 2002): Sentences are very different in perceptual representation that is implied.

Do readers represent perceptual information?

John put the pencil in the cup.

• Sentence-picture verification task Was this mentioned in the previous

sentence?

Significant differences in RT latencies when objects matched vs. mismatched.

838 (331)

882 (329)

*

Stanfield & Zwaan (2001); Psych. Sci.

Perceptual Traces (Zwaan & Kaschak, 2008)• Orientation (Stanfield & Zwaan, 2001)• Shape (Zwaan et al., 2002; Dijkstra et al., 2004)• Size (Taylor & Zwaan, 2010)• Movement and Motion (Kaschak et al., 2005; 2006) • Color (Richter et al., 2009; Therriault et al., 2009)

Previous Research: Nature of language representation

Sentence Picture

Can this perceptual information affect online language processing?

Picture Sentence

Do readers represent perceptual information?

Current StudyDoes a recent visual exposure to an object in a specific orientation affect later language comprehension?

Wassenburg & Zwaan (in press); QJEP.

A 3-phase “visual memory” paradigm (Zwann et al., 2010)1. Word-picture verification task

-Experimental items shown in vertical or horizontal orientation2. 15- minute filler task3. Eye-tracking session

Three phases are presented as unrelated experiments to deter some kind of strategy use.

Predict a match/ mismatch effect: fixation times on the prepositional phrase (into the wall/ in the cup) that implies the object orientation should be sensitive to the orientation of the previously seen image.

MethodParticipants:•N = 34; N= 28 after track loss/errors. 50%Female •Age= 20.3 (18-24). •Native Dutch Speakers

MaterialsP1• 80 word-picture items • 60 fillers; 20 critical items• Each critical item formed a match with its word• Orientation (H;V) counterbalanced across participants over 2 lists

P2• ??? Maybe the flag test

P3 • Tobii 2150 eye tracker• 40 Dutch Sentences• 20 filler; 20 critical sentences using critical words from P1 • Half of each orientation matched and half mismatched

Method

Aunt Karen finally found the toothbrush in the sink of the bathroom.

Phase 3:

ToothbrushPhase 1:

Phase 2: VSP rotation filler taskApprox. 15 min.

1 2 3 4* 5

Procedure

Results*

†

†p = .06

*p < .05

Much in the way that language affects later visual processing (Stanfield & Zwaan, 2001), visual memory also influences language processing.

Prior exposure to a picture of an object in a particular orientation affects later reading times for phrases that imply the orientation of that object.

Match/mismatch effects occur on first pass measures on the disambiguating PP and diminish quickly, suggesting that these effects are both early and immediate.

Reading comprehension may be multimodal, not only using linguistic representations, but sensory/perceptual representations as well.

Conclusion