An analysis of generative dialogue patterns across interactive learning environments: Explanation,...

An analysis of generative dialogue patterns across interactive learning environments: Explanation, elaboration, and co-construction

Robert G.M Hausmann

Pittsburgh Science of Learning Center (PSLC)Learning Research and Development Center

University of Pittsburgh

Top-level Goal

XCognitive

Social

Affective

DeepLearning

+

Outline Introduction

– Thesis– Definitions– Methodology

Evidence– Individual learning – Human tutoring (novice & expert)– Peer collaboration– Observing Tutorial Dialogs Collaboratively

Discussion– Integration with serious games

Thesis Part 1: There are several paths toward learning.

– Some paths are better suited for the acquisition of different representations (Nokes & Ohlsson, 2005).

Part 2: Generative interactions produce deep learning.– Increasing the probability that a generative interaction

occurs should increase the probability of robust learning.

Part 3: Different interactive learning environments differentially support generative interactions.– Learning with understanding vs. performance

orientation (Schauble, 1990; Schauble & Glaser, 1990).

Definitions Learning

– Revise of mental model– Application of conceptual knowledge– Reduction of errors during the acquisition of procedural knowledge

Interaction– Dialog: situation-relevant response that occurs between two or

more agents (human or computer).– Monolog: statements uttered out loud that reveal the individual’s

understanding processes. Generative

– Produce inferences (ex: the lungs are the site of oxygenation)– Apply knowledge to a problem (ex: applying Newton’s second law)

Methodology: high-level description

1. Collect & transcribe a corpus of learning interactions

2. Categorize statements (Chi, 1997)3. Assess learning at multiple levels of depth4. Correlate statements with shallow and

deep learning5. Follow-up Study: Experimentally

manipulate interaction type (goto 1)

Study 1: Self-explaining vs. Paraphrasing

Procedure– Domain: Circulatory system– Pretest– Prompting intervention (41.1 min.)– Posttest

Participants– University of Pittsburgh undergraduates (N = 40)– Course credit

Research Questions– Can a computer interface use generic prompts to

inspire students to self-explain?– If so, what is the effect on learning?

“Self-explanation” Field

Textual Materials

Generic TutorialPrompts

Results: Self-explanation Frequency (Exp. 2)

6.5 27.9 3.50.0

5.0

10.0

15.0

20.0

25.0

30.0

Self-Explanation Paraphrase MetacognitiveAverage Number of Statements

Source: Hausmann and Chi (2002)

Results: Correlations with Learning (Exp. 2)

Verbatim (shallow)

Integrated (deep)

Paraphrasing r = -.10, p = .66 r = -.14, p = .56

Self-explainingn.s. r = .38, p = .10

High SE: r = .18, p =.77 High SE: r = .92, p = .03

Source: Hausmann and Chi (2002)

Study 2: Coverage vs. Generation Method

– Domain: Electrostatics– Procedure: Alternate between problem-solving and example-

studying (110 min.)– Design: Example (complete vs. incomplete) x Study Strategy (self-

explain vs. paraphrase) Participants

– U.S. Naval Academy midshipmen (N = 104)– Course credit

Research Questions– Does learning depend on the type of processing or the

completeness of the examples?– Does prompting for self-explaining also work in the classroom?

Write Equations

Draw Vectors

Draw Coordinates

TutorialHints

Define Variables

Method: Timeline

Problem1 Problem2 Problem3 Problem4

Self-explainComplete

Self-explainIncomplete

ParaphraseComplete

ParaphraseIncomplete

Example1



ParaphraseComplete


Example2



ParaphraseComplete


Example3

Results: Bottom-out Help

6.9 2.5 6.7 4.30

12

34

56

78

9

CompleteParaphrase

CompleteSelf-explain

IncompleteParaphrase

IncompleteSelf-explain

Bottom-out Hints

Source: Hausmann and VanLehn (in prep)

Results: Assistance Score

83.7 70.9 97.4 72.40

20

40

60

80100

120

140

160

180

IncompleteParaphrase

IncompleteSelf-explain

CompleteParaphrase

CompleteSelf-explain

Assistance Score

Source: Hausmann and VanLehn (in prep)

Study 3: Novice, Human Tutoring Procedure

– Domain: circulatory system– Pretest (w/ textbook)– Intervention (120 min.)– Posttest

Participants– Tutors: Nursing Students (N = 11)– Tutee: Eighth-Grade Students (N = 11)– Paid volunteers

Research Questions– How do novice tutors naturally interact with students?– Can tutors be trained to interact in specific way, and what impact

does alternative tutorial dialogs have on learning?

Results: Learning

Knowledge pieces: – Study 1: Pretest 13%; Posttest 46%, p < .001– Study 2: Pretest 22%; Posttest 45%, p < .001

Mental model:– Study 1: Pretest 0%; Posttest 73%– Study 2: Pretest 0%; Posttest 64%

Double Loop-2

Lungs

Body

Source: Chi, Siler, Jeong, Yamauchi, and Hausmann (2001)

Results: Learning

Question-answering:

Exp. 1 Exp. 2

Category 1 64% 65%

Category 2 45% 46%

Category 3 35% 33%

Category 4 31% 41%


Dep

th

Results (Exp. 1): Types of tutor moves and student responses


0

10

20

30

40

50

60

Explanation Feedback Scaffold CGQuestions

Tutors' Moves

Frequency per Response Type

Continuer

Shallow Follow -up

Deep Follow -up

Reflection

Initiate a New Topic

Students' Responses

Results (Exp. 2): Types of tutor moves and student responses


0

10

20

30

40

50

60

Explanation Feedback Scaffold CGQuestions

Tutors' Moves

Frequency per Response Type

Continuer

Shallow Follow -up

Deep Follow -up

Reflection

Initiate a New Topic

Students' Responses

Study 4: Comparison of Multiple Interactive Learning Environments Procedure

– Domain: Newtonian Mechanics– Pretest (w/ textbook)– Instructional Intervention (next 5 slides)– Posttest (w/o textbook)

Participants– University of Pittsburgh undergraduates (N = 70)– Paid volunteers

Research Questions– What types of interactions are related to learning from

tutoring and collaboratively observing tutoring?– Why do peers learn from collaboration?

Experimental Design

Intervention 1: Tutoring– Learning Resource: Expert human tutor– One student (n = 10 video tapes)

F=mam=3kg Tutoring session #1

Experimental Design

F=ma m1+m2=?

Intervention 2: Observing Collaboratively– Learning Resource: Peer & Videotape– Two students (n = 10; yoked design)

Tutoring session #1

Experimental Design

Intervention 3: Observing Alone– Learning Resource: Videotape– One student (n = 10; yoked design)

Tutoring session #1

Experimental Design

Intervention 4: Collaborating– Learning Resource: Peer & Text– Two students (n = 10)

Fx=max

Fx=7kgx2.6m/s2

Halliday & Resnick

Experimental Design

Intervention 5: Studying Alone– Learning Resource: Text– One student (n = 10)

Fy=mgFy=2kgx9.8m/s2

Halliday & Resnick

Results: Condition Differences

Source: Chi, Roy, and Hausmann (accepted)

0

10

20

30

40

50

60

70

Studying Alone ObservingAlone

Collaborating ObservingCollaboratively

Tutoring

Deep Matched Steps %

Pre

Post

Results: Interaction Analysis

AverageNo. per Session

ProportionTutees‘

Learning(n = 10)

Observers‘Learning(n = 20)

T: ScaffoldingS: Substantive

Responses59.28 59.84%

r = .656,p = .039

r = .434,p = .056

T: ExplainingS: Substantive

Responses39.79 40.16% r = .576,

p = .082n.s.

Total 99.06 100.00%


Results: Condition Differences


0

10

20

30

40

50

60

70

Studying Alone Observing

Alone

Collaborating Observing

Collaboratively

Tutoring

Deep Matched Steps %

Pre

Post

19% 20% 29%45% 67% 64%0%

20%

40%

60%

80%

100%

Other-directedExplaining

Co-Construction Self-DirectedExplaining

Percent

% Corpus % Gain

Results: Collaborative Dyads

Source: Hausmann, Chi, and Roy (2002)



42% 58% 71%60%0%

20%

40%

60%

80%

Elaborative Co-construction

Critical Co-construction

Percent

Freq. Gain.



19% 20% 29%45% 67% 64%0%

20%

40%

60%

80%

100%


Co-Construction Self-DirectedExplaining

Percent

% Corpus % Gain



82% 45% 29%71%

0%

20%

40%

60%

80%

100%


Self-directedExplaining

Percent Gain

Speaker Listener

Study 5: Interaction Training Procedure

– Domain: Conceptual Engineering (bridge design)– Pretest (w/ textbook)– Intervention (5 min.)– Problem-solving Task (30 min.)– Posttest

Participants– University of Pittsburgh undergraduates (N = 136)– Course credit

Research Questions– Can undergraduate dyads be trained to interact effectively (i.e., co-

construct)?– What effect do certain dialog types have on problem solving and

learning?

Fabrication Cost

Modify Properties Member

List

Color-coded Feedback

Results: Manipulation Check

Condition

ControlDyads

Elaborative Dyads

Clarification Questions

26.37 (11.32) 18.00 (9.71)

Elaborative Statements

8.94 (6.46) 13.38 (5.76)

Source: Hausmann (2006)

Results: Problem Solving


0.56 0.55 0.630.45

0.50

0.55

0.60

0.65

Individuals ControlDyads

ElaborativeDyads

Optimization Score

Results: Learning


5% 3% 11%0%

2%

4%

6%

8%

10%

12%

Individuals ControlDyads

ElaborativeDyads

Deep Knowledge Gain

Summary Individual Learning (Studies 1 & 2)

– Paraphrasing => shallow learning– Self-explaining => deep learning

Human Tutoring (Studies 3 & 4)– Listening to tutor explain => shallow learning– Receiving scaffolding => deep learning– Reflective comments => deep learning

Peer collaboration (Studies 4 & 5)– Listening to peer explain => shallow learning– Giving an explanation to peer => deep learning– Co-constructing knowledge => deep learning

Observing Tutoring Collaboratively (Studies 4)– Observing tutor explain is not correlated with deep learning– Observing student receive scaffolding => deep learning

Integration with Serious Games

What is the implication of these results for the design of serious games?– How can a game inspire explanation,

elaboration, or even co-construction?

Acknowledgements Funding Agencies

Support @ LRDC– Gary Wild– Shari Kubitz– Eric Fussenegger

Advisors– Michelene T.H. Chi– Kurt VanLehn

Physics Instructors– Donald J. Treacy, USNA– Robert N. Shelby, USNA

Other Influences– Mark McGregor– Marguerite Roy– Rod Roscoe

Programmers– Anders Weinstein– Brett van de Sande

ReferencesStudy 1: Hausmann, R.G.M. & Chi, M.T.H. (2002)

Can a computer interface support self-explaining? Cognitive Technology, 7(1), 4-15.

Study 2: Hausmann, R.G.M. & VanLehn (in prep). The effect of generation on robust learning.

Study 3: Chi, M.T.H., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R.G. (2001). Learning from human tutoring. Cognitive Science, 25(4), 471-533.

Study 4a: Chi, M.T.H., Roy, M., & Hausmann (accepted). Observing Tutorial Dialogues Collaboratively: Insights about Tutoring Effectiveness from Vicarious Learning, Cognitive Science, x, p. xxx-xxx.

Study 4b: Hausmann, R.G.M., Chi, M.T.H., & Roy, M. (2004) Learning from collaborative problem solving: An analysis of three hypothesized mechanisms. 26nd Annual Meeting of the Cognitive Science Conference, Chicago, IL.

Study 5: Hausmann, R. G. M. (2006). Why do elaborative dialogs lead to effective problem solving and deep learning? Poster presented at the 28th Annual Meeting of the Cognitive Science Conference, Vancouver, Canada.

Inferential Mechanisms Simulation of a mental model (Norman, 1983) Category membership (Chi, Hutchinson, &

Robin, 1989) Analogical reasoning (Markman, 1997) Integration of the situation and text model

(Graesser, Singer, & Trabasso, 1994) Logical reasoning (Rips, 1990) Self-explanation (Chi, Bassok, Lewis, Reimann,

& Glaser, 1989)

Cognitive + Social Factors Different types of interaction lead to

different representations:– Non-generative interactions lead to shallow

learning.» Definition: does not modify material in any

meaningful way.

– Generative interactions lead to deep learning.» Definition: significantly modifies the material in a

meaningful way.

An analysis of generative dialogue patterns across interactive learning environments: Explanation,...

Documents

Transcript of An analysis of generative dialogue patterns across interactive learning environments: Explanation,...