Conversational Strategies that Support Idea Generation Productivity in Groups Rohit Kumar Jack L....

Conversational Strategiesthat SupportIdea Generation Productivityin Groups

Rohit KumarJack L. BeuthCarolyn P. Rosé

2

Overview• User Study:

– Design of Tutor prompts for supporting Collaborative Learning• Software Agents: Monitoring Student Interactions

– Social Interaction Strategies– Presentation Style

• Outline:– Background: Research Objectives, Earlier Work– Presentation Style: Heteroglossia– User Study: Task, Procedure, Metrics, Design– Results

Background Presentation Style User Study Idea Generation Results

3

Bridges

CollaborativeLearning

CommunicationStudies

Small GroupCommunicati

on

TutorialDialog

DialogSystems

ConversationalTutors

Our Work


4

Prior Work• Benefits of

– 0.90σ : Collaborative Learning– 1.06σ : Dynamic Support (Software Agents)– 1.24σ : Combined

• Kumar et. al., 2007

• Students ignore/abuse Tutors– Tutors are unable to manage the

attention of the students

A20

wow that tutor is annoying

A13

Yeah

A13

STFU tutor!

B9

hi tutor

B9

tutor, you're a jerk

a7 so, got any ideas?Tutor

Starting now, you have around 40 minutes to work together…

A2 hrmmma7 stupid tutor no one asked you for help

Tutor

There will be more potential for power generation ... Where might the increased heat input go instead?

St16

this tutor is really annoying

St5 Agreed


5

Social Interaction Strategies• Small Group Communication– Instrumental vs. Socio-Emotional

Interaction• Bales, 1950


Shows Solidarity

Shows Tension Release

Agrees

Gives Suggestion

Gives Opinion

Gives Orientation

Asks for Orientation

Asks for Opinion

Asks for Suggestion

Disagrees

Shows Tension

Shows Antagonism

1. Showing Solidarity: Raises other's status, gives help, reward 1a. Do Introductions: Introduce and ask names of all participants 1b. Be Protective & Nurturing: Discourage teasing 1c. Give Reassurance: When student is discontent, asking for

help 1d. Compliment / Praise: To acknowledge student contributions 1e. Encourage: When group or members are inactive 1f. Conclude Socially

2. Showing Tension Release: Jokes, laughs, shows satisfaction 2a. Expression of feeling better: After periods of tension, work

pressure 2b. Be cheerful 2c. Express enthusiasm, elation, satisfaction: On completing

significant task steps

3. Agreeing: Shows passive acceptance, understands, concurs, complies

3a. Show attention: To student ideas as encouragement 3b. Show comprehension / approval: To student opinions and

orientations

Kumar & Rosé, 2010

6

Social Interaction Strategies• Implemented Using Basilica

• Kumar & Rosé, 2011 (IEEE Trans. Learning Tech.)


139

Tutor

One last thing on this topic, Does more (or less) stress in a wrench make it easier to use?

140

S95 no change?

141

Tutor

You are correct about that.. Stress doesn't determine ease of use.

142

Tutor

It's the moment achieved by the wrench that determines the ease of use.

143

S89 yay!

144

Tutor

It's good to have your contributions in the discussion Jackie :-)

145

Tutor

Go team :-)

146

S89 Go team yay

147

Tutor

I am happy to be working with our team

148

S89 Me too

149

S95 whoa the bot knows my name

150

Tutor

Unfortunately maximum stress (12800) in our design1 is way above the maximum allowed stress (i.e. 8750)

151

Tutor

This wrench cannot be safely used!

7

Social Interaction Strategies• Significant Benefits

– 0.71σ : Using Fully-Automated Social Interaction Strategies• Kumar et. al., 2010 (ITS)

– More Efficient at delivering the instructional content

• Other Experiments– Amount of Social Behavior

• Learning, On-Task Interaction Hua et. al., 2010 (ITS)

– Why/How & Timing of Behavior• Learning, Perception of Tutor Kumar & Rosé, 2011(SigDial)

– Transfer to other domains• Current Work

– Style of Presentation• This talk


8

Presentation Style• Heteroglossia vs. Monoglossia– Lexical/Grammatical choices– Indicates social positioning of the speaker

• Heteroglossia– Awareness/Acknowledgement of multiple

perspectives


Original/Neutral

Think about how we can improve this?

Heteroglossic

I suggest that you think about how to improve this

Monoglossic

Think about how to improve this

Original/Neutral

Maybe we can use a safer material this time!

Heteroglossic

Using a safer material would be strongly advisable!

Monoglossic

Use a safer material this time!

9

User Study• Design

– 2 x 3– Full factorial– Between subjects

• Using Social Interaction Strategies– Yes (S)– No (NS)

• Style of Presentation• For Tutor Prompts

– Original (N)– Monoglossic (M)– Heteroglossic (H)


Social Interaction Strategies

Presentation Style

N + NS N + S

M + NS M + S

H + NS H + S

10

User Study: Learning Activity• Mechanical Engineering Course– Freshmen: Wrench Design• Underlying Concepts

– Force, Moment, Stress, Strength, …

– Collaborative Design Lab• Design a better wrench

– Ease of Use, Cost, Safety• Team of 3-4 students• Communicate using ConcertChat• Supported by Automated tutors

– Randomly assigned to conditions


11

User Study: Procedures1. Pre-Test: 10 minutes– Qs: 7 multiple-choice, 4 brief explanation

2. Collaborative Design Competition: 35 minutes– Interact with tutors

3. Post-Test: 10 minutes

4. Perception Survey: 5 minutes• Details in the paper

– 10 items 7-point Likert scale– Personality descriptors


12

User Study• Metrics– Learning Outcomes

– Perception Ratings (of Tutor , Task)

– Idea Contributions (by Students)• Count is estimated using dictionary

– how about, can we, we should, …– increase, decrease, steel, …

• Noisy Measure– For all conditions


13

Idea Generation: Review• Idea Generation & Learning

• Wang et. al., 2007 (Upcoming IJCSCL Article)• Depends on the Task

• Idea Generation in Groups– Process Loss

• Affect Productivity Reduce Learning

• Objective:– To create support that enable the benefits of idea

generating interactions with negative side-effects

• Instrument:– Automatically generated intervention prompts


14

Results• Learning Outcomes:

– No significant effects of either factor

• Perception Ratings:– Heteroglossic prompts:

• Happier with the discussion p<0.05• As good as human tutor p=0.08• Perceived task success p<0.08

– Heteroglossic & Monoglossic prompts(i.e. Style Consistency)• Liked the tutor very much p<0.05

– With Social Interaction Strategies• Tutor perceived more supportive & less pushy p<0.05


15

Results• Idea Generation Productivity

– As estimated by counts of idea contributions

• Social Interaction Strategies– No Significant Effect

• Presentation Style– Heteroglossic > Original – Heteroglossic > Monoglossic

• Interaction Effect• p<0.05

– Heteroglossic prompts could compensate for lack of social capabilities


16

Conclusions• Exploration of style / function of tutor prompts

• Among other factors (amount, timing, transfer, …)

– Perception Ratings• Preference towards Heteroglossic prompts & the use of

Social behavior

– Idea Generation• Heteroglossic style encourages more productive idea

generation

– Observation:• Unlike Wang et. al. 2007, no relationship between idea

generation & learning

Thank You• Acknowledgements

– Colleagues• Gregory Dyke

– CMU Mechanical Engineering Faculty & TAs• Maarten P. De Boer, Philip Leduc, Robert Reid, Joshua

Eickmeier, Mallory Elbert, Lina Gonzalez, Prerna Singh

– C-MITES• Ann Shoplik, Pamela Piskurich

– National Science Foundation

Questions

19

Prior Work• Benefits of

– 0.90σ : Collaborative Learning– 1.06σ : Adaptive Support (Tutor Agents)– 1.24σ : Combined

• Kumar et. al., 2007

• Students ignore/abuse Tutors

• Remedial Interaction Tactics– Attention Grabbing

• Chaudhuri et. al., 2008/2009

A20

wow that tutor is annoying

A13

Yeah

A13

STFU tutor!

B9

hi tutor

B9

tutor, you're a jerka7 so, got any ideas?

Tutor

Starting now, you have around 40 minutes to work together…

A2 hrmmma7 stupid tutor no one asked you for help

Tutor

There will be more potential for power generation ... Where might the increased heat input go instead?

St16

this tutor is really annoying

St5 Agreed


20

Presentation Style• Heteroglossia vs. Monoglossia– Lexical/Grammatical choices– Indicates social positioning of the speaker

• Heteroglossia– Awareness/Acknowledgement of multiple

perspectives– Criteria

• Martin & White, 2005– Propositional content must be asserted– Awareness to alternative perspective must be visible– Awareness is expressed grammatically or paraphrastically


21

Presentation Style• Examples


Original/Neutral

Think about how we can improve this?

Heteroglossic

I suggest that you think about how to improve this

Monoglossic

Think about how to improve this

Original/Neutral

Maybe we can use a safer material this time!

Heteroglossic

Using a safer material would be strongly advisable!

Monoglossic

Use a safer material this time!

Original/Neutral

We keep a factor of safety to avoid reaching yield stress.

Heteroglossic

It would make sense to keep a factor of safety to avoid reaching yield stress.

Monoglossic

Keep a factor of safety to avoid reaching yield stress.

22

Presentation Style– System of

Engagement• Martin & White, 2005

– Limited exploration of this system in our study


Conversational Strategies that Support Idea Generation Productivity in Groups Rohit Kumar Jack L....

Documents

Transcript of Conversational Strategies that Support Idea Generation Productivity in Groups Rohit Kumar Jack L....