StoryStation

Peter Wiemer-Hastings Judy RobertsonKath Glasswell DePaul University

University of EdinburghUniversity of Illinois at Chicago

Motivation

• Learning to write (well) is hard• Multiple constraints / voices

(Flower, 1994)– Reader: What do I want R to know? What do I

know about what R knows? What do I want from R?

– Situation: What am I supposed to write about? What do I know about my topic? What do I want to say about it?

– Socio-linguistic context: How can I say this in a way that my teacher will accept?

Metacognitive Double Whammy

• Students must learn how to think about meeting the demands of multiple, possibly conflicting constraints.

• Students ordinarily don’t know what those constraints are.

AI to the rescue!?

• Appropriate feedback on compositions can point students in the right direction.

• Teachers don’t have time to give the best feedback to every student all the time.

• NLP techniques can analyze compositions, generate feedback.

• Associating different types of feedback with different animated agents might help the students learn the various constraints.

Research goals

• To investigate the effect of animated pedagogical agents on 10-11 year old pupils’…

– motivation,– satisfaction,– learning (eventually).

• To develop an intelligent tutoring system to help children write stories

Agenda

• Previous work• Design process• StoryStation features• Evaluation – preliminary results• Future directions

Previous research on agents• Mayer and Wittrock, 1996:

– With agents, students have more social and emotional investment, more motivated, try harder, problem solve better.

• Herman the Bug (Lester et. al, 1997)– Students more motivated when using agents, and learn more.

• Johnson, Rickel, and Lester, 2000– increased communication bandwidth and social interaction with agents

• Review of agent research (Dehn and van Mulken, 2000)– Some serious methodological problems. No control in Lester et al.

• Moreno, Mayer, and Lester, 2001: – Included control. Agents are social actors, increase motivation,

learning, transfer. Voice is important, face not.

Previous writing research• Previous writing environments were

designed to support cognitive processes of writing (e.g. Sharples 1988), based on Flower and Hayes (1980) theory.

• Writing apprehension (Madigan, Linton and Johnson, 1996)

• More recently, interest in providing emotional support for writers (e.g. T’riffic Tales project)

StoryStation basic strategy: Provide positive feedback to encourage pupils to take responsibility for their own learning

Design process

• Requirements gathering with 2 teachers and 37 pupils

• Integration of National English Language Curriculum goals

• Design team of 8 pupils and 2 teachers, working on the project regularly for 18 months

• Observation of classroom practice• 3 pilot studies

StoryStation features

• Spelling• Word banks• Dictionary• Thesaurus• Vocabulary• Characterization• Plot (in progress)

Story re-telling Task

• Students hear a story-teller present a story (or a video)

• Students are asked to create their own version with StoryStation.

• Scaffolds student’s writing.• Facilitates text analysis.

Evaluation – 18 teachers• Positive feedback is a good feature,

contrasts with programs which simply point out mistakes.

• Useful for independent learning because it teaches pupils how to self correct.

• Good for pupils to get instant feedback rather than waiting till the next day.

• Would like StoryStation to help them in assessing pupils’ progress

• Generally, teachers are surprisingly trusting of technology

Evaluation - pupils• 60 pupils (10 –12 year olds)• Half used agents version, half text-only• Pupils:

– Listened to 10 minute recording of “The Screaming Skulls of Calgarth”

– Watched demo of StoryStation– Wrote their own version of “Screaming Skulls” using

StoryStation (40 minutes to 75 minutes, depending on timetabling constraints)

– Filled in questionnaire – Answered structured interview questions

Children’s comments on StoryStation

• “I’d probably buy it for my home computer” – Roseanna, age 12

• “If I had StoryStation on my computer I’d definitely use it more than Microsoft Word”– Rosemary age 12

• “It’s really good and it helps you to write better”Shenel, age 12

• “It makes writing fun. You enjoy it more when you’ve got the helpers there” – Joanna, age 11

Questionnaire results

6. StoryStation makes writing stories easier7. I think I need someone to help me use

StoryStation8. I think the StoryStation advice was

useful9. A teacher is more helpful than

StoryStation

10. StoryStation is boring

1. I enjoyed using StoryStation2. I think StoryStation made my writing worse3. I think I would like to use StoryStation again4. Using StoryStation help me to write better5. I found StoryStation confusing

0

2

4

6

8

10

1 2 3 4 5 6 7 8 9 10questions

mea

sure

men

ts (

cm)

non-agent

agent

Non-agent Agent

Girls SD Boys SD Girls SD Boys SD

1. I enjoyed using StoryStation 0.88 0.20 0.90 0.12 0.97 0.06 0.90 0.13

2. I think StoryStation made my writing worse 0.06 0.06 0.09 0.12 0.02 0.02 0.07 0.11

3. I think I would like to use StoryStation again 0.85 0.22 0.86 0.18 0.98 0.06 0.93 0.13

4. Using StoryStation help me to write better 0.67 0.32 0.78 0.24 0.71 0.08 0.73 0.29

5. I found StoryStation confusing 0.22 0.29 0.15 0.24 0.04 0.06 0.10 0.13

6. StoryStation makes writing stories easier 0.85 0.20 0.78 0.32 0.95 0.16 0.76 0.26

7. I think I need someone to help me use StoryStation 0.08 0.09 0.11 0.14 0.15 0.21 0.18 0.20

8. I think the StoryStation advice was useful 0.78 0.29 0.85 0.18 0.89 0.03 0.86 0.16

9. A teacher is more helpful than StoryStation 0.28 0.22 0.36 0.29 0.24 0.28 0.33 0.32

10. StoryStation is boring 0.12 0.16 0.09 0.13 0.11 0.28 0.09 0.17

Nice picturesI think the StoryStation advice was useful

software version

AgentNon-agent

Est

ima

ted

Mar

gin

al M

ea

ns

8.5

8.0

7.5

7.0

6.5

6.0

5.5

5.0

4.54.0

3.5

3.0

2.5

2.0

1.5

1.0

.50.0

gender

Boys

Girls

Using StoryStation helped me to write better

software version

AgentNon-agent

Est

imat

ed M

argi

nal M

eans

8.0

7.5

7.0

6.5

6.0

5.5

5.0

4.5

4.0

3.5

3.0

2.5

2.0

1.5

1.0

.50.0

gender

Boys

Girls

Total interactions

software version

Non-agentAgent

Est

imat

ed M

argi

nal M

eans

11

10

9

8

7

6

5

4

3

2

1

0

Gender

Boys

Girls

Some issues• Children’s thoughts about agents

– Mixed ideas about whether agents are “real”.

• Children’s comparisons of teachers to StoryStation

• “A teacher can tell from your eyes what you’re thinking, but StoryStation can’t” Saima, age 11

• Social relationships: The teacher can shout at you, but you can shout at StoryStation

• Knowledge: Take StoryStation’s advice more because it is a computer

Current Proposal

• NSF-ROLE grant proposal with Elliott and Glasswell

• Clean up the metacognitive hypotheses and evaluations

• Address some of the questions about agents.

Conclusions

• Children think StoryStation is useful for writing stories

• StoryStation is motivating to children• Agent users more likely to find

StoryStation advice helpful, and want to use it again

• Agent users less likely to find StoryStation confusing

• Interesting social / gender interactions

Challenges

• How to foster metacognition– “Appropriate” feedback depends on genre,

context, audience, etc– Dialog about student choices?

• How to measure learning• How/where to get funding to

continue

Questions or Comments?