Improving Online Educational Resources using Cognitive Science

Joseph Jay WilliamsOffice of Online LearningStanford University

September 2013

Slides & Discussion at tiny.cc/inacolwebinar

2

Improving Online Educational Resources using Cognitive Science(& Online Collaborations between Scientists and Educators)

Joseph Jay Williamsjosephjaywilliams@stanford.edu

www.josephjaywilliams.com/educationLytics Lab, Office of Online Learning, Graduate School of Education

Stanford UniversitySlides at tiny.cc/inacolwebinar

3

What’s different about Online Educational Resources?

• A lot!

• Align Scientists & Educators– Practical Improvements– Scientific Research

• Refine Resources through Repeated Improvement• Facilitate Collaboration• Like Wikipedia?

Content Exercise

4

Overview

• I. Applying cognitive science to online learning– I-a: How to increase motivation?– I-b: Increase learning without changing materials?– I-c: Teaching Learning Strategies– I-d: Lessons for Growth Mindset + Learning Strategies

• II. Online collaborations between Scientists & Educators– II-a: Learning, Education & Research Network (LEARN)– II-b: R.E.P.E.A.T. criteria for Research-Practice friendly resources

5

I. Reviewing & Synthesizing Cognitive Science Research

• Williams, J.J. (2013). tiny.cc/improveonlinelearningImproving Learning in MOOCs by Applying Cognitive Science. Paper to be presented at the MOOCshop Workshop, International Conference on Artificial Intelligence in Education, Memphis, TN.

• Harvey, A. G., Lee, J., Williams, J. J., Hollon, S., Walker, M., Thompson, M., & Smith, R. (in press). Improving outcome of psychosocial treatments by enhancing memory and learning. Perspectives in Psychological Science.

• www.josephjaywilliams.com/education

• Willingham, D. T. Why Don't Students Like School. Jossey-Bass (2010)• Willingham, D. T. (2012). When can you trust the experts: How to tell good science

from bad in education.

6

I-a: How to increase motivation?

• Many ways to increase motivation…• Change students’ implicit beliefs about whether

intelligence is fixed or malleable (Dweck, 2011; Yeager & Walton, 2012)

7

Implicit beliefs about Intelligence

• On a scale from 1 to 10, how much do you agree that?

• Your intelligence is something very basic about you that you can’t change very much.

• No matter how much intelligence you have, you can always change it quite a bit.

• Fixed Mindset vs. Growth Mindset (Dweck, 2006)– Self-fulfilling prophecy– Avoid uncomfortable challenges vs. Pursue Learning Opportunities– Avoid asking questions vs. Examining your mistakes

8

Research to link Science and Practice

• Experiment: Using Motivation Research to boost students’ learning of Math online on Khan Academy

• Approach:• 1. Quantify outcomes in real-world resource

• Math exercises at www.khanacademy.org

• 2. Synthesize recommendations from research• Believing intelligence is malleable increases motivation

• 3. Embed experiment to evaluate• Upcoming slides…

9

1. Learning from Math exercises on www.KhanAcademy.org

• Typical Exercise

1. Number of Problems Completed

2. Percent Correct

10

2. Synthesize Scientific Findings

• Williams, J.J. (2013)Improving Learning in MOOCs by Applying Cognitive Science. Paper presented at the MOOCshop Workshop, International Conference on Artificial Intelligence in Education, Memphis, TN.

• www.josephjaywilliams.com/education • Teaching Growth Mindset of intelligence• Research at Stanford by Dweck (2008) & Yeager &

Walton (2011)

11

3. Add motivational messagesPractice-as-usual

Remember, the more you practice the smarter you become!

Growth Mindset Message

12

3. Embedded in vivo Experiment

• Growth Mindset Message

• "Remember, the more you practice the smarter you become.”,

• "Mistakes help you learn. Think hard to learn from them.”

• Practice-as-usual

• Benefit of Growth Mindset Message?

Jascha Sohl-Dickstein

13

Number of Problems...

Practice-as-usual

Growth Mindset Message

Results: Problems Attempted & Accuracy

50 000+ students

Series1

Accuracy

14

Practice-as-usualGrowth Mindset Message

Positive Message

3. Add Positive messages

Some of these problems are hard. Do your best!

15

Does any positive message work?

• Growth Mindset Message

• "Remember, the more you practice the smarter you become.”,

• "Mistakes help you learn. Think hard to learn from them.”

• Positive Message• "Some of these problems are

hard. Just do your best."• "This might be a tough problem,

but we know you can do it.”

• Practice-as-usual

16

Number of Problems ...

Practice-as-usual

Growth Mindset Message

Positive Message

Results to Analyze

Series1

Accuracy

Research to link Science & Practice

1. Outcomes in Online

Resources

2. Recommendatio

ns from Research

3. Experiments

Khan Academy Math Exercises

Malleability of Intelligence

Ongoing Experiments

? ? ?

17

18

I-b: Increase learning without changing materials?

• Promote reflection with Socratic questions• Questions before: Problem-Based Learning• Questions during: Prompt for explanations• Questions after: Use assessments for instruction

Slides & Discussion at tiny.cc/inacolwebinar

19

Add object

to bucket

Integrate webpage

with internet

Content Exercise

Learning: Add vs. Integrate

Before: Start with Questions & Problems

• Problem Based Learning (Hmelo-Silver, 2006; Needham & Begg, 1998; Schwartz, 1998)

20

How do you…?Is it possible to…?

During: Request explanations

• Fonseca & Chi, 2011• Renkl, 1997• McNamara, 2004• Rittle-Johnson, 2006• Williams & Lombrozo, 2010

21

Why…? How?What are you thinking? What next?

22

Explanation and Learning

• Does explaining…– Provide a General boost to Learning Engagement– Selectively guide learners to look for patterns

• The Subsumptive Constraints Theory: Interpret target of why-explanation in terms of a broader generalization (Williams & Lombrozo, 2010)

• Discover general patterns (Williams & Lombrozo, 2010, Cognitive Science)

• Use pre-existing knowledge (Williams & Lombrozo, 2013, Cog. Psych.)

• May mistakenly overgeneralize by ignoring specific examples (Williams et al, 2013, JEP: General)

After: Use Assessments as Instructional Tools

• “Testing Effect” & Ten Benefits of Testing (Roediger et al, 2011)

23

Right away: Study+Study ~= Study+TestDay/Week later: Study+Study < Study+TestLearners think: Study+Study > Study+Test

After: Achieve Mixing Effect with assessments

• Mixing Effect (Rohrer, 2009)

24

25

Applying Cognitive Science

• Learning: Add vs. Integrate Knowledge (Instructionism vs. Constructivism)

• Questions before: Problem-Based Learning• Questions during: Prompt for explanations• Questions after: Use assessments for instruction• Further resources:

www.josephjaywilliams.com/education

I-c: Teaching Learning Strategies

• Spend a lesson teaching a concept vs. general strategy?

• Online: Iteratively refine excellent lessons• Online: Repeatedly reinforce habits & educational

behaviors

• Teach “What? Why? How?” self-questioning/explanation strategies

26

27

Add “Socratic” prompts to explain & reflectClickable link. + Prompts embedded into hints.

Click here to learn about the What? Why? How? strategy

28

Embedded Prompts between Hint/Solution Steps

29

Self-questioning strategy: What? Why? How?

30

I-d: Lessons for Growth Mindset + Learning Strategies

• Learning Assistant: tiny.cc/learningassistant

• Short 5 minute videos• Guided prompts

• Growth Mindset• Self-Questioning strategy

• Experiment: How do lessons influence grades?

31

II. Collaborations between Scientists & Educators

• How to gain best of both worlds?• Ask Questions, “Crowdsource” answers• The Internet removes barriers to communication &

connects people

32

II-a: Learning, Education And Research Network (LEARN)

• www.learnnetwork.net• Creating a virtual community:• Educators, Researchers, Developers• Mailing List, & Member list• Discussion Forum – any teacher can ask a question,

so can any researcher• Wiki of resources• Collaborate in improving online educational

resources

33

II-b: R.E.P.E.A.T. Criteria: Research on Online Resources

• Can you collaborate to help your students & improve online resources?

• Yes, if Online Educational Resources are:– Realistic– Experimental– Product– Evaluated– Accessible– Theoretically motivated

• REPEAT – iteratively improve through revision & collaboration• http://edlab.tc.columbia.edu/ • Organizations like iNACOL!

34

Review

• I. Applying cognitive science to online learning– I-a: How to increase motivation?– I-b: Increase learning without changing materials?– I-c: Teaching Learning Strategies– I-d: Lessons for Growth Mindset + Learning Strategies

• II. Online collaborations between Scientists & Educators– II-a: Learning, Education & Research Network (LEARN)– II-b: R.E.P.E.A.T. criteria for Research-Practice friendly resources

• Send advice & questions! josephjaywilliams@stanford.edu

Contact Info

• Joseph Jay Williams• josephjaywilliams@stanford.edu• www.josephjaywilliams.com/education• Slides at http://tiny.cc/inacolwebinar