Scientifically Informed Digital Learning Interventions

Financial and Intellectual Support:• The William and Flora Hewlett Foundation• The National Science Foundation• A.W. Mellon Foundation• Carnegie Mellon University

One Example: The Open Learning Initiative at Carnegie Mellon

The Challenge• To learn ways to design and build fully

web-based courses which by rigorous assessments are proven to be as good or better than traditional teaching methods

• Why?– Increasing access– Improving effectiveness– Providing flexibility for faculty and students– Containing costs

A Flaw and an Opportunity

• Current structure of higher education presents substantial roadblocks to the application of proven results and methodologies from the learning sciences

• eLearning interventions, developed by teams rather than individuals, are more conducive to making the practice of education more scientific and effective

OLI Guiding Assumptions• Digital learning interventions can make a

significant difference learning outcomes• Designs grounded in contemporary learning

theory and scientific evaluation have the best chance of achieving that goal

• A possible, acceptable outcome of the OLI efforts is failure or mixed failures and successes – we are doing “action research,” not promoting eLearning for its own sake

OLI Guiding Assumptions• Formative assessment will be a major

feature (and a major component of the cost) of the designs and iterative improvements of the courses

• IT staff working with faculty is too limited a partnership – learning scientists, HCI experts, and assessment experts must be part of design, development, production and iterative improvement

Open Learning Initiative Courses• Statistics• Modern Biology• Chemistry• French• Engineering Statics• Causal and Statistical Reasoning• Economics• Logic and Proofs• Physics• Empirical Research Methods• Computational Discrete Mathematics

Try it Yourself

• http://www.cmu.edu/oli

• Don’t expect an “OCW experience”…this project has a different set of goals than OCW

• “Clicking around” will be unsatisfying: these interventions are designed to support a novice learner in acquiring knowledge working on their own

Key Elements in OLI Courses

• Theory Based: Course and individual lesson designs based on current theories in the learning sciences

• Feedback Loops: Courses record student activity for robust feedback mechanisms

• Diversity of Perspectives, Roles and Contexts: Courses developed and deployed by teams that include faculty content experts, learning scientists, software engineers

Theory Based: Build on Prior/Informal Knowledge

Theory Based: Provide Immediate Feedback in the Problem Solving Context

Theory Based: Promote Authenticity, Flexibility & Applicability

• Learning environments with ambiguous problems that require flexible application of procedural knowledge

Feedback Loops in Learning

Evaluation• Chemistry: Post-test scores by treatment group show

significant positive correlation for the OLI treatment. Most significant indicator was time spent in Virtual Lab Activities – made all other variables drop out.

• Biology: End of the 3rd week showed an advantage for the OLI section. There was a positive and significant association between students’ time spent working on particular activities and performance on quiz questions testing the corresponding topics even after total time with OLI has been regressed out

Evaluation• Statistics 1st Study:

n Average % correct

Pre 488 43.3

Post 488 51.2

Increase: 7.9% Increase: 7.9% [t(487) = 13.8, p <.001][t(487) = 13.8, p <.001]

n Average % correct

Pre 24 55.8

Post 24 66.5

Increase: 11.7%Increase: 11.7%[t(23) = 4.7, p <.001][t(23) = 4.7, p <.001]

CAOS Sample:CAOS Sample:

CMU OLI Course Sample:CMU OLI Course Sample:

Evaluation

Measured learning Outcome

% correct CAOS % correct CMU

Pre Post Pre Post

Box plots provide accurate estimates of % data above & below only for quartiles

22.2 22.2 22.2 50.0

Correctly estimate and compare SD’s for different histograms.

31.5 46.4 66.7 83.3

41.8 46.4 59.3 75.0Correlation does not imply causation

51.9 49.4 48.1 70.8

Calculating appropriate conditional probabilities given table of data

49.6 47.4 70.4 70.8

Evaluation

Accelerated Learning Study

• Taught Carnegie Mellon Introductory Statistics course in a blended mode (one in class meeting per week) in half a semester

• The OLI Statistics course was the “textbook”• OLI course provided the professor immediate

feedback on students’ performance• We compared learning outcomes in the two

different treatments

Accelerated Learning Study• OLI students significantly outperformed

Traditional “control” students on the CAOS post-test. CAOS Post-test

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

OLI TRAD

Pro

port

ion

Corr

ect

Accelerated Learning Study• OLI students showed significantly greater gains (pre

to post) than the Traditional “control” students on the CAOS test.

0

0.05

0.1

0.15

0.2

0.25

OLI TRAD

Gain

Score

Student Satisfaction

– End of course survey for online section:

• All students reported at an increase in their interest in statistics.

• 75% Definitely Recommend

25% Probably Recommend

0% Probably not Recommend

0% Definitely not Recommend

Feedback Loop – Current Research

Instructors can use such data to adjust their teaching to students’ needs.

Learning Curve Analysis on Stoichiometry Data

The Vision – Digital Dashboard for Teaching and Learning:• Instructor assigns students to work

through online instruction• System collects data as students work • System automatically analyzes and

organizes the data to present instructor with the students’ current “learning state”

• Instructor reviews this data summary and adapts instruction accordingly

The Anticipated Benefits

• Instructors get a window onto students’ progress

• They can adapt their teaching accordingly

• Students get better feedback to monitor and adjust their learning

• Strengthens the student-instructor connection

Core OLI Community• Faculty Content Experts• Learning Scientists• Human Computer Interaction• Software Engineers• Evaluation/Assessment Specialists• Learners• A community of scholars from diverse

disciplines who are committed to improving quality and access to instruction. The collaborative nature of the OLI course design process inspired participating faculty to rethink their approach to classroom teaching.

“Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity”

Herbert Simon

www.cmu.edu/oli

joelms@cmu.educthille@cmu.edu

(Candace Thille – Director)