Moving through MOOCs: Pedagogy, Learning and Patterns of EngagementRebecca Ferguson, Doug Clow (OU)Russell Beale, Alison J Cooper (Birmingham)Neil Morris (Leeds)Siân Bayne, Amy Woodgate (Edinburgh)

Current context

Students seek not merely access, but access to success

“John Daniel, 2012

% complete from: www.katyjordan.com/MOOCproject

”

Patterns of engagement: Coursera

●Samplinglearners explored some course materials

●Auditinglearners watched most videos, but completed assessments rarely, if at all

●Disengaginglearners completed assessments at the start of the course and then reduced their engagement

●Completinglearners completed most assessments

MOOC designers can apply this simple and scalable categorization to target interventions and develop adaptive course features

“”

Coursera study Kizilcec, R., Piech, C., and Schneider, E., 2013. Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. LAK13

4

ReplicationOpen University FutureLearn data

Replication

MOOC1 MOOC2 MOOC3 MOOC4

Subject area Physical sciences

Life sciences

Arts Business

M 51% 39% 32% 35%

F 48% 61% 67% 65%

Participants 5,069 3,238 16,118 9,778

Fully participating 1,548 684 3,616 1,416

Participation rate 31% 21% 22% 14%

5

Calculating an activity profileReplicating the method

●T = on track (3) undertook the assessment on time

●B = behind (2) submitted the assessment late

●A = auditing (1)engaged with content but not assessment

●O = out (0)did not participate

Replication

6

ReplicationIdentifying dissimilarity between engagement patterns

Assigned numerical value to each label●On track = 3●Behind = 2●Auditing = 1●Out = 0Calculated L1 norm for each engagement patternUsed that as the basis for one-dimensional k-means clusteringRepeated clustering 100 times and selected solution with highest likelihoodFocused on extracting four clusters

Replication

7

ReplicationCoursera and FutureLearn results were different

●Samplinglearners explored some course materials

●Auditinglearners watched most videos, but completed assessments rarely, if at all

●Disengaginglearners completed assessments at the start of the course and then reduced their engagement

●Completinglearners completed most assessments

√

√

xx

They also differed when we tried●Different values for k●A one-dimensional approach●Running k means directly on engagement profiles

Replication

8

FutureLearn is differentConversation is a central feature

Sharples, M., & Ferguson, R. (2014). Innovative Pedagogy at Massive Scale: Teaching and Learning in MOOCs. ECTEL 2014.

9

Revising the numeric values

OU FL study

1 only visited content (for example, video, audio, text)

2 commented but visited no new content

3 visited content and commented

4 did the assessment late and did nothing else that week

5 visited content and did the assessment late

6 did the assessment late, commented, but visited no new content

7 visited content, commented, late assessment

8 assessment early or on time, but nothing else that week

9 visited content and completed assessment early / on time

10 assessment early or on time, commented, but visited no new content

11 visited, posted, completed assessment early / on time

10

Typical engagement profilesThese profiles apply to an eight-week course

●Samplers visit only briefly[1, 0, 0, 0, 0, 0, 0, 0] – 1 means they visited content

●Strong Starters do first assessment[9, 1, 0, 0, 0, 0, 0, 0] – 9 means they visited content and did assessment on time

●Returners come back in Week 2 [9, 9, 0, 0, 0, 0, 0, 0]

●Mid-way Dropouts[9, 9, 9, 4, 1, 1, 0, 0] – 4 means they submitted assessment late

●Nearly There drop out near the end[11, 11, 9, 11, 9, 9, 0, 0] – 11 means full engagement, 8 means submission on time

●Late Completers finish[5, 5, 5, 5, 5, 9, 9, 9] – 5 means they viewed content and submitted late

●Keen Completers do almost everything [11, 11, 9, 9, 11, 11, 9, 9]

OU FL study

11

Samplers & Strong starters

Samplers (1, 0, 0, 0, 0, 0, 0, 0)

●The largest group in all MOOCs

●Typically accounted for 37% – 39% of learners

●Visited the materials, but only briefly

●Active in a small number of weeks

●25% – 40% joined after Week 1

●Very few Samplers posted comments (6% – 15%)

●Almost no Samplers submitted any assessment

Strong starters (9, 1, 0, 0, 0, 0, 0, 0)

●All Strong Starters submitted the first assignment

●Engagement dropped off sharply after that

●A little over a third of them posted comments

●Typically posted fewer than four comments

OU FL study

12

Returners & Mid-way dropouts

Returners (9, 9, 0, 0, 0, 0, 0, 0)

●Completed the assessment in the first week

●Completed the assessment in the second week

●Then dropped out

●Over 97% completed those two assessments, although some submittted late

●No Returner explored all course steps

●Average amount of steps visited varied (23% – 47%)

Mid-way dropouts (9, 9, 9, 4, 1, 1, 0, 0)

●A much smaller cluster (6% of learners on MOOC1, 7% on MOOC4)

●These learners completed three or four assessments

●They dropped out around halfway through the course

●Mid-way dropouts visited about half the steps on the course

●Just under half posted comments

●Posted just over six comments on average

OU FL study

13

Nearly There

Nearly there (11, 11, 9, 11, 9, 9, 0, 0)

●Another small cluster (5% – 6% of learners)

●Consistently completed assessments

●Dropped out just before the end of the course

●Visited around 80% of the course

●Submitted assignments consistently (>90%) and typically on time until Week 5

●Activity then declined steeply

●Few completed the final assessment

●None completed the final assessment on time

OU FL study

14

Late completers & keen completers

Late completers (5, 5, 5, 5, 5, 9, 9, 9)

●Submitted the final assessment

●Submitted most other assessments

●However, either submitted late or missed some assessments

●Each week, more than 94% of this cluster submitted their assessments

●More than three-quarters submitted the final assessment on time (78% – 90%)

●Around 40% of them postedcomments (76% did so on MOOC3)

Keen completers (11, 11, 9, 9, 11, 11, 9, 9)●Accounted for 7% – 23% of learners●All Keen Completers submitted all assessments

●More than 80% of these were submitted on time

●Typically, Keen Completers visited more than 90% of course content

●Over two-thirds contributed comments(68% – 73%)

●Mean number of comments varied from 21 to 54

OU FL study

15

Cross-university datasetFutureLearn data from four universities

Cross-university

Name Duration University Discipline Active learners

LongMOOC1 8 OU Hard science

5,069

LongMOOC2 7 Edinburgh Hard science

10,136

TalkMOOC3 6 Edinburgh Politics 6,141

ShortMOOC4 3 Birmingham Medical science

6,839

ShortMOOC5 3 Leeds Medical science

4,756

16

Values for k used in this studyDifferent values were used for the three study phases

Cross-university

Name Phase 1 Phase 2 Phase 3

LongMOOC1 7 – –

LongMOOC2 7 – –

TalkMOOC3 – 7 3

ShortMOOC4 – 7 4

ShortMOOC5 – 7 5

Phase 1: Best-fit value for k aligned with OU studyPhase 2: Testing k=7 where this was not the best fitPhase 3: Most suitable value for k in each set of data

17

Phase two: k ≠ 7Why k≠7 in Talk MOOC3

Phase two

The absence of assessment in TalkMOOC3 limited its coding profile

1 only visited content (for example, video, audio, text)2 commented but visited no new content3 visited content and commented4 did the assessment late and did nothing else that week5 visited content and did the assessment late6 did the assessment late, commented, but visited no new content7 visited content, commented, late assessment8 assessment early or on time, but nothing else that week9 visited content and completed assessment early / on time10 assessment early or on time, commented, but visited no new content11 visited, posted, completed assessment early / on time

18

Phase two: k ≠ 7Why k≠7 in ShortMOOC4 and ShortMOOC5

Phase two

Three clusters are indistinguishable in a three-week MOOCReturners who come back in Week 2

Mid-way Dropouts who drop out mid-course

Nearly There who drop out near the end

With only three opportunities for late submission, there are no

Late Completers (who typically submit assessment late five times)

The three-week course design means other clusters emerge, such as:

Surgers concentrate their effort after the first week of a three-week course

Improvers fall behind in Week 1, begin to catch up in Week 2 and complete on time

19

Phase three: suitable values for kTalkMOOC3: k=3

Phase three

Quiet (1, 0, 0, 0, 0, 0)

●The largest cluster

●Visit a quarter of course steps

●Do not comment in first week

●Only 7% comment at all

●Only 9% engage with second half of course

Contributors (3, 1, 1, 0, 0, 0)

●19% of cohort

●Visit 38% of course steps

●Every cluster member posts in first week of course

●Half do not comment again

Consistent engagers(3, 3, 3, 3, 1, 1)

●11% of cohort

●Visit 82% of course steps

●Engage throughout course

●Every cluster member posts a comment

●95% contribute more than three comments

●7% contribute more than 100 comments

20

Phase three: suitable values for kShortMOOC4: k=4

Phase three

Very weak starters (2, 1, 0)

●The largest cluster

●Visit 20% of steps

●20% do not engage in first week

Strong starters (truncated) (10, 1, 0)

●17% of cohort

●Submit week 1 assessment

●Do not submit another assessment

●Almost half post comment

Returners (truncated)(3, 3, 3, 3, 1, 1)

●Most submit week 1 assessment

●All submit week 2 assessment

●Half submit at least one comment

Keen completers (truncated) (9, 9, 9)

●Visit more than 90% of steps

●Submit work on time

●Engage throughout

21

Phase three: suitable values for kShortMOOC5: k=5

Phase three

Samplers (truncated) (1, 0, 0)

●Visit few steps

●Includes many latecomers (>25%)

●Very few submit assessment Strong starters (truncated)

(9, 1, 0)

●Submit week 1 assessment

●Do not submit another assessment

Returners (truncated)(8, 8, 2)

●Most submit week 1 assessment

●All submit week 2 assessment

Keen completers (truncated) (9, 9, 9)

●Visit more than 90% of steps

●Submit work on time

●Engage throughout

Improvers (5, 6, 9)

●Activity increases each week

●Final assessment submitte on time

22

Learning design and pedagogy

●The Coursera Study suggested that MOOC designers would be able to apply the four engagement patterns they had identified ‘to target interventions and develop adaptive course features’

●These subsequent studies show that this is not necessarily the case –engagement patterns are not consistent across MOOCs

●Changes to the basic pedagogic elements of a course are associated with shifts in patterns of engagement.

●Shifts in pedagogic approach can change the elements of a course that can be regarded as key

●Changes to some elements of learning design can change learners’ patterns of engagement with a MOOC

23

Shorter courses

●Reducing course length does not necessarily increase engagement

●Many learners do not approach a three-week course in the same way as an eight-week course

●Many focus their attention on later weeks and may miss out the content and activities in the first week

●A three-week course offers limited opportunities to get ahead of, or behind, the cohort

●It is possible to dip in at different points without losing the sense of being a cohort member

24

Improving learning and learning environments

Closing the loop

●Previews of course material would allow Samplers to make a more informed decision about whether to join the course

●Sign-up pages could draw attention to the problems experienced by those who are out of step with the cohort

●Discussion steps for latecomers could support those who fall behind at the start

●Prompts might encourage flagging learners to return and register for a subsequent presentation

●Bridges between course weeks could indicate links and point learners forward

25

View these slides at www.slideshare.net/R3beccaF

Rebecca Ferguson @R3beccaFDoug Clow @dougclow