Learning design & learning

analytics – building the links

Rebecca Ferguson

The Open University

What the Research Says: November 2014

Why learning design?

• Provides a set of tools and information to support a

learner-activity based approach

• Helps to show the costs and performance outcomes of

design decisions

• Puts the learning journey at the heart of the design

process

• Enables the sharing of best practice

• Helps partners choose and integrate a coherent range of

media, technologies and pedagogies

• Enables a consistent and structured approach to review

and analytics

Advantages of design for analytics

Helps to frame and focus analytics questions

What did they learn?… in relation to learning outcomes

Were they social?... when they were collaborating

Did they share links?... when encouraged to browse

Did they return to steps?... when encouraged to reflect

Helps to identify appropriate forms of analysis

The same step, but with a focus on

• Number of visits if content

• Length, quality, number of comments if conversational

• Dwell time and repeat visits if reflection

MOOC plannerPrompts designers

to block out different

types of learning

activity:

• Delivered

• Reflection

• Collaboration

• Conversation

• Networking

• Browsing

• Assessment

Planner and analytics

Delivered Content

(reading, watching, listening and observing)

Analytics: amount of content viewed, dwell time

Reflection

(thinking, considering and reflecting)

Analytics: returns to the same material, reflection

exercises completed, quality of reflection

Collaboration

(constructing, collaborating, defining and engaging)

Analytics: collaboration exercises completed, quality

of collaboration

MOOC map

• Guidance & support

• Content & experience

• Reflection &

demonstration

• Communication &

collaboration

Activity and learning design

Patterns of engagment: Coursera

Sampling: learners explored some course videos.

Auditing: learners watched most videos, but

completed assessments rarely, if at all

Disengaging: learners completed assessments at

the start of the course and then reduced their

engagement

Completing: learners completed most assessments

Kizilcec, R., Piech, C., and Schneider, E., 2013. Deconstructing disengagement:

analyzing learner subpopulations in massive open online courses. In LAK13

Cluster analysis

Patterns of engagementOn an eight-week MOOC

Samplers visit only briefly

[1, 0, 0, 0, 0, 0, 0, 0]

Strong starters do first assessment

[9, 1, 0, 0, 0, 0, 0, 0]

Returners come back in Week 2

[9, 9, 0, 0, 0, 0, 0, 0]

Mid-way Dropouts

[9, 9, 9, 4, 1, 1, 0, 0]

Nearly There drop out near the end

[11, 11, 9, 11, 9, 9, 8, 0]

Late Completers finish

[5, 5, 5, 5, 5, 5, 9, 9, 9]

Keen Completers do almost

everything [11, 11, 9, 9, 11, 11, 9, 9] Patterns vary with pedagogy

and learning design

Short MOOCs

Surgers concentrate their effort after the first week of a three-week

course. They do little in Week 1 other than submit their

assessment late, engage more in Week 2, but still submit their

assessment late (working on it in Week 3), and engage but do not

submit in Week 3. On average, they post one or two comments.

A typical engagement profile for this cluster is: [4, 6, 2]

Improvers fall behind in Week 1, submitting their first assessment

late. However, they engage more in Week 2 and by Week 3 they

are on schedule and submit their assessment on time. They view

the majority of steps on the course and typically post more than

one comment.

A typical engagement profile for this cluster is: [5, 6, 9]

Developing this work