Building Analytics Capability @open.edu

25
Simon Buckingham Shum Professor of Learning Informatics Knowledge Media Institute, The Open University, UK http://simon.buckinghamshum.net @sbskmi JISC CETIS 2013 Conference: Analytics and Institutional Capabilities Building Analytics Capability @open.edu

description

Invited presen

Transcript of Building Analytics Capability @open.edu

Page 1: Building Analytics Capability @open.edu

Simon Buckingham Shum

Professor of Learning Informatics Knowledge Media Institute, The Open University, UK http://simon.buckinghamshum.net @sbskmi

JISC CETIS 2013 Conference: Analytics and Institutional Capabilities

Building Analytics Capability @open.edu

Page 2: Building Analytics Capability @open.edu

bit.ly/OULAprof  

Page 3: Building Analytics Capability @open.edu

Same  outcomes,    but  higher  scores?  

 Learning  Analy=cs  as    

Evolu&onary  Technology.  Same  training  +  educa=onal  paradigms  

 •  more  engaging  •  beBer  assessed  •  beBer  outcomes  

•  deliverable  at  scale  3  

Page 4: Building Analytics Capability @open.edu

Learning  dynamics  we    couldn’t  assess  before?  

 Learning  Analy=cs  as    

Revolu&onary  Technology.  A  vehicle  for  paradigm  shiF?  

 •  interpersonal  learning  networks  •  quality  of  discourse  +  wri=ng    •  lifelong  learning  disposi=ons  •  problem  solving  strategies  

•  lifewide  learning  

Page 5: Building Analytics Capability @open.edu

open.edu BI perspective

5

Page 6: Building Analytics Capability @open.edu

OU  data  warehouse  (in  progress)  

Data  Warehouse  

IT corral key institutional data in the

central warehouse 1 IT provide data dictionary 2

IT provide data marts and cubes for commonly used data sets

3 Business data users propose

action 5

Explore the challenge/issue/problem/opportunity/question using SAS/preferred tool

4 “Data Wranglers”

assist staff in understanding BI OU Analytics

Board

Page 7: Building Analytics Capability @open.edu

open.edu VLE

perspective 7

Page 8: Building Analytics Capability @open.edu

VLE  Analy;cs  @  the  OU  

Virtual  Learning  

Environment  

Usage  sta;s;cs  at  system,  faculty  and  module  level  –  general  paCerns  

‘Par;cipa;on  Tracking’  func;on  to  track  individual  students’  interac;on  with  specific  

online  learning  ac;vi;es  In  pilot  2012/13  

Data  Warehouse  

Page 9: Building Analytics Capability @open.edu

VLE  Analy;cs  @  the  OU  

Virtual  Learning  

Environment  

Usage  sta;s;cs  at  system,  faculty  and  module  level  –  general  paCerns  

‘Par;cipa;on  Tracking’  func;on  to  track  individual  students’  interac;on  with  specific  

online  learning  ac;vi;es  In  pilot  2012/13  

Data  Warehouse  

Page 10: Building Analytics Capability @open.edu

VLE  Analy;cs  @  the  OU  

Virtual  Learning  

Environment  

Usage  sta;s;cs  at  system,  faculty  and  module  level  –  general  paCerns  

‘Par;cipa;on  Tracking’  func;on  to  track  individual  students’  interac;on  with  specific  

online  learning  ac;vi;es  In  pilot  2012/13  

Data  Warehouse  

Page 11: Building Analytics Capability @open.edu

open.edu predictive modelling

11

Page 12: Building Analytics Capability @open.edu

Predictive analytics

Registra=on  PaBern  

CRM  contact  

VLE  interac=on  

Assignment  grades  

Demo-­‐graphics  

? How early can we predict likelihood of dropout, formal withdrawal, failure? Now exploring conventional statistics, machine learning and growing datasets New fees regime may well change student behaviour…

Library  interac=on  

OpenLearn  interac=on  

Futurelearn  interac=on  

Social  App  X  interac=on  

OU  track  record  

Page 13: Building Analytics Capability @open.edu

OU Analytics: Predictive modelling

§  Probability models help us to identify patterns of success that vary between: §  student groups / areas of

curriculum / study methods §  Benefits

§  provide a more robust comparison of module pass rates

§  support the institution in identifying aspects of good performance that can be shared, and aspects where improvement could be realised

13 OU Student Statistics & Surveys Team, Institute of Educational Technology

Best predictors of future success:

previous OU study data – quantity

and results

Page 14: Building Analytics Capability @open.edu

Improving student retention with predictive analytics

A.L. Wolff and Z. Zdrahal (2012). Improving Retention by Identifying and Supporting “At-risk” Students. EDUCAUSE Review Online, July-August 2012. http://www.educause.edu/ero/article/improving-retention-identifying-and-supporting-risk-students

4 predictive models: final result (pass/fail) final numerical score drop in the next TMA score of the next TMA

Demo- graphics

Previous results

VLE activity

Page 15: Building Analytics Capability @open.edu

open.edu Library

perspective 15

Page 16: Building Analytics Capability @open.edu

Learning Analytics – the Library dimension

http://www.flickr.com/photos/davepattern/6928727645/sizes/o/in/photostream/

Library Impact Data Project – Huddersfield University

‘Students who looked at this article also looked at this article’

‘Students on your course are looking at these articles’

Student achievement

Library use

Recommender services

Page 17: Building Analytics Capability @open.edu

open.edu Research

perspective 17

Page 18: Building Analytics Capability @open.edu

Visualizing  and  filtering  social  ;es  in  SocialLearn  by  topic  and  type  

Schreurs,  B.,  Teplovs,  C.,  Ferguson,  R.,  De  Laat,  M.  and  Buckingham  Shum,  S.,  Visualizing  Social  Learning  Ties  by  Type  and  Topic:  Ra;onale  and  Concept  Demonstrator.  In:  Proc.  3rd  Interna6onal  Conference  on  Learning  Analy6cs  &  Knowledge  (Leuven,  BE,  8-­‐12  April,  2013).  ACM  hCps://dl.dropbox.com/u/15264330/papers/Schreurs-­‐etal-­‐LAK2013.pdf  

Page 19: Building Analytics Capability @open.edu

-60

-40

-20

0

20

40

60

80

9:2

8 9

:32

9:3

6

9:4

0

9:4

1

9:4

6

9:5

0

9:5

3

9:5

6

10:

00

10:

05

10:

07

10:

07

10:

09

10:

13

10:

17

10:

23

10:

27

10:

31

10:

35

10:

40

10:

45

10:

52

10:

55

11:

04

11:

08

11:

11

11:

17

11:

20

11:

24

11:

26

11:

28

11:

31

11:

32

11:

35

11:

36

11:

38

11:

39

11:

41

11:

44

11:

46

11:

48

11:

52

11:

54

12:

00

12:

03

12:

04

12:

05

Average Exploratory

Discourse analytics on webinar textchat

Sheffield, UK not as sunny as yesterday - still warm Greetings from Hong Kong Morning from Wiltshire, sunny here!

See you! bye for now! bye, and thank you Bye all for now

Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664

Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar, but if we zoom in on a peak…

Page 20: Building Analytics Capability @open.edu

Discourse analytics on webinar textchat

-100

-50

0

50

100

9:2

8 9

:40

9

:50

1

0:00

1

0:07

1

0:17

1

0:31

1

0:45

1

1:04

1

1:17

1

1:26

1

1:32

1

1:38

1

1:44

1

1:52

1

2:03

Averag

Classified as

“exploratory talk”

(more

substantive for learning)

“non-exploratory

Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations?

Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664

Page 21: Building Analytics Capability @open.edu

Discourse analytics on webinar textchat Visualizing by individual user. The gradient of the threshold line is

adjusted to every 5 posts in 6 classified as “Exploratory Talk”

Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664

Page 22: Building Analytics Capability @open.edu

Analytics for “21st Century Competencies & Learning Dispositions”

Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, Vancouver). Eprint: http://oro.open.ac.uk/32823

Different social network patterns in different contexts

may load onto Learning

Relationships

Questioning and challenging may load onto Critical Curiosity

Sharing relevant resources from other

contexts may load onto Meaning Making

Repeated attempts to pass an online test

may load onto Resilience

Page 23: Building Analytics Capability @open.edu

open.edu coming soon…

23

Page 24: Building Analytics Capability @open.edu

On the horizon… MOOCs + Analytics…

Ethics

What Data? Biz Models

‘vs’ Open

Partnerships/Collab

Research

http://people.kmi.open.ac.uk/sbs/2013/01/emerging-mooc-data-analytics-ecosystem

Educ Research at SCALE

Page 25: Building Analytics Capability @open.edu

On the horizon… Educational Data Scientists