How do Learning Analytics “act” in Education? Simon Buckingham Shum Knowledge Media Institute, Open University UK simon.buckinghamshum.net twitter @sbskmi

ESRC  Code  Acts  in  Educa/on  seminar  series  h.p://codeactsineduca7on.wordpress.com  University  of  Edinburgh,  9th  May  2014  

h.ps://twi.er.com/Wiswijzer2/status/414055472451575808  

Siri is smart…

“Find code acts in education”

Siri is smart…

“Find code acts in education”

h.ps://twi.er.com

/Wisw

ijzer2/status/414055472451575808   “Note:  check  the  

huge  difference  between  knowing  and  measuring…”  

A word of introduction… What does this tell you?

Learning  Technology  

KMi,  OU  

AI  &  Argumenta7on  

Learning  Disposi7ons  

Human-‐Centred    Informa7cs  

Learning  Analy7cs  

Seman7c  Scien7fic    Publishing  

Dialogue  /  Issue  /  Argument  Mapping  

A word of introduction… What does this tell you?

Critical scepticism to economic modelling

Critical scepticism to economic modelling

The conversation you don’t want to have with your learning analytics wonk

LAK13 Panel: Educational Data Scientists: A Scarce Breed http://people.kmi.open.ac.uk/sbs/2013/03/lak13-edu-data-scientists-scarce-breed

John Behrens (Pearson)

http://people.kmi.open.ac.uk/sbs/2013/03/lak13-edu-data-scientists-scarce-breed

Data ethics and societal readiness ny7.ms/1k5rY72  

Data ethics and societal readiness ny7.ms/1k5rY72  

In  what  senses  do    learning  analy1cs    “act”  in  educa1on?  

make  the  invisible  visible  

make  solids  permeable  

make  the  ephemeral  persistent  

Educa7onal  (indeed  all)  

Researchers  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

make  the  invisible  visible  

make  solids  permeable  

make  the  ephemeral  persistent  

Learning  Analy7cs  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

Educa/onal  researchers  have  always  done  this  —  what’s  new  is  that  these  tools  are  now  in  the  hands  of  those  who  actually  produce  the  ac/vity  

outcome  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

By  changing  the  system  dynamics  

researchers  /  educators  /  instruc7onal  designers  administrators  /  leaders  /  policymakers  

intent  

outcome  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

By  changing  the  system  dynamics  Faster  feedback  loops  could  enable  more  rapid  adapta7on:  of  agents’  behaviour,  and  of  learning  resources  and  designs  

researchers  /  educators  /  instruc7onal  designers  administrators  /  leaders  /  policymakers  

intent  

Delega7on  of    authority  to  define  goals,  analy7cs,    and  implica7ons  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

Distribution of power between educators, learners, leaders, community…?

?  

?  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

What data, concepts and relationships do the analytics designers seek to model?

Ontologically  

Bowker, G. C. and Star, L. S. (1999). Sorting Things Out: Classification and Its Consequences. MIT Press, Cambridge, MA, pp. 277, 278, 281

“Classification systems provide both a warrant and a tool for forgetting [...] what to forget and how to forget it [...] The argument comes down to asking not only what gets coded in but what gets coded out of a given scheme.”

Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, 2012, Vancouver, BC). ACM. Eprint: http://oro.open.ac.uk/32823

“A marker of the health of the learning analytics field will be the quality of debate around what the technology renders visible and leaves invisible.”

http://oro.open.ac.uk/32823

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

What thresholds, samples, relationships, patterns, etc. do the algorithms encode and seek? On what basis is a recommendation engine proposing interventions?

Algorithmically  

governingalgorithms.org   Learning  Analy7cs  

In  an  increasingly  algorithmic  world  […]  What,  then,  do  we  talk  about  when  we  talk  about  “governing  algorithms”?   

governingalgorithms.org  

A  technology  or  an  epistemology?  

Barocas,  S.,  Hood,  S.  and  Ziewitz,  M.  (2013).  Governing  Algorithms:  A  Provoca1on  Piece.  Social  Science  Research  Network  Paper  2245322.  DOI:    h.p://dx.doi.org/10.2139/ssrn.2245322    

Secrecy,  obscurity,  inscrutability  

Agency,  automa7on,  accountabili7es  

A  typology  of  algorithms  by  genre?  

The  inscrutability  of  algorithms  

Norma7vity,  bias,  values  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

What meaning-making does the representation and interaction design encourage?

Semio7cally  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

epistemology

pedagogy assessment

Knight, S., Buckingham Shum, S. and Littleton, K. (In Press, 2014). Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of Learning Analytics. Open Access Eprint: http://oro.open.ac.uk/39226

the ‘middle

space’ of learning analytics

in the context of computation...

What epistemological assumptions are shaping the assessment regime, and hence the pedagogy? What questions are analytics used to help answer?

http://oro.open.ac.uk/39226

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

Analytics reports at the organisational and national levels come with consequences at different scales — sometimes punitive, often impacting millions of people.

Poli7cally  

Figure  from  Doug  Clow:  h.p://www.slideshare.net/dougclow/the-‐learning-‐analy7cs-‐cycle-‐closing-‐the-‐loop-‐effec7vely  (slide  5)  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

All  of  the  above  are  encapsulated  in  any  learning  analy7cs  deployment  

28  

What  kinds  of  learners?  What  kinds  of  learning?  

What  data  could  be  generated  digitally  from  

the  use  context?    How  is  it  ‘cleaned’?  

Does  your  theory  predict  pa.erns  signifying  

learning?  

What  human  +/or  sooware  interven7ons  /recommenda7ons?  

How  to  render  the  analy7cs,  for  whom,  and  will  they  

understand  them?  

What  analy7cal  tools  could  be  used  to  find  such  

pa.erns?  

In  what  senses  do  learning  analy7cs  “act”  in  educa7on?  

Learning  Analy7cs  approaches  act  in  educa7on…  

Ontologically   Algorithmically   Semio7cally  

Systemically  

Poli7cally  

Authority?  

>>>  Learning  Analy7cs  cri7cal  template…  

Ontology …

Algorithms …

Semiotics …

Epistemology …

Pedagogy …

Assessment regime …

System Dynamics …

Authority …

Politics …

examples  

English  primary  school  data  RAISEonline  sta7s7cs  

for  a  school,  on  ‘core’  subjects  (maths,  literacy),  a.endance  and  demographics  

Visualising “attainment” and “progress”

Which analytics could reflect the progress that ‘Joe’ has made on so many other fronts other than his SATS?

34

student  modelling  by  an  adap1ve  tutor  

Intelligent tutoring for skills mastery

http://oli.cmu.edu

http://oli.cmu.edu

37

https://grockit.com/research

Adaptive platforms generate fine-grained analytics on curriculum mastery

https://grockit.com/research

analy1cs  for    learning  disposi1ons  

Qualities of effective learners Factor analysis of the literature plus expert interviews: identified seven dimensions of effective “learning power”, since validated empirically with learners at many levels. (Deakin Crick, Broadfoot and Claxton, 2004)

Qualities of effective learners

40

ELLI: Effective Lifelong Learning Inventory

Quantifying learning dispositions agency; identity; motivation; responsibility

Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, Vancouver). Eprint: http://oro.open.ac.uk/32823 http://learningemergence.net/2012/04/30/learning-powered-learning-analytics

A  wholis7c  visual,  intended  to  build  intrinsic  mo7va7on,  invi7ng  stretch,  providing  a  new  language,  provoking  conversa7on  that  7es  to  the  learner’s  iden7ty  

http://oro.open.ac.uk/32823http://learningemergence.net/2012/04/30/learning-powered-learning-analytics

Quantifying learning dispositions agency; identity; motivation; responsibility

http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsets

http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsets

http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsets

2020? personal data cloud generates my dispositional profile for reflection >>> help me take responsibility for my own learning

Shaofu Huang: Prototyping Learning Power Modelling in SocialLearn http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium Simon Knight: http://people.kmi.open.ac.uk/knight/2014/02/knowledge-in-search

Social network patterns, teamwork effectiveness

and initiation of relationships

Questioning, arguing and search behaviours

reveal intrinsic curiosity and epistemic commitments

Tagging/sharing/blogging/social patterns

reveal how you see connections between

ideas

Perseverance, grit, tenacity; overcoming

panic/stress when challenged — through

behavioural and somatic traces

http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsetshttp://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposiumhttp://people.kmi.open.ac.uk/knight/2014/02/knowledge-in-search

Open source approach to analytics: towards true transparency and emergence for complex systems?

http://www.solaresearch.org/mission/ola

http://www.solaresearch.org/mission/ola

conclusion    analy1cs  act  in  educa1on  —  on  mul1ple  dimensions  an  analy1cs  approach    perpetuates  an  educa1onal  worldview    —  so  let’s  ensure  this  is  

inten1onal  –  not  accidental...