Analytics(as if learning mattered)

Adam Cooper,In Focus: Learner analytics and big data symposium,University of London, December 10th 2013

Structure of the Talk

• My Perspective

• Threats

• Avoiding the threats

Image: cb

a Jaskirat Singh Baw

a

PERSPECTIVE

Perspective of this TalkWhere am I coming from?

UK

Higher Ed

InstitutionCetis is based in the Institute for Educational Cybernetics at the University of Bolton.

“Our mission is to develop a better understanding of how information and communications technologies affect the organisation of education from individual learning to the global system.”

Defining “Analytics”

Analytics is the process of developing

actionable insights

through

problem definition

and the application of statistical models and

analysis against existing and/or simulated future

data.

Learning vs Business Venn

Insights to support

educational aims and

objectives

Insights to support

operational and

strategic activity.

Learning vs Business Venn

Retention

Student Satisfaction

Employability

Achievement

Applications

Easiest Case forInvestment

Structure to Manage ComplexityThe SMT – “what makes a viable organisation?”

Subject/Discipline Member - “what makes a good chemist?” (knowledge, approach to problems, creativity, attitudes, values...)

Course Team – “what curriculum?” (topics, suitable LOs and assessment methods to match our typical student career etc)

Class Teacher – “where are my students collectively and individually?” (intellectually, emotionally, meta-cognitively...)

Student/Learner – time, place and method of studySelf-organising and self-regulating

The Intersection Doppelgänger

Retention

Student Satisfaction

Employability

Achievement

Applications

Values

Epistemological

Cause & Effect Models

Moral Imperatives

The “Accommodation”“The introduction of these techniques [Learning Analytics] cannot be

understood in isolation from the methods of educational management as

they have grown up over the past two centuries. These methods are

conditioned by the fact that educational managers are limited in their

capability to monitor and act upon the range of states which are

taken up by teachers and learners in their learning activities. ... Over

the years, an accommodation has developed between regulatory authorities,

management and teaching professionals: educational managers indicate the

goals which teachers and learners should work towards, provide a

framework for them to act within, and ensure that the results of their activity

meet some minimum standards. The rest is left up to the professional skills

of teachers and the ethical integrity of both teachers and learners.”Prof. Dai Griffiths, “Implications of Analytics for Teaching Practice in Higher Education”

TWO THREATS(+1 AND A BIT)

Threat #1: Interventionism

Image: via Wikimedia Commons p

Threat #2: BI Tradition

• Centralised projects• IT + senior leader• Management reports• KPIs

Image: Paul H

odge

Image: via W

ikimedia Com

mons p

Hhultgren

So, you want to optimise student success?

• Are the following well-specified?• “optimise student success”• “maximise the probability of success”

• What does success look like?• Really?• Says who?• Is a “good education” indicated by student success?

Does everyone understand?Attainment: an academic standard, typically

shown by test and examination results

Progress: how far a learner has moved between assessment events

Achievement: takes into account the standards of attainment reached by learners and the progress they have made to reach those standards

Enrichment: the extent to which a learner gains professional attitudes, meta-cognition, technical or artistic creativity, delight, intellectual flexibility, ...

+1: Technological Solutionism‘Recasting all complex social situations either as neatly defined problems with

definite, computable solutions or as transparent and self-evident processes that

can be easily optimized—if only the right algorithms are in place!—this quest is

likely to have unexpected consequences that could eventually cause more

damage than the problems they seek to address.

I call the ideology that legitimizes and sanctions such aspirations “solutionism.” I borrow this

unabashedly pejorative term from the world of architecture and urban planning, where it

has come to refer to an unhealthy preoccupation with sexy, monumental, and narrow-

minded solutions—the kind of stuff that wows audiences at TED Conferences—to problems

that are extremely complex, fluid, and contentious … solutionism presumes rather

than investigates the problems that it is trying to solve, reaching “for the answer

before the questions have been fully asked.” How problems are composed matters

every bit as much as how problems are resolved.’

Evgeny Morozov, “To Save Everything Click Here: The Folly of Technological Solutionism”, 2013

3M and Six-Sigma“McNerney axed 8,000 workers (about 11% of the workforce), intensified the

performance-review process, and tightened the purse strings at a company that had

become a profligate spender. He also imported GE's vaunted Six Sigma program—a

series of management techniques designed to decrease production defects and

increase efficiency. Thousands of staffers became trained as Six Sigma ‘black belts’.

...

Now his successors face a challenging question: whether the relentless emphasis on

efficiency had made 3M a less creative company. ... Those results are not

coincidental. Efficiency programs such as Six Sigma are designed to identify

problems in work processes—and then use rigorous measurement to reduce

variation and eliminate defects.

...

Indeed, the very factors that make Six Sigma effective in one context can

make it ineffective in another. Traditionally, it uses rigorous statistical

analysis to produce unambiguous data that help produce better quality,

lower costs, and more efficiency. That all sounds great when you know

what outcomes you'd like to control. But what about when there are few

facts to go on—or you don't even know the nature of the problem you're

trying to define?”Source : Bloomberg Business Week, June 2007.

The Bit

Data:

it may be big

but its not clever

The Bit: Net Scale Fascination (delusion?)

Data:

it may be big

but its not clever

• Hendrik’s photo

Source unknown – thanks to Hendrik Drachsler for passing it on.

Threat Rating=UBI*(I+D)UTS

Where:UBI = Uncritical BI tradition factor

I = Interventionist tendenciesD = Big data fascinationUTS = Uncritical technological solutionism

This is not serious...

Why am I Worried?

From

: “An

alyti

cs in

Hig

her E

duca

tion:

Ben

efits

, Bar

riers

, Pr

ogre

ss, a

nd R

ecom

men

datio

ns”

(ECA

R Re

port

)

NEUTRALISING THREATS

Data Is Useless Without the Skills to Analyze It

‘...employees need to become:

Ready and willing to experiment: Managers and business

analysts must be able to apply the principles of scientific

experimentation to their business. They must know how to construct

intelligent hypotheses. They also need to understand the principles

of experimental testing and design, including population selection

and sampling, in order to evaluate the validity of data analyses.

...

Adept at mathematical reasoning: How many of your managers

today are really “numerate” — competent in the interpretation and

use of numeric data?’Jeanne Harris, HBR Blog, Sept 2012

Is This a Real Problem?

From

: “An

alyti

cs in

Hig

her E

duca

tion:

Ben

efits

, Bar

riers

, Pr

ogre

ss, a

nd R

ecom

men

datio

ns”

(ECA

R Re

port

)

Make Some Progress By:

• Directly investing in specialists• E.g. OU, statisticians and data wranglers• ECAR Survey Report 2012: “invest in people over tools...

hiring and/or training... analysis”

• Operating a “shared service”• Higher Education Statistics Agency (HESA)• Universities and Colleges Admissions Service (UCAS)• Jisc

Data Is Useless Without the Skills to Analyze It

‘...employees need to become:

Ready and willing to experiment: Managers and business

analysts must be able to apply the principles of scientific

experimentation to their business. They must know how to construct

intelligent hypotheses. They also need to understand the principles

of experimental testing and design, including population selection

and sampling, in order to evaluate the validity of data analyses.

...

Adept at mathematical reasoning: How many of your managers

today are really “numerate” — competent in the interpretation and

use of numeric data?’Jeanne Harris, HBR Blog, Sept 2012

We Get Further with “Soft Capability”

• Appreciation of the limitations of analytics technologies

• Expertise to comprehend, critique, and converse(as opposed to originate or implement)

Visualisations are the Opiumof the People

(c)2008, Ryan Block/Engadget

Sparklines: Tufte vs ...

Significance and Sample Size“Flagged for action because 74% of students felt

they were well supported by their supervisor compared to 80% in similar institutions (difference >5%)”

“We need to talk to Mr. Benn; the drop-out rate doubled in his class this year.”

Correlation is not Causation“The predictive model indicates that students with

attributes {X, Y, Z} are only 50% likely to complete.”

What Does This Give Us?(aside from people less likely

to cut themselves on sharp tools)

Absorptive Capacity

“...the ability of a firm to recognize the value of new,

external information, assimilate it, and apply it to

commercial ends is critical to its innovative capabilities. We

label this capability a firm’s absorptive capacity and suggest that

it is largely a function of the firm’s level of prior related

knowledge. [...] We formulate a model of firm investment in

research and development (R&D), in which R&D contributes to

a firm’s absorptive capacity, and test predictions relating a

firm’s investment in R&D to the knowledge underlying technical

change within an industry.” Cohen and Levinthal, Admin. Science Quarterly 35(1), 1990

How to do LA as if learning mattered

1.Build soft capability.

2.Engage in participatory (collaborative) design.

3.Develop & prototype technology (& models)

organically.

4.Reflect on the effect LA innovation has on practice.

5.Design and develop with the educator IN the

system.

6.Optimise the environment, not only the outcome.

7.Be realistic about scale/quality of data and its

meaning.

Personal Learning Analytics?• Changed dyamic• Less “big brother”

Image: cbn Jesse Millan

ParticipatoryDesign

LearningAnalytics

Thank-you for your attention

These slides are at: http://is.gd/AaiLM

Reading and ReferencesJacqueline Bichsel, “Analytics in Higher Education: Benefits, Barriers, Progress, and

Recommendations”Available from http://www.educause.edu/ecar

Cohen and Levinthal (1990), “Absorptive Capacity: A New Perspective on Learning and Innovation” Admin. Science Quarterly 35(1)http://faculty.fuqua.duke.edu/%7Echarlesw/s591/Bocconi-Duke/Papers/C10/CohenLevinthalASQ.pdf

Adam Cooper, “What is Analytics? Definition and Essential Characteristics” http://publications.cetis.ac.uk/2012/521

Jeanne Harris, “Data is Useless Without the Skills to Analyze it”http://blogs.hbr.org/2012/09/data-is-useless-without-the-skills/

Bloomberg Business Week, “At 3M, A Struggle Between Efficiency And Creativity”http://www.businessweek.com/stories/2007-06-10/at-3m-a-struggle-between-efficiency-and-creativity

Dai Griffiths, “CETIS Analytics Series: The impact of analytics in Higher Education on academic practice”http://publications.cetis.ac.uk/2012/532

Evgeny Morozov, “To Save Everything Click Here: The Folly of Technological Solutionism”ISBN-10: 9781610391382

Phil Winne, “Improving Education” (lecture)http://www.youtube.com/watch?v=dBxMZoMTIU4

Licence

“Analytics (as if learning mattered)” by Adam Cooper, Cetis,

is licensed under the Creative Commons Attribution 3.0 Unported Licence

http://creativecommons.org/licenses/by/3.0/

Case studies and white papers:

Cetis Analytics Series, http://publications.cetis.ac.uk/c/analytics

Me:

http://blogs.cetis.ac.uk/adam

a.r.cooper@bolton.ac.uk

Cetis:

http://www.cetis.ac.uk