Institutional Reflection on Readiness: The Learning Analytics Readiness Instrument (237810993)

8/11/2019 Institutional Reflection on Readiness: The Learning Analytics Readiness Instrument (237810993)

http://slidepdf.com/reader/full/institutional-reflection-on-readiness-the-learning-analytics-readiness-instrument 1/80

Institutional Reflection on Readiness:The Learning Analytics Readiness Instrument

May 19, 2014

Kimberly Arnold, University of Wisconsin-Madison

Steven Lonn, University of Michigan

Matthew D. Pistilli, Purdue University



Poll

Who is attending this seminar with you?

A) I’m attending by myself

B) There are a small handful of us here

(< 5)C) There are many people here (6+)



Who are we?

Where are we from?Why are we here?

Introductions and Purpose



Introduction

What is Learning Analytics?

The Learning Analytics Readiness Instrument

Breaking Down the Factors

Concluding Thoughts



What is Learning Analytics:

Definitions



Definitions of Learning Analytics Using analytic techniques to help target instructional,

curricular, and support resources to support theachievement of specific learning goals (van Bareneveld,Arnold, & Campbell, 2012)

the process of developing actionable insights throughproblem definition and the application of statisticalmodels and analysis against existing and/or simulatedfuture data (Cooper, 2012)

The use of data, statistical analysis, and explanatoryand predictive models to gain insights and act oncomplex issues (Bichsel, 2012)



Challenge: How do you find the student at risk?

https://reader010.{domain}/reader010/html5/0613/5b2057b131b74/5b2057b683499.jpg



https://reader010.{domain}/reader010/html5/0613/5b2057b131b74/5b2057b79bc43.jpg

Challenge: How do you find the student at risk?



Analytics is about…

Actionable intelligence

Moving research to practice

Basis for design, pedagogy, self-awareness

Changing institutional culture

Understanding the limitations andrisks



http://i.imgur.com/nZArTnc.jpg



New Possibilities

Using data that exists on campus

Taking advantages of existing programs

Bringing a “complete picture” beyond academics

Focusing on the “Action” in “Actionable Intelligence”



The Concept of LA Readiness



What is “Readiness”

• What things do you think of when

you hear Institutional Readinessfor Learning Analytics?

• Discuss / Brainstorm

– What does the institution need?

• For whom? When?



Institutional Readiness

- Share Out -

• What did you / your team list?

• What is the highest / first priority?

– Why?

• What is already in place vs. needs to be

purchased / developed / trained?

• Are their larger / cultural issues to discuss

before logistical issues can be addressed?



Poll

How ready is your institution to

implement learning analytics?

A.We’re ready!

B.We need a little planning, but we are

basically ready.C.We’ve talked about learning analytics,

but made no plans.

D.We are not ready.



Readiness (ˈred-ē-nis)

noun1 willingness to do something.

2 the state of being fully prepared

for something.

3 immediacy, quickness, or promptness.

• Adopting learning analytics requiressignificant investment of:

– Time

– Resources – Money

http://www.flickr.com/photos/gadgetgirl70/2918038830/



Readiness for

Learning Analytics Will not happen

accidentally

Logistics can

be daunting

Institutional

reflection is

criticalhttp://ieatgrass.com/2011/05/ready‐set‐go/



The Need for Reflection

• Comprehensive understanding

• Multiple perspectives needed

– Cross-disciplinary

– Diverse expertise & skills• More realistic understanding

of resources

• Limit bias that inadvertentlyinterjected

“Thinking” designed by

Ashley Fiveash

from

The Noun Project



Assessing Readiness



Analytics “Maturity”Educause’s (2012) Center for Analysis and Research’s Analytics Maturity Index



Framework for Creation

Provide an institutional profile with

readiness indicators

Serve as a prescriptive diagnostic

Situated within L.A. literature

Formative instrument focusing on“3 Ps”

Parsimony

Practicality

Proactive

http://www.flickr.com/photos/tschiae/8417927326/



The Learning Analytics

Readiness Instrument(LARI)



Item Development Questions based on:

Literature

ECAR Maturity Index

Practitioners’ experience

Original Factors:

ABILITY DATAGOVERNANCE/

INFRASTRUCTURE

CULTURE PROCESS



Poll

What is the most important factor to

consider when preparing for a learning

analytics implementation?

A. Ability

B. DataC. Culture and process

D. Governance and infrastructure

E. Readiness perception



Participants• 33 participants across 9 institutions

(8 US, 1 Canadian)• R1s

• Already established foothold in LA



Exploratory Factor Analysis Began with 139 items

Varimax rotation with Kaiser normalization Exclusion criteria of |.30|

42 survey items eliminated

Second factor analysis 7 additional items eliminated



New Factors

ABILITY

DATA

GOVERNANCE /INFRASTRUCTURE

CULTURE

PROCESS



New Factors

ABILITY ABILITY

DATA


CULTURE

PROCESS



New Factors

ABILITY

DATA

ABILITY

DATA


CULTURE

PROCESS



New Factors

ABILITY

DATA


ABILITY

DATA


CULTURE

PROCESS



New Factors

ABILITY

DATA


CULTURE /

PROCESS

ABILITY

DATA


CULTURE

PROCESS



New Factors

ABILITY

DATA


CULTURE /

PROCESS

OVERALL

READINESS

PERCEPTIONS

ABILITY

DATA


CULTURE

PROCESS



Exploratory Factor Analysis

Final 90 survey items

Explained 55.7% of variation

Chronbach’s alpha of .946

Factor # of Items Median Skewness Kurtosis α

Ability 19 -.038 -.401 -.640 .940

Data 22 .228 -1.157 .686 .945Culture &

Process23 .146 -.420 .073 .898

Governance &

Infrastructure15 .164 -1.299 2.131 .896

OverallReadiness

Perceptions11 .006 -.734 1.421 .841



Results



-2

-1.5

-1

-0.5

0

0.5

1

1.5

Ability Data

Culture / Process Governance / Infrastructure

Readiness Perceptions

Institutional Differences(at least 2 respondents)

A B C E F G I



-2

-1.5

-1

-0.5

0

0.5

1

1.5

Ability Data




A B C E F G I



Limitations• Sample size

• Institution type

• Policy implication in international

settings



Breaking Down the Factors



Breaking Down the Factors: Data



Data is…

Foundational

Abundant

Diverse



Data is often…

Abundant (confusing)

Not archived

Dispersed

Siloed

Inaccessible



Data considerations…

“Big” or “small”

Validity and reliability of data sources

Storage, archival, and access

Ownership/stewardship/utilization



LARI Items

The extent to which my institution currently

collects admissions data (e.g., prospect

demographics, interests, applicationevaluation)

At my institution, the primary “owner” (the

person / group responsible for granting accessand stewarding the data) of the following data

sources is....

At my institution there are routine scripts /processes for refreshing / updating institutional

reports.



Reflection Questions…

On what topic(s)/issue(s) do you plan to

focus and why?

What data is important to gather? How will

you know if you have the “right” data? The

“best” data?

What are the challenges associated with

gathering, assembling, and maintaining

data?



Breaking Down the Factors: Ability

P ll



Poll

Which abilities and expertise are most

important for a successful learning analytics

initiative?

A. Technical architecture

B. Analytics proficiencyC. Strategic leadership

D. Instructional design

Abilit



Ability

Wide-ranging abilities and expertise will

need to accessible

Different abilities will need to be accessed

at different points

Abilities needed will be different dependingon goals

Abilit



Ability

Technical expertise (architecture, coding,

development)

Analytic proficiency

Strategic leadership

Student support skills Instructional and curriculum expertise

Researchers and evaluators

Abilit C id ti



Ability Considerations

Technical expertise (architecture, coding,

development)

Analytic proficiency

Strategic leadership

Student support skills Instructional and curriculum expertise

LARI Items



LARI Items

My institution has professionals with

mathematical/statistical experience in deriving

and validating statistically sound algorithms(e.g., simulations, cross validation).

My institution has professionals who have

specialized analytics training. My institution has professionals with technical

knowledge / expertise in documenting data

store / warehouse structure and datadictionaries.

Reflection Questions



Reflection Questions

How do you identify where expertise exists?

How will you manage “territorial disputes”

between departments/units that have staff

with needed expertise?

To what extent is your executive leadershipinvolved in moving this project to fruition?



The Learning Analytics Readiness Instrument (LARI)



How would you describe your institution’s approach to using data on a day‐

to‐day basis?

A) We ALWAYS use data to bolster actions we’re going to take

B) Data is available, and sometimes we access it to inform our decision

C) Data is available, but we usually don’t refer to it

D) Data is difficult to get, so we only ask for it when absolutely necessary

E) Data? What’s data? We don’t need data to make good decisions

Poll



• Shared vision of what analytics is

• Shared vision of student success

• Awareness and acceptance of data‐driven decision

making

• Goal

orientation

for

implementing/using

analytics• Stakeholder support of the effort

• Policies and practices amenable to examining/using

student data

• Communication between units about data/data use is

commonplace

Culture



• Stakeholder involvement in the effort

• Project

management

resources• Sustaining new initiatives

• Data and report use in practice

• Established procedures for collecting, organizing, storing, accessing, sharing, using, and securing data

Process



• My institution has the ability to store, manage, and analyze increasingly large volumes of

data.• Institutional reports routinely inform

institutional decisions.

• There is

a clear

vision

of

where

my

institution

can make changes to help students be more

successful academically.

• My institution

has

measurable

objectives

that

will indicate if success has been achieved.

LARI

Items



• What are the different policies that exist on

campus regarding access to and use of data?

• To

what

extent

is

data

currently

reported

on

your

campus?

What

makes

the

determination

of

what is reported, and the audience(s) to whom

those reports go? How are those reports used

by

the

recipients?

To

what

extent

is

the

information presented trustworthy?

• In

what

ways

has

your

institution

defined

student

success?

If

multiple

definitions

exist, are they competing?

Campus

Discussion



Institutional Report Out



Breaking down the factors:

Governance & InfrastructureThe Learning Readiness Instrument (LARI)

Poll



Poll

How would you characterize your current

governance and infrastructure around learning

analytics?

A) A benefit. Resources and professionals are

ready to assist in LA efforts.

B) A hindrance. No cohesive rules or resources.

Individuals can block progress.

C) A mixed bag. Some elements work great, someneed a lot of work.

67

Governance



Governance

• Broad, long-term involvement of LA efforts

• Sustainability

• Policy, goals ownership

(related to Culture & Process) – Policies that allow appropriate use

68

Infrastructure



Infrastructure

• Professional knowledge / expertise

– Building & maintaining large data warehouses

– Documenting data sources

– Query building, extraction – Manipulating & combining data sources

• Customer Service

– Diverse constituencies

– Documentation, tutorials

69

LARI Items - Governance



LARI Items Governance

• My institution has had conversations regarding

the sustainability of an analytics effort.

• People from multiple offices across my institution

are involved in the effort.

• My institution views learning analytics as a long

term investment, rather than a short term

expense.

• My institution has IT professionals who know

how to support analytics.

70

LARI Items - Infrastructure



• My institution has professionals with expertise in:

– Building / Maintaining complex data stores /warehouses

– Query building and data extraction

– Manipulating data from multiple sources and

platforms to conform to institutional

specifications

– documenting data store / warehouse structure

and data dictionaries

71

Worksheet



Discuss the following questions:

1. What institutional data is collected, curated, andmaintained for use in research projects?

2. How accessible are the subject matter experts on

institutional data?3. What institutional policies exist that govern the use of

varying kinds of data?

1. To what extent do these policies compete with one

another?

2. Who has the ability to determine which policies apply or

are superfluous to the project?72



Institutional Report Out

73



Future of the LARI

Next Steps



• Continue iterating the instrument

• Create tailored feedback and automate

– Overall profile score; domain scores

– Low, medium, high readiness

– Broad,

domain‐

specific

feedback – Prompts for exploration

• Build out scaffolds and activities

• Identifying international

partners

Beta Partners



• Lead individual

at

American

institution

• 10‐15 individuals in various roles representing

the diversity of job roles associated with

analytics

• Provide honest and constructive feedback

• Participate

in

follow‐

up

study

on

use

and

usefulness of report

• To become involved, email

[email protected]

Expectations Reality



p y

Plug and Play

Immediate results

Solve every problem –

ever!

Universal adoption

Everyone would love it!

Fits, starts, reboots

Mostly long termoutcomes

Solve some problems,

create some newproblems

Lackluster use

Not everyone loved it

Institutional Challenges



Data in many places, “owned” by manypeople/organizations

Different processes, procedures, and regulationsdepending on data owner

Sustainability

Everyone can see potential, but all want something slightlydifferent

“can’t you just…”

Faculty participation is essential

Staffing can be a challenge

In the end…



It is challenging

It can be difficultBUT

It is doable

This is the purpose of the LARI

Please consider joining the beta

[email protected]



Institutional Reflection on Readiness:

The Learning Analytics Readiness Instrument

Kimberly Arnold, University of Wisconsin-Madison

Steven Lonn, University of MichiganMatthew D. Pistilli, Purdue University

Institutional Reflection on Readiness: The Learning Analytics Readiness Instrument (237810993)

Documents

Transcript of Institutional Reflection on Readiness: The Learning Analytics Readiness Instrument (237810993)