Critical Interventions FrameworkDr Ryan Naylor, CSHE, University of Melbourne 05/02/2014

Developing a

Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

Project Background

• Commissioned project for DIICCSRTE, completed by Dr Ryan Naylor, Dr Chi Baik, Professor Richard James

• Are we on track in achieving national social inclusion goals?

1. Where are we?

2. How effective are our current initiatives? (What appears to work well? What doesn’t?)i. Is it possible to generate a typology of equity initiatives to allow consolidation of research evidence?

ii. Is there evidence in the literature or from HEPPP evaluations to support their efficacy?

Caution!

• Some necessary simplifications had to be made in creating the typology and fitting the literature to it

• “Intervention” is a contentious term with troubling associations

• New coalition government changes the policy context

What has had the biggest effect on equity?

• Uncapping/deregulation of volume of undergraduate places?

• National target for low SES participation (and associated Mission Based Compacts)?• HEPPP funding?• Wider societal trends in community beliefs about the value of undertaking higher education, entry requirements, eligibility for participation?• Efficacy of equity initiatives depends on underlying factors such as these• Many variables, highly inter-related

Where are we?A quick look at the numbers

Since 2007, there has been an explosion in domestic student numbers

2007: approx. 722,000 domestic students

2011: approx. 888,000 domestic students

= An increase of 23% over 4 years, oran annual growth rate of 5%

This level of growth is unprecedented in Australian HE

Gains in participation share have been made…

Participation Ratio (2011)

…But they have been relatively modest and not universal

Participation Ratio (2011)

It is difficult to improve equity during growth periods

Stud

ents

with

a d

isab

ility

Indi

geno

us s

tude

nts

Stud

ents

from

a N

on E

nglis

h Sp

eaki

ng

Back

grou

nd

Wom

en in

Non

-Tra

ditio

nal A

reas

Low

SES

(pos

tcod

e m

easu

re)

Low

SES

(CD

mea

sure

)

Regi

onal

Rem

ote

Tota

l

0

2

4

6

8

10

12

Tota

l

Average grow

th rate (2008-2011) (%)

But some groups whose share has historically been stable have

increased their participation share

Stud

ents

with

a d

isabi

lity

Indi

geno

us st

uden

ts

Stud

ents

from

a N

on E

nglis

h Sp

eaki

ng B

ackg

roun

d

Wom

en in

Non

-Tra

ditio

nal

Area

s

Low

SES

(pos

tcod

e m

easu

re)

Low

SES

(CD

mea

sure

)

Regi

onal

Rem

ote

Tota

l

0

2

4

6

8

10

12

Tota

lA

verage growth rate (2008-2011) (%

)

This change has not been uniform – low SES

Access rate (2011) (%)

Change since 2007 (%)

A 4.25 -0.01

B 6.92 2.38

C 7.10 -0.39

D 15.14 -0.14

E 20.49 -1.38

F 32.82 0.66

G 33.83 2.88

Sector average 16.87 0.75

• No correlation between access rate (2011, 2007) and change• Complex factors – different geographical contexts, access policies,

etc• Traditional strong performers didn’t do better

Most groups appear no less likely to succeed

Tota

l

Most groups (bar 2) are no less likely to succeed

Tota

l

May not have seen full effects yet – not all students from DDS cohort have moved through the system yet (early days yet!)

For most groups, the key problem continues to be access

Equity group Participation Retention Success

Students with a disability 0.48 0.96 0.93

Indigenous students 0.55 0.85 0.81

Students from Non English Speaking Background

0.82 1.04 0.97

Women in Non-Traditional Areas 0.35 1.01 0.99

Low SES (postcode measure) 0.67 0.98 0.97

Low SES (CD measure) 0.62 0.97 0.96

Regional 0.64 0.98 0.99

Remote 0.39 0.91 0.94

Participation and access are (should be?) key focal points in student equity and social inclusion

Equity group Participation Retention Success

Students with a disability 0.48 0.96 0.93

Indigenous students 0.55 0.85 0.81

Students from Non English Speaking Background

0.82 1.04 0.97

Women in Non-Traditional Areas 0.35 1.01 0.99

Low SES (postcode measure) 0.67 0.98 0.97

Low SES (CD measure) 0.62 0.97 0.96

Regional 0.64 0.98 0.99

Remote 0.39 0.91 0.94

This is not to argue they don’t need support once enrolled• less academically well prepared students from any background• Indigenous students

For most groups, the key problem continues to be access

How effective are our current initiatives:The Critical Interventions Framework

The Critical Interventions Framework

• Is it possible to create a typology of equity initiatives?• Can we find sufficient evidence in the literature to point to the efficacy of particular types of initiatives?• Can we identify the initiatives that are most effective?• We don’t know for sure, so we had to make some guesses

The Critical Interventions Framework

• Is it possible to create a typology of equity initiatives?• Yes

• Can we find sufficient evidence in the literature to point to the efficacy of particular types of initiatives?• Often no

• Can we identify the initiatives that are most effective?• No. There simply isn’t enough evidence.

The equity initiative terrain across a notional student lifecycle

The Critical Interventions Framework (one small section)

The Critical Interventions Framework (one small section)

The Critical Interventions Framework (one small section)

• We all have intuitions about what works • Attempted to base framework on evaluative science, not intuition• Unfortunately, the science is largely not there, so some estimation and judgement was involved

The Critical Interventions Framework (one small section)

All things considered, what did we rate highly?

High

1B. Later-year outreach (Years 10-12)

2B. Bridging/foundation programs

2D. Scholarships

4C. Student services provision

5A. Monitoring student completion rates

Very High

1D. School curriculum enhancement/support

2A. Pathway/articulation programs

2C. Alternate selection criteria and tools

3A. First year orientation/transition support

• Not intended to narrow or homogenise people’s efforts

Red - rated very high; green – rated high

Do we have good evidence?

High Quality of Evidence

1B. Later-year outreach (Years 10-12) Limited

2B. Bridging/foundation programs Some (from US)

2D. Scholarships Strong (needs-based, not merit)

4C. Student services provision Varies depending on service

5A. Monitoring student completion rates N/A

Very High

1D. School curriculum enhancement/support Some (from US); strong for need

2A. Pathway/articulation programs Mixed

2C. Alternate selection criteria and tools Strong

3A. First year orientation/transition support Strong (from US)

Red – strong evidence

Green – some evidence

Blue – may be impossible to get evidence

How are we spending our HEPPP funding?

High Proportion of HEPPP funding

1B. Later-year outreach (Years 10-12) 14

2B. Bridging/foundation programs 7

2D. Scholarships 14

4C. Student services provision 15

5A. Monitoring student completion rates N/A

Very High

1D. School curriculum enhancement/support 1

2A. Pathway/articulation programs 3

2C. Alternate selection criteria and tools 2

3A. First year orientation/transition support 10Total: 66%

• This is purely descriptive, not normative!• Not all initiatives require the same amount of funding

Where to next?

• Written for national policy purposes – not meant to function at the institutional level • We need better evidence of program efficacy (and this is where institutions come in)

o Detailed, rigorous and published evaluationso To enable a sector-wide conversation about

equity initiatives• How can the CIF be used in an institutional context? Is this typology helpful?

Where to next?

• Funding sought to research: o Process and attitudinal factors affecting program

evaluationo Resources for embedding evaluation into core

business• First year experience survey• Collaborations with NCSEHE staff

For more information:

cshe.unimelb.edu.au

rnaylor@unimelb.edu.au

ncsehe.edu.au

ncsehe@curtin.edu.au