Using insights from behavioral science to break barriers to post-secundary education

Saugato Datta Managing Director, ideas42

Behaviorally Optimizing Higher Education: Improving Student Success through Behavioral Science

ESF Flanders Conference on Social Innovation Brussels, Belgium. October 25-26, 2015

Follow us on Twitter

@ideas42

agenda for session

2

esf flanders conference on social innovation

Day 1: Introduction to Behavioral Science & ideas42 • Why behavioral insights in Post-Secondary Education? • Case Study 1: Increasing On-Time Financial Aid Applications at Arizona

State Key Psychology: Social Norms • Case Study 2: Increasing Uptake of Tutoring Services at West Kentucky

Community and Technical College Key Psychologies: Time Inconsistency and Procrastination

agenda for session

3

esf flanders conference on social innovation

Day 2 • Recap of Cases • How did we get here? Overview of ideas42 RFP Process in Post-

Secondary Education • Discussion: What do we need for innovation?

first, a small quiz

© 2015 ideas42 5

question 1

A cup and a saucer cost $1.10 in total. The cup costs a dollar more than the saucer. How much does the saucer cost?

cents

© 2015 ideas42 6

question 2

If it takes 5 machines 5 minutes to make 5 widgets, how long will it take 100 machines to make 100 widgets?

minutes

© 2015 ideas42 7

question 3

In a pond, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 24 days for the patch to cover the entire pond, how long would it take for the patch to cover half the pond?

days

© 2015 ideas42 8

quiz answers

• A cup and a saucer cost $1.10 in total. The cup costs a dollar more than the saucer. How much does the saucer cost?

• If it takes 5 machines 5 minutes to make 5 widgets, how long will it take 100 machines to make 100 widgets?

• In a pond, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 24 days for the patch to cover the entire pond, how long would it take for the patch to cover half the pond?

$0.05 $0.10

5 min 100 min

23 days 12 days

incorrect (but illuminating) correct

© 2015 ideas42 10

do these gentlemen want to get fit?

© 2015 ideas42 11

what do we do?

Use the theories of behavioral science to design solutions to some of the world’s most

persistent social problems.

© 2015 ideas42 12

With many partners in many different domains…

© 2015 ideas42 13

In many countries across the world…

© 2015 ideas42 14

ideas42 projects around the world

© 2015 ideas42 15

Including internal government ‘nudge units’

Operational/Past: • Chicago Nudge Unit • 4 staff seconded to White House Social and

Behavioral Sciences Team • Western Cape, South Africa

In development: • Advisory role in World Bank’s new Global

Insights Unit (GINI) • NYC and Boston nudge units

© 2015 ideas42 16

why work in post-secondary education?

© 2015 ideas42 17

Just 59% of full-time, four-year college students graduate within six years

Source: https://nces.ed.gov/fastfacts/display.asp?id=40

© 2015 ideas42 18

Just 29% of students in full-time community college students graduate within 1.5x the standard time period

Source: http://nces.ed.gov/programs/coe/indicator_cva.asp

© 2015 ideas42 19

College is hard. (in ways we overlook)

to understand why, a small (fun) test

why do behavioral work in post-secondary education?

Student retention and performance depend on behavioral phenomena: self-control, attention,

cognitive bandwidth, salience, etc….

self control is hard

© 2015 ideas42 31

what we do what we intend ≠

Typically, when a program seeks to change this

We design it to change this

We actually need to

change this

limited self-control, inattention, status quo bias, etc. mean that…

© 2015 ideas42 32

which may explain why only 29% of full-time community college students graduate within 1.5x the standard time period


© 2015 ideas42 33

work in post-secondary education

Using the theories of behavioral science to improve student outcomes in colleges and

universities across the US.

© 2015 ideas42 34

EXAMPLE – FAFSA FORMS

Traditional perspective: - They don’t want to go to college - They can’t afford it - They don’t understand its value, etc.

Behavioral perspective:

- FAFSA forms are too hard - They make families feel too “unsophisticated” to

go to college

Problem: Students are not applying for financial aid

©2013 ideas42 34

© 2015 ideas42 35 © 2015 ideas42 35

Increasing FAFSA Applications with Behavioral Design Spring Semester 2015

© 2015 ideas42 36

1. In 2011-2013 http://money.cnn.com/2014/03/07/pf/college/fafsa-financial-aid/

Problem: only 18% of students file the FAFSA by ASU’s priority deadline, and many eligible students never file

• Priority filers are guaranteed maximum aid package

• Nationally, priority filers are offered 2x as much aid as those who apply later

• Nationally, at least 2 million students did not receive grants they qualify for because they did not file the FAFSA1

© 2015 ideas42 37

Diagnosed six primary barriers to file from survey, literature review, and conversations with stakeholders

Students do not understand what information they need to file the FAFSA

Students do not adequately plan to collect the information they’ll need (once they know) Parent information is difficult to obtain and may require significant effort by both student and parent

1

2

3

Priority deadline not salient at the right time

Inaccurate mental model of who receives financial aid

Misperceived social norms of how many students submit the FAFSA

4

5

6

© 2015 ideas42 38

befo

re p

riorit

y de

adlin

e Intervention 1: students Intervention 2: parents

• Population: 63,000 continuing students • Treatment group: 8 behavioral emails sent to students

- 3 sub-treatments compared BE email, short BE email, and peer BE email

• Control group: 1 standard ASU email sent to students*

• Population: 22,000 continuing students with parent emails on file

• Treatment group: 2 behavioral emails sent to parents • Control group: no communications*

RCT evaluated behavioral communications to students and parents before priority deadline

*Note this is standard protocol for ASU

© 2015 ideas42 39

Intervention 1: design targeted identified barriers to action for students

© 2015 ideas42 40



Students do not adequately plan to collect the information they’ll need (once they know)

Parent information is difficult to obtain and may require significant effort on part of both student and parent

1

2

3 3

2

2

1

© 2015 ideas42 41


4


4


5


6

4

5

6

© 2015 ideas42 42

Intervention 2: design targeted identified barriers to action for parents

4

1

3


Students do not adequately plan to collect the information they’ll need (once they know)

Parent information is difficult to obtain and may require significant effort on part of both student and parent

1

2

3

© 2015 ideas42 43

Intervention 2: design targeted identified barriers to action for parents

4

4

6

5




4

5

6

© 2015 ideas42 44

**indicates significance at the 95% level

Interventions 1 and 2 increased priority FAFSA filers by as much as 72%

29%

40% 44%

50%

0%10%20%30%40%50%60%

Control Student emails only(averaged)

Parent emails only Both student and parentemails

Number of Priority FAFSA Filers by Treatment Condition (Interventions 1 & 2)

** ** ** +38% +52%

+72%

© 2015 ideas42 45

**Indicates significance at the 95% level

29%

40% 44%

50%

0%10%20%30%40%50%60%

Control Student emails only(averaged)

Parent emails only Both student and parentemails

Number of FAFSA Filers by Treatment Condition (at End of Study Period )

** ** **

Interventions increased priority FAFSA filers by up to 72%

+72%

*No difference in Intervention 1 between behavioral email conditions; all increased relative to control

© 2015 ideas42 46

Insights for future projects

• Email intervention was effective, nearly doubling rate of filing by priority deadline and increasing applications overall

• Parents are a valuable communication channel; merit careful consideration of

when to involve them

• Despite aggressive campaign, 66% of students said they received “the right amount” of emails; only 7% reported getting “too many”

• Project is scalable and cost-effective (vs $90 H&R block experiment)

© 2015 ideas42 48

IMPLICIT NORMS

©2013 ideas42 48

© 2015 ideas42 49

NEGATIVE SOCIAL NORMS

©2013 ideas42 49

© 2015 ideas42 50

PERCEPTIONS OF SOCIAL NORMS

©2013 ideas42 50

© 2015 ideas42 51

FEATURES OF SOCIAL NORMS

Peer groups reinforce norms

Visibility of behavior is key

Framing matters

©2013 ideas42 51

© 2015 ideas42 52

Framing matters

FRAME THE BEHAVIOR

©2013 ideas42 52

© 2015 ideas42 53

FRAME THE BEHAVIOR

Framing matters

©2013 ideas42 53

© 2015 ideas42 54


FOCUS ON THE RIGHT BEHAVIOR

©2013 ideas42 54

© 2015 ideas42 55


FOCUS ON THE RIGHT BEHAVIOR

©2013 ideas42 55

© 2015 ideas42 56

DEFINING THE PEER GROUP

Peer groups reinforce norms

©2013 ideas42 56

© 2015 ideas42 57

DEFINING THE PEER GROUP

“A lot of people I know don’t go to college”

“66% of high school seniors go to college”

©2013 ideas42 57

© 2015 ideas42 58

GROUP ACTIVITY #1

• Think of a time when you started something new and at first felt like you didn’t fit in (immediately felt that you fit in).

• Take your favorite of the above examples and dig a little deeper into that situation. What were the details (location, mood, surroundings, actions of others) that prevented you from feeling like you fit in (helped you feel you belonged)?

• Discuss with your group. • Can you think of a situation faced by the population your

organization serves where a similar issue might exist?

© 2015 ideas42 59 © 2015 ideas42 59

Increasing Uptake of On-campus Tutoring at West Kentucky Community and Technical College Dana Guichon, Andrew White, Katy Davis, and Piyush Tantia 2014-2015

© 2015 ideas42 60

Problem: students do not use academic support services (tutoring) provided by the college

• 35% of students fail at least 1 class per semester • 20% of students withdraw from at least 1 class per

semester

High rates or students failing or withdrawing from classes

Low utilization of academic support services at the Tutoring Center

• 3-4% of enrolled students use tutoring per semester

© 2015 ideas42 61

Availability of tutoring was not salient to students.

Feedback on coursework provided too late, thus students do not seek academic help at the optimal time.

Students hold misperceptions around tutoring and/or the Tutoring Center.

Steps associated with scheduling a tutoring appointment create hassles.

Faculty play a large role in guiding students toward tutoring, yet few students receive actionable encouragement from professors.

Diagnosis focused on five factors limiting tutoring uptake

© 2015 ideas42 62

Randomized Trial 1 – Fall 2014

4,455 degree-seeking

students

treatment 4 ideas42

emails

industry standard

1 Starfish template email

control No emails

randomly sorted

© 2015 ideas42 63

Behaviorally designed treatment emails targeted identified diagnoses

Reduce hassles associated with scheduling a tutoring appointment by providing clear and actionable information

1

2

Correct misperceptions around tutoring

3 Set a deadline so that students seek tutoring at the optimal time

© 2015 ideas42 64

Intervention led more students to start going to tutoring and doubled the number of tutoring sessions attended

*Significant to the 90% level; **Significant to the 95% level Results look at students that started to go to tutoring on or after 10/14, which was the date the intervention was launched

Proportion of students that started attending tutoring

Number of tutoring sessions attended

1,14%

1,68% 1,82%

0,00%

0,40%

0,80%

1,20%

1,60%

2,00%

33

56

80

0

20

40

60

80

100 **

© 2015 ideas42 65

Randomized Trial 2 – Spring 2015

261 faculty teaching that semester

treatment 8 ideas42 emails

control no emails

randomly sorted

© 2015 ideas42 66


Encourage faculty to send referrals at the optimal time in the semester.

1

2

Peer testimonial around the benefit of tutoring for students

3 Reduce hassles associated with making a referral for tutoring

© 2015 ideas42 67

Intervention tripled the number of faculty sending referrals and doubled the number of referrals sent

*Significant to the 90% level; **Significant to the 95% level

Proportion of faculty that sent referrals

Number of referrals sent

**

6,45%

18,98%

0,00%

4,00%

8,00%

12,00%

16,00%

20,00%

24,00%

Control Treatment

49

106

0

20

40

60

80

100

120

Control Treatment

© 2015 ideas42 68

Source: Faculty responses to emails sent

Faculty appreciated the email campaign

Thank you! I look forward to providing my students with

these resources. As a former WKCTC student I benefitted

tremendously from The Tutoring Center.

Dr. Baker, I appreciate this

idea; a reminder can sometimes do the trick and I will

be more cognizant of tutoring for my

students.

© 2015 ideas42 69

Randomized Trial 3 – Spring 2015

3,121 degree-seeking

students on financial aid


control no emails

randomly sorted

© 2015 ideas42 70


Reduce hassles associated with scheduling a tutoring appointment.

1

2

Peer testimonial from a student sharing a tutoring success story and correcting misperceptions around what to expect at a tutoring session.

© 2015 ideas42 71

4,99%

6,71%

0,00%

2,00%

4,00%

6,00%

8,00%

10,00%

12,00%

Control Treatment

Treatment students were 34% more likely to attend tutoring and attended 53% more tutoring sessions

*Significant to the 90% level; **Significant to the 95% level Sample included all students taking at least one class for which tutoring was offered and excluded three students who had each attended over 20 tutoring sessions during the semester

Proportion of students that attended tutoring

Number of tutoring sessions attended

* 171

263

0

50

100

150

200

250

300

Control Treatment

*

© 2015 ideas42 72

Source: Student responses to the emails sent

Students appreciated the email campaign

Just words of encouragement throughout

the semester would be awesome. I work full time

and have a family. Sometimes I get frustrated

and just need words of encouragement.

…your encouragement was extremely

valuable in motivating me to

take action

© 2015 ideas42 73

Suppose I offer you a choice…

$100 today

$105 next month Or

© 2015 ideas42 74

Which would you choose?

1. $100 today

2. $105 next month

1 2

50% 50%

© 2015 ideas42 75

Now suppose I offer you the following choice…

$100 In 6 months

$105 In 7 months Or

© 2015 ideas42 76

Which would you choose?

1. $100 in 6 months

2. $105 in 7 months

1 2

50% 50%

© 2015 ideas42 77 77

CONTEXTUAL TRIGGERS Present-biased preferences often play out when something has immediate costs (often small, but feel big and immediate) and benefits in the future (may be large, but feel small or distant) Benefits

Costs

© 2015 ideas42 78

PROCRASTINATION

Putting off doing something at the point where you previously predicted you would do it.

Definition

© 2015 ideas42 79 79

PROCRASTINATION

© 2015 ideas42 80

PROCRASTINATION: INSIGHT #1

80

It’s about Deferral • You postpone something you’ve previously

decided to do, and still in principle want to do

Change Mind

© 2015 ideas42 81

Today

. . . . . .

Reality

PROCRASTINATION

Reality . . . . . Reality

. . . .

© 2015 ideas42 82

Open-ended processes with

no fixed deadlines

WHEN DO WE PROCRASTINATE?

© 2015 ideas42 83


no fixed deadlines

Processes with too many

component steps


© 2015 ideas42 84


no fixed deadlines

Processes with too many

component steps

Actions whose benefits lie far

ahead but costs are borne now


© 2015 ideas42 85

HASSLE FACTORS

Small roadblocks that need to be performed before you complete an action

Definition

© 2015 ideas42 86

HASSLE FACTORS

86

HASSLE FACTORS: INSIGHT #1 Small hassles can have a big impact because they play into our other problem psychologies. For example, if we never procrastinated, some hassles would be less powerful.

© 2015 ideas42 87

HASSLE FACTORS: INSIGHT #2 Effect is disproportionate to the size of the hassle. In other words, a small hassle can have an enormous impact.

87

Devoto, F. Duflo, E., Dupas, P., Pariente, W., Pons, V. (2011) “Happiness on Tap: Piped Water Adoption in Urban Morocco.” American Economic Journal: Public Policy, September 2011

Experiment: Households were given help with the administrative steps needed to get a piped water connection.

0

0,2

0,4

0,6

0,8

1

1,2

% signing up for piped water

control treatment

Results of Piped Water Adoption in Morocco

10%

70%

© 2015 ideas42 88

HASSLE FACTOR TRIGGERS

Processes with many

(sub)steps to complete

Processes with too many kinds of steps to be

taken

© 2015 ideas42 89

GROUP ACTIVITY #1

Part 1 • Think of times where you had a big deadline that you

failed to meet (were able to meet) • Take your favorite of the above examples and dig a little

deeper into that situation. What were the details (timing, location, mood, presence of others) that prevented you from meeting the goal (helped you reach the goal)?

• Discuss with your group. Part 2 • Discuss with your group a case (ideally among the

population your organization serves) where people fail to meet their goals.

• What has your institution done to help? • What could you do to help in the future?

© 2015 ideas42 90

+

© 2015 ideas42 91

rfp: a request for problems, not proposals


647 people across 600 organizations

72 attendees for RFP webinar

70 organizations interested in applying

© 2015 ideas42 92

RFP APPLICATIONS RECEIVED

56 Applications Received

6

5

7

17

2

2

3

7

1 6 Community Based4-Year InstitutionNational NetworksNon-ProfitsResearch GroupsHigh SchoolsEd TechCommunity CollegesGovernmentMisc.

15 Financial Aid

10 Summer Melt

9 Utilization of Resources

9 Persistence

3 College Savings

10 Other

16 17

23

Pre

para

tion

Tran

sitio

n

Per

sist

ence

&

Com

plet

ion

© 2015 ideas42 93

APPLICATION RATING PROCESS

7 dimensions:

Overall Feel Client Pool

Organizational Capacity

Leadership Buy-In

Scalability Intervention Promise

Appropriate Project Scope

© 2015 ideas42 94

NARROWING THE LIST

Eliminated 12 based on pool size and implementation promise

Arrived at final list through rankings and group discussion

Final pool included both strong potential partners and thought leaders in the space

1

2

3

© 2015 ideas42 96

Pay attention to the individual’s inattention You might think that your customer is making an active choice not to use your product. Are you sure s/he considered it?

Inattention

© 2015 ideas42 97

general lesson: define, diagnose, design … then test

DEFINE DIAGNOSE DESIGN TEST

REDEFINE PROBLEM

FIND ANOTHER BOTTLENECK

DISENTANGLE PRESUMPTIONS

INTERVENTION CONCEPT

CONTEXT RECONNAISSANCE

BEHAVIORAL MAP

HYPOTHESIZED BOTTLENECKS

POLISH INTERVENTION

DETERMINE FEASABILITY

INITIAL EXPERIMENT

DESIGN

ACTIONABLE BOTTLENECKS

SCALABLE INTERVENTION

DEFINED PROBLEM

© 2015 ideas42 98

Today’s focus is tomorrow’s neglect The pressing needs of today are far more important than what might arise three months from now.

Present Bias

TODAY

© 2015 ideas42 99

Small hassles can have large effects Small hassles can trigger the “not now” response, and lead to procrastination and inaction.

Hassle Factors

© 2015 ideas42 100

How can we stimulate innovation?

• What sorts of institutions should be targeted?

• How can we insert the necessary technical expertise

• How do we ensure rigor in measurement?

• How can we make innovation ongoing and iterative?

Saugato Datta Managing Director, ideas42

Behaviorally Optimizing College: Improving Student Success through Behavioral Science

October 25-26, 2015

Follow us on Twitter

@ideas42

© 2015 ideas42 102 © 2015 ideas42 102

Increasing Applications to Work-Study Jobs at Arizona State University Andrew White, Nicki Cohen, Alissa Fishbane, and Piyush Tantia Spring Semester 2015

© 2015 ideas42 103

Problem: few eligible students apply for work-study jobs in ASU’s new SEED jobs program

20%

80%

Low Hiring Rate for SEED jobs

Position filled Position open

11%

89%

Low Application Rate for SEED jobs

Applied Did not apply

Fall 2014 data

© 2015 ideas42 104

Incorrect mental models, non-salient deadlines, and hassle factors prevent students from applying

Students did not have the correct mental model about work- study jobs Application deadline not salient Hassle factors in the application process prevented action

© 2015 ideas42 105

Randomized controlled trial used one treatment arm

2,335 eligible freshmen


control 12 standard emails

randomly sorted

© 2015 ideas42 106

Shape the right mental model of SEED jobs — emphasize financial and academic benefits in clear language

Make deadline salient and force a moment of choice

Reduce hassle factors with plan-making activity

1

12 emails targeted identified diagnoses

2

3

© 2015 ideas42 107

More unique applicants and applications Trend towards more hires

**significant at the 95% level 1. Not a significant result. A large portion of applications were never reviewed due to organizational constraints

109 140

-

50

100

150

control treatment

Number of Unique Applicants

+30%

304

475

-

100

200

300

400

500

control treatment

Number of Applications

+60% ** **

Number of Hires1

55 hires

in treatment

50 hires

in control <

© 2015 ideas42 109 10

DEFINE (THE PROBLEM): Ideas42 worked with Sinclair on the problem of late registration

The Problem Too many students register too late in the process. In 2014, a third of continuing, non-audit students registered for classes during the last month of the registration window and over a thousand students registered in the last week. Why It Matters • Sinclair has evidence that late registration correlates with decreased academic

success and retention. • Students who don’t register for classes early get locked out of classes and

sections that they need, which impedes on-time completion and retention. • When students register late it is difficult to project course demand and offer the

courses students will need and want.

© 2015 ideas42 110 11

DIAGNOSE: WE IDENTIFIED THE KEY BEHAVIORAL BOTTLENECKS IMPEDING EARLY REGISTRATION

#1: Students are anchored to the perceived deadline as the day to act. This could be the registration deadline or the start of the semester.

#2: Registration isn’t salient for most students, particularly those that are not on campus everyday.

#3: There is little perceived value in registering early. Students don’t realize the consequences for delaying registration.

#4: Students appear to have various mental models (e.g., “don’t register until you’re sure”) which lead them to do something other than register early.

#5: The registration process contains a number of complex choices and hassle factors which lead to procrastination.

© 2015 ideas42 111

DESIGN: WE CREATED TWO INTERVENTIONS TO ADDRESS THE BOTTLENECKS

Advising Intervention

111

Messaging Intervention

» Students were assigned opt-out “appointments” with academic advisors via email and text message. In the appointments, advisors encouraged them to update their My Academic Plan (MAP) and register for Fall courses.

» Daytime appointments took place during regular advising walk-in hours. Students with an appointment were able to skip the normal line.

» Evening appointments were conducted over the phone. Advisors called students directly at the prescribed time.

» ideas42 sent students in the treatment group a series of emails and text messages

» Students were sent 1-2 messages a week over the course of 12 weeks, 19 messages in total.

» Messages were sent in email-text pairs.

© 2015 ideas42 112

Diagnosis Design: THE INTERVENTIONs WERE TARGETED AT THE DIAGNOSED BEHAVIORAL BOTTLENECKS

112

Advising Intervention


#1: Anchoring to

Deadline

#2: Salience of Registration

#3: Little

Perceived Value

#4: Different

Mental Models

#5: Hassle Factors

© 2015 ideas42 113

TEST: WE evaluated THE INTERVENTIONS USING A randomized control trial

113

Control Treatment

Con

trol

6,355 students did not receive

either intervention

6,390 students received the Messaging intervention

alone

Trea

tmen

t 2,600 students received the

Advising intervention

alone

2,556 students received both interventions


Adv

isin

g In

terv

entio

n

»ideas42 ran a 2x2 intervention design to test the effects of the interventions separately and combined.

© 2015 ideas42 114

TABLE OF CONTENTS

I. Project Recap: Define, Diagnose, Design, Test II. Key Results III. Subgroup Analysis IV. Discussion of Results

114

© 2015 ideas42 115

KEY OUTCOME VARIABLE: EARLY REGISTRATION DEFINED AS ON OR BEFORE JULY 17

APRIL

S M T W Th F S

1 2 3 4

5 6 7 8 9 10 11

12 13 14 15 16 17 18

19 20 21 22 23 24 25

26 27 28 29 30

MAY

S M T W Th F S

1 2

3 4 5 6 7 8 9

10 11 12 13 14 15 16

17 18 19 20 21 22 23

24 25 26 27 28 29 30

31

JUNE

S M T W Th F S

1 2 3 4 5 6

7 8 9 10 11 12 13

14 15 16 17 18 19 20

21 22 23 24 25 26 27

28 29 30

JULY

S M T W Th F S

1 2 3 4

5 6 7 8 9 10 11

12 13 14 15 16 17 18

19 20 21 22 23 24 25

26 27 28 29 30 31

AUGUST

S M T W Th F S

1

2 3 4 5 6 7 8

9 10 11 12 13 14 15

16 17 18 19 20 21 22

23 24 25 26 27 28 29

30 31

115

© 2015 ideas42 116

33,1% 35,4%

0%

5%

10%

15%

20%

25%

30%

35%

40%

No Messaging(Control)

Messaging

% S

tude

nts w

ho re

gist

ered

ear

ly **

MESSAGING INTERVENTION increased early reg By 2.3 percentage points, or 6.9 percent.

116 ** = significant at 95% level All students, collapsing across advising treatment

© 2015 ideas42 117

STUDENTS WHO OPENED AT LEAST ONE MESSAGING intervention EMAIL REGISTERED AT EVEN HIGHER RATES

117

26,8%

21,0%

42,7%

0%5%

10%15%20%25%30%35%40%45%

No Messaging (Control) No Opened Messages Opened at Least OneMessage

% S

tude

nts g

oing

to

advi

sing

this

sum

mer

• Positive effect of opening email on registration, but there may be selection effects

© 2015 ideas42 118

MESSAGING EMAILS HAD GENERALLY HIGH OPEN RATES THAT DECLINED OVER TIME

118

• Two-thirds of all students who received a messaging email opened at least one of them.

43,0%

22,6%

12,9%

0%5%

10%15%20%25%30%35%40%45%50%

Weeks 1-4 Weeks 5-8 Weeks 9-12

Emai

l ope

n ra

te (%

)

© 2015 ideas42 119

29,5% 31,4%

0%

5%

10%

15%

20%

25%

30%

35%

40%

No Advising(Control)

Advising

Stud

ents

who

regi

ster

ed e

arly

**

ADVISING INTERVENTION increased early reg by 1.9 percentage points, or 6.4 percent

119 ** = significant at 95% level Excluding students who registered before advising treatment began, and groups Sinclair specified

© 2015 ideas42 120

The advising intervention led MORE students to ENGAGE WITH advising

120

38%

47%

0%

10%

20%

30%

40%

50%

No Advising(Control)

Advising

% S

tude

nts e

ngag

ing

with

Ad

visi

ng fr

om 5

/4 to

7/3

1

• The intervention made students more likely to engage with Advising between May 4 and July 31 (not necessarily at the scheduled time).

• The range of what constitutes “engagement” requires further discussion.

© 2015 ideas42 121

Messaging & Advising EACH increased early registration BUT NO ADDED BENEFIT OF BOTH

121

***

• The Messaging and Advising interventions both increased early registration, but there was no statistically significant additional benefit of a student receiving both.

26,8%

32,1% 30,4%

32,4%

0%

5%

10%

15%

20%

25%

30%

35%

40%

None Messaging Only Advising Only Messaging & Advising

% S

tude

nts w

ho re

gist

ered

ear

ly ** ** **

© 2015 ideas42 122

The ADVISING intervention INCREASED Fall semester RETENTION (7th day)

122

***

47,1% 46,3% 48,9% 48,0%

0%

10%

20%

30%

40%

50%

60%


% S

tude

nts

who

regi

ster

ed

**

• Based on data updated on August 30, 2015 • ** = significant at 95% level

© 2015 ideas42 123

The ADVISING intervention INCREASED fall semester RETENTION (14th day)

123

***

47,2% 46,3% 48,8% 48,1%

0%

10%

20%

30%

40%

50%

60%


% S

tude

nts

who

regi

ster

ed

*

• Based on data updated on September 7, 2015 • * = significant at 90% level

© 2015 ideas42 124

TABLE OF CONTENTS

I. Project Recap: Define, Diagnose, Design, Test II. Key Results III. Subgroup Analysis

IV. Discussion of Results

124

© 2015 ideas42 125

THE INTERVENTIONS HAD STRONGER EFFECTS ON SOME VULNERABLE GROUPS BUT ON NOT OTHERS

Group Messaging Advising Combined

Overall +20% +13% +21%

Minority Students +18% +18% +36%

Developmental +32% +35% +34%

First-Generation +24% +2% +21%

No EFC +17% +9% +15%

Full-Time +21% +16% +31%

Part-Time +18% +10% +8%

• No consistent pattern emerged in the subgroup analysis. 125

© 2015 ideas42 126

TABLE OF CONTENTS

I. Project Recap: Define, Diagnose, Design, Test II. Key Results III. Subgroup Analysis IV. Discussion of Results

126

© 2015 ideas42 127

SUMMARY OF KEY RESULTS

127

Both the Messaging and Advising interventions succeeded in increasing early registration.

The Advising intervention increased overall registration (retention) in addition to early registration.

Subgroup analysis showed both interventions helped students taking developmental courses more than other students, but the pattern wasn’t consistent.

Students opened emails in the first 4 weeks of the Messaging intervention at a much higher rate (43.0%) than in the last 4 weeks (12.9%).

The mechanism of the Advising intervention’s success appears to be getting more students to engage with Advising.

© 2015 ideas42 128

TAKEAWAYS & Recommendations FOR SINCLAIR

128

Advising Consider formalizing an outbound calling program in Advising as an effective way to

increase early registration and retention. Engaging with Advising is good for students, so finding innovative ways to increase such

engagement can drive positive outcomes.

Messaging Behaviorally designed reminder messages that keep registration top of mind, facilitate

next steps, and promote interim deadlines are effective in prompting action on key priorities.

Students opened emails in the first 4 weeks at much higher rates (43.0%) than in the last 4 weeks (12.9%), so limit overall messaging volume by focusing on the first part of the registration period.

Continue experimenting with text messages to nudge critical actions.

Using insights from behavioral science to break barriers to post-secundary education

Government & Nonprofit

Transcript of Using insights from behavioral science to break barriers to post-secundary education