Predictive Analytics as a Tool to Improve Retention and ... · Predictive Analytics as a Tool to...

Predictive Analytics as a Tool to Improve Retention and Success

Craig KelleyDean of Academic AchievementStonehill College

Brian OlesDirector, Institutional Research & AssessmentStonehill College

Carmen CidDean of Arts & SciencesEastern Connecticut State University

Cohorting of First-Year Students :: AAC&U January 2017

About Stonehill

• Small Catholic, founded by the Congregation of Holy Cross

• 2,400 undergraduate students

• Offer BA, BS, and BSBA degrees in the humanities, arts, natural and social sciences, business, and education

• 93% of students reside on campus

• Located south of Boston in Easton, MA


Eastern student profile

5200 students– the CT public liberal arts universityFull time undergraduate – 4,30095% from Connecticut–4% out of state- 1% international54% female/46% male – 50% first-generation26% minority – 29% minority for freshman class89% freshman class lives on campus59% full-time undergrads living on campus1130 graduates for undergraduate class----16% enroll in further education within 2 yrs--71% gain employment in CT within 10 months

of graduation


What motivated us to do strategic cohortingof incoming first year students?

• Maximizing effective use of resources for advising and other academic support services

• Increasing student retention • Facilitating progress to graduation in 4 years• Faculty willing to use their talents as “problem solvers”

and students already serving as effective “change agents”

• Nellie Mae Foundation Project Compass grant initiative (2007-2012) and Title III Support Services grant (2009-2014) as funding catalysts


Eastern’s approach rely heavily on data analyses to identify students at risk early in their relationship with the University.

Initially a multivariate model to predict the likelihood that a student would withdraw from Eastern prior to 2nd year.

The model was composed of characteristics that the student brought to the University, such as gender, race/ethnicity, SAT scores, high school GPA, SES/income, etc.

Now using two-model approach now to cross classify students by two types of risks –withdrawal and low academic performance

Forming Targeted Advising Cohorts (TAC)

Most

Least

MostLeast


Data-Intensive Approach

Targeted Advising Cohorts (TAC) Assignments For Fall 2011 Cohort*

TAC1=IntensiveTAC2=TutoringTAC3=EngagedTAC4=Monitor

33.2%

25.5%

22.8%

18.5%

* 35 students originally in TACs 3 and 4 that did not participate in the library assessment and orientation were reassigned to TAC2.

By using the two-model approach it was determined that 18.5% of the FTFT students entering were at risk of withdrawal and low academic performance (TAC1), and another 25.5% were at risk of low academic performance (TAC2).


Utilization and Outcomes - Summary of Outcomes by Support Area and TAC

Group General Characteristics Outcomes (Average)

N % Female % 1st Gen % Students of Color # Credits Att. # Credits Earned1st Yr. GPA Lib.Score Retention

FTFT 2011 927 53.4 32.5 22.0 29.40 26.18 2.79 1.11 75.5TAC1 171 32.2 27.5 15.7 28.88 24.18 2.34 0.98 67.3TAC2 235 37.0 40.0 42.4 28.83 23.13 2.38 0.99 74.9TAC3 210 66.7 23.8 3.4 29.23 27.85 3.10 1.15 71.9TAC4 306 69.0 35.0 22.6 30.23 28.49 3.14 1.23 83.7

Most

Risk

• There is very little variation in credits attempted across the four TACs.• Students in TAC1 and TAC2 on average were slightly less likely to achieve the same

level of academic momentum as their classmates in terms of credits earned.• In addition, the average GPAs for students in TAC1 and TAC2 are significantly lower

than those in TACs 3 and 4.• TAC assignment is clearly related to retention.


Utilization and Outcomes - Targeted Advising Cohorts (TACs) by Supportive Services

Ave. Minutes Writing Ctr

Ave. Math Ctr (MAC) Hrs

Ave. No. Advising Center Visits

Ave. Lib.Score

TAC1 73.20 12.38 2.53 .98

TAC2 90.90 12.91 2.66 .99

TAC3 111.16 15.31 1.69 1.15

TAC4 86.79 13.93 1.90 1.23

Academic/Retention Risk

Most

Least

• Students most at risk are less likely to devote as much time to the Writing Center (TAC3 /TAC4 combined average was 96.06 hours).

• Students in TAC1 and TAC2 did not spend as much time in the Math Center either, even though they had a higher propensity for taking MAT 101P and MAT101W; both of which require MAC utilization.

• On average, students in TAC1 and TAC2 experienced more visits with the Advising Center. This is by design; these TACs are earmarked for the most intensive advising within the FTFT cohort.

• Library Score (a proxy for Freshmen readiness and/or motivation for college work) is lowest for those assigned to the most at risk TACs.


Utilization and Outcomes - Targeted AdvisingCohorts (TACs) by Supportive Services

• About 1 in 5 (19.1%) first-year students used the Writing Center.• While the percentage of students in TACs 3 and 4 that used the Writing Center was

higher than 19.2%, the percentage in TACs 1 and 2 were lower than that figure.• So, students in TACs 1 and 2, those most at risk, show a weaker propensity to use

writing support, and invest less time when they do seek writing help.

Pct of TAC that used Writing Center

Total FTFT 2011 Cohort 19.2

TAC1 14.6

TAC2 17.4

TAC3 20.0

TAC4 22.5


Most

Least

Table 4


Utilization and Outcomes - Targeted Advising Cohorts (TACs) by Outcomes

Did not use Writing Ctr

Used Writing Ctr

TAC1 67.1% 68.0%

TAC2 72.7% 85.4%

TAC3 71.4% 73.8%

TAC4 81.9% 89.9%


Most

Least

• Regardless of what TAC, students that used the Writing Center were more likely to be retained.

1st to 2nd Year Retention Rate (Writing Center)


Incremental Effectiveness – Writing Center

Modela(n= 177)

R Square R Square Change Beta

1 Library Score .088 -- .297***

2 Library Score

Total Minutes/ Writing Center

.099 .021 .311***

.146*

Adjusted R Square, R Square Change, and Betas for stepwise regression predicting 1st Year GPA

Time spent in the Writing Center is significantly (p<.05) predictive of increased academic performance (1st Yr. GPA) above the motivation indicated via the Library Score. However, total minutes in the Writing Center did not add significantly to predicting the number of credits earned in Year 1.

a Final outcomes of two-stage regression model entering Library Score in step 1 and Total minutes in Writing Center in step 2. * p < .05; ** p < .01; *** p < .001


Utilization and Outcomes - Targeted Advising Cohorts (TACs) by Outcomes

Did not use MAC

Used MAC

TAC1 63.9% 73.0%

TAC2 71.7% 79.4%

TAC3 71.8% 72.2%

TAC4 83.3% 84.5%


Most

Least

• Regardless of what TAC, students that used the MAC were more likely to be retained.

1st to 2nd Year Retention Rate (Math Achievement Center)


Some Conclusions/ImplicationsEastern’s use of data analytics has evolved in three stages of development First, the predictive modeling and Targeted Advising Cohorts that came out of Project

Compass, both of which relies exclusively on student past performance and characteristics that s/he bring to college (i.e., input variables);

The introduction of a proxy for motivation/academic integration (i.e., library score) Eastern is now at the point where a third stage that measures a student’s level of

academic readiness early after they enroll at Eastern needs to be explored – updating and assessing First Year Experience

• The models that developed TACs are extremely reliable in identifying which groups are most at risk of academic difficulty in the first year. Graduation rates have increased greatly for all student cohorts and highest in our CSCU system.

• An added predictor has been the library orientation score – serving as a “real-time” measure of student motivation.

• Measures of self confidence and/or appraisals of academic self efficacy need to be explored to tackle other reasons that limit first-year retention.


•Upgrading the First Year Program Experience, and Liberal Arts Curriculum with assessment

•Improving choice of major connection to incoming career aspirations - to facilitate initial engagement with the university

•Continued emphasis on the Liberal Arts Work! and LEAP real-world problem-solving initiatives

•Continued focus on university-wide Civic Action Plan initiative to promote student engagement and continued progress to graduation

Current Focus


Think. Act. Lead.• An integrative and collaborative program and philosophy for

working with students• Focus on mentorship and the individualized holistic (i.e.,

academic, professional, personal, and spiritual) development of each student

• Creating a campus-wide culture that promotes:– Intentional and meaningful conversation between students and the

mentors they encounter from across the entire College community– Ongoing purposeful planning, intentional engagement, and thoughtful

reflection upon all experiences


Funding

– $233,800 over two years

Community AwarenessTechnologyRevision of First-Year Experience (FYE) CourseCohorting• An academic success and retention initiative, inspired by the work done at Eastern

Connecticut State University


First-to-Second Year Retention;

1st Semester College GPA

Academic Ability(proxied by HS GPA, SAT scores, HS Class Rank)

Course of Study(Major & Program)

Engagement or Sense of Fit

(proxied by pre-enrollment visits to campus,

geographical proximity, legacy, commuter, athlete)

Check for any Demographic Differences

(gender, ALANA, first gen, non-citizen)

Finances & Perceived Value

(demonstrated need, net price)

Using data known prior to enrolling, examine if/how variables are correlated with college academic performance & retention.


High Academic Risk/ Low Withdrawal RiskHS GPA: 3.23

SAT: 1580HS Rank: 33%

Male: 43%ALANA: 15%First Gen: 27%

Demonstrated Need : $25KLegacy: 20%

Massachusetts Resident: 61%Visited Campus (prior to enrolling): 96%

Low Academic Risk/ Low Withdrawal RiskHS GPA: 3.65


Male: 22%ALANA: 2%

First Gen: 23%Demonstrated Need : $18K

Legacy: 22%Massachusetts Resident: 65%

Visited Campus (prior to enrolling): 95%

Low Academic Risk/ High Withdrawal RiskHS GPA: 3.59





High Academic Probation Risk

Low Academic Probation Risk

Low

Wit

hdra

wal

Ris

kH

igh Withdraw

al Risk

Source: Education Advisory Board

High Academic Risk/ High Withdrawal RiskHS GPA: 3.08

SAT: 1621HS Rank 33%





Cohort 2Tutoring

Cohort 1IntrusiveAdvising

Cohort 4Monitor

Cohort 3Engagement

High Academic Probation Risk

Low Academic Probation Risk

Low Withdrawal Risk High Withdrawal Risk

Academic Risk

Receives targeted tutoring

Academic & Attrition Risk

Receives targeted tutoring, int rusive adv ising, & a

focus on engagement

High Likelihood of Persistence

Monitor engagement and first semester performance

High-Flyer Population

Increase campus engagement but realize likelihood of t ransfer

Source: Education Advisory Board

CohortingCreate an Initial Risk Profile Based on Pre-Enrollment Data


CohortingActions Taken as a Result of Cohorting• ONLY Academic Services & Advising Staff are aware of which

Quadrant Any First-Year Student Resides

• Assignment of Primary Faculty Advisors

• Assignment of Secondary Professional Staff Advisors• Why would a student be assigned a Secondary Advisor?

• Academic and/or Retention Risk Indicators

• Who serves as Secondary Advisors for First-Year Students?• Academic Services & Advising Staff• Intercultural Affairs Staff


CohortingActions Taken By Secondary Advisors• Establish Regular Meetings• Forge a Relationship in Order to Leverage Proactive Advising

Approaches• Academic Intervention Outreach (e.g., Meetings with Faculty, Tutoring

Referrals, Mid-Semester Assessment Follow-up, etc.)• Co-Curricular Engagement (e.g., Member of Club, Society, Organization,

and/or Athletic Team, Participating in Service or Volunteer Program, Attending Campus Programming, etc.)

• Personal Wellbeing (e.g., Exercising, Eating Regularly/Healthy, Engaging with a circle of Friends, Connecting with Resources (e.g., Counseling Services, Student Financial Assistance, etc.)


CohortingNext Steps in Pilot Initiative• Collaborate with Retention sub-committee on potential takeaways and

future initiatives• Assess the retention cohorts:

• Are they predictive of actual Class of 2019 retention and academic performance?• Are the cutoff points that determine the four quadrants effective or do they need

adjustment?


Cohorting• Assessing the retention cohorts

• Are they predictive of actual Class of 2019 retention and academic performance?

• Are the cutoff points that determine the four quadrants effective or do they need adjustment?

Retention Quadrant (Fall 2015 cohort)1

(H/H)2

(H/L)3

(L/H)4

(L/L)

Number in Quadrant 207 218 20 207

Number Returning for Second Year 184 203 18 198

% Returning for Second Year 88.9% 93.1% 90.0% 95.7%Cumulative GPA as of third semester start 2.89 3.00 3.44 3.45


• Outcomes– Utilization of campus resources

– Number of students on probation and separation

– 1st- to 2nd-year retention rate


Number of Enrolled Students

Difference AY15-16 vs. AY14-15Total Number of Students +3.2% (N = +76)

First-Years +6.5% (N = +39)


Career Development Center

Difference AY15-16 vs. AY14-15Total Appointments +14% (n = +225)First-Year Students

Unique Students +14% (n = +27)Total Visits +45% (n = +75)


Center for Writing & Academic Achievement

Difference AY15-16 vs. AY14-15Referrals +142% (n = +27)Total Visits +41% (n = +1,039)First-Year Students

Referrals +133% (n = +12)Unique Students +26% (n = +68)Total Visits +62% (n = +851)


Probation & Separation

Difference AY15-16 vs. AY14-15Probation -31% (n = -42)Suspended Separation -56% (n = -5)*Separation -13% (n = -3)*

*Estimated net tuition revenue impact fromretaining seven students: $140,000.


Probation & Separation

Difference AY15-16 vs. AY14-15First-Year Students

Probation -56% (n = -45)Suspended Separation -88% (n = -7)Separation -25% (n = -2)*

*Data suppressed due to sample size.


Class of 2019 1st- to 2nd-Year Retention

92.4%!!!First-to-Second Year Retention

Class of 2018 Class of 2019 Change

First Gen 88.7% 92.1% +3.4%

ALANA 83.3% 87.4% +4.1%

First Gen & ALANA 83.3% 92.7% +9.4%


Questions, Comments, DiscussionCarmen Cid: [email protected]

Craig Kelley: [email protected] Oles: [email protected]


Predictive Analytics as a Tool to Improve Retention and ... · Predictive Analytics as a Tool to...

Documents

Transcript of Predictive Analytics as a Tool to Improve Retention and ... · Predictive Analytics as a Tool to...