Predictive AnalyticsTurning Insights into Action to Improve Student Success

Darren Catalano – Vice President of Analytics, UMUCKaren Vignare, Ph.D. - Associate Provost, Center for Innovation in Learning, UMUCLaura Malcolm - Vice President of Product, Civitas Learning

Sloan C Emerging Technologies for Online Learning April 2014

• Pioneer in adult and distance education since 1947• One of 11 accredited, degree-granting institutions in the

University System of Maryland• Focus on the unique educational and professional

development needs of adult students• More than 90,000 students enrolled worldwide

About UMUC

About Civitas Learning• Founded in 2011 by Charles Thornburgh and Mark Milliron• Provides cloud-based predictive analytics applications for administrators, faculty,

students, and advisors• Helps answer the question of what’s working, what’s not working, for which

students, at each point in their learning journeys

Organize Institutional Data

Unlock Foundational Insights

Enable Targeted Action

Your Analytics Story

• Are you pulling all your data? Some of your data? None of your data?

• If you have data, how are you analyzing it?• Do you measure student interventions?• Do you see analytics driven interventions as

strategic?

4

Descriptive Analytics

• What has happened?

• Informative

• Reports & dashboards

• Describing information or data

Predictive Analytics

• What is most likely to happen?

• Potentially transformative

• Targeted alerts

• Delivering actionable, real-time insights

5

MORE ACTIONABLE DATA

What is Predictive Analytics?

Asking the Right Questions to Prompt Action

6

Predictive Analytics

Which applicants are most likely to succeed at this institution?

Which students are most likely to not complete a particular course?

Which students are at the highest risk of withdrawing?

Which students and faculty have low online engagement within a course?

Which students are more likely to persist in their program of study? Which students are

least likely to complete a particular course?

Predictive Analytics @ UMUC

Opportunity• UMUC has a wealth of historical data that can be used to

understand student risk profiles• Predictive modeling of UMUC data shows there is a complex set

of variables contributing to student success or failure

Solution• Partner with a vendor for advanced modeling expertise and rapid

application development• A Student Success System provides a daily, individualized risk

prediction for every enrollment • Pilot programs underway to determine best ways to leverage and

act upon student risk predictions

7

Predictive Modeling in Action

Data is gathered and normalized for analysis.

Predictive factors are identified and custom models are created.

Personalized, real-time recommendations are delivered via web-based apps.

8

Model Development

1. Create features based on available datasets

2. Segment students based on data availability

3. Select top variables through model development (allow the data to tell the story)

4. Create models and test predictions

5. Cluster students based on top feature value similarities(Repeat steps 3 and 4)

6. Choose model through extensive model competition

7. Iterate models as data refreshes, new variables become available, and methodologies improve

This approach uses the same predictive methodologies as consumer industries such as equity trading, internet search, book recommendation (Amazon).

Overall Model Accuracy (for Predicting Course Completion) 84% at day 0 88% at day 6 (10% into course)

10

Five types of factors measured: Student Incoming Profile: Factors present at the time the student enrolls

in the course and do not typically change during the course of the term Student Attendance: Frequency and patterns of online course attendance Student Activity: Student behavior (tool usage, interaction, etc.) in the

LMS classroom Faculty Activity: Faculty behaviors in the LMS online classroom Grades: Assignment submissions and grades from throughout the course

Risk Factors and Model Results

Student Success Application

• Early and through-out term, identify enrollments at risk of not successfully completing the course

• Identify the courses and faculty with the highest number of at-risk enrollments

• Provide a student-level summary of risk factors to better understand why a student is at risk and inform necessary action

• Facilitate intervention to impact predicted outcome

DEMO

Cross Functional Intervention Team

Interventions Team

Advising

Academic Departments

Provost Office

Analytics

Inst. Research

Student Services

13

Academic vs. Advising Interventions

Academic• Email only• Includes subject or course-

specific content• Promotes value of course—

what they will learn and why its important

• Highlights key milestones or challenges in the course

• Provides specific tips for success

Advising• Email followed-up by a phone

call• Generic talking points used for

all courses, such as…• Are they prepared and in the

right class?• Do they have any issues and

are they getting the help they need?

• Do they have a clear plan for success?

14

Pilot Results

• Currently in 4th iteration of pilot• Fall 2013 results are statistically significant

– Successful course completion for undergraduate courses was 3 percentage points higher

– Graduate courses did not show meaningful change

15

Current Pilot Activities Based on Analytics

• Re-envisioned Onboarding (Jumpstart)• Continuation of Nudge Interventions• OERs/Eresources• Adaptive Learning• Competency Based Education

Can We Change the Equation?

17

Learner Characteristics

Academic Integration

Course & Program characteristics

Instructor behaviors

Learner Behaviors

Social-Psychological integration

Completion, Reenrollment,

completion

Predictive Analytics

Adapted from PAR Predictive Analytics & Retention Model

Next Steps

• Building UMUC Roadmap with Civitas Learning– Illume– ISSM (Incoming Student Success Model)– Inspire for Advisors

• Learn from Civitas Community• Learn from PAR and Kresge grant on transfer

students

18

19

Predictive Analytics Discussion

Thank you!