Is That a Check Engine Light? The UCF Student ......2015 Sum 2015 Fall 2015 Spr 2016 Sum 2016 Fall...

Is That a Check Engine Light?The UCF Student Performance DashboardFebruary 22, 2018

Thomas Cavanagh, Ph.D.Vice Provost for Digital Learning

UCF

• Orlando, FL

• Metropolitan, suburban university

• 66,300+ students

• One of the largest universities in U.S.

• Carnegie classification: RU/VH Research University: Very High Research Activity

• 216 degree programs across 11 colleges

• 11 Campuses throughout Central Florida

2

Digital Learning Context

3

Online Learning at UCF

4

Pegasus Innovation

Lab

Blended Adaptive

Pilot

Adaptive Learning

Learning Analytics

Online Courses

Online Programs

Mixed-Mode

Lecture Capture

Web-Enhanced

Academic Year 2017-18

• 44.24% total university SCH online and blended• 32.43% fully online SCH

• 82.26% of all students took at least one online or blended course

• 84.97% of all undergraduates

• 64.91% of all graduate students

• 72.85% of all students took at least one fully online course

• 75.5% of all undergraduates

• 55.72% of all graduate students

5

Colleges over 50% online SCH

oHospitality

70.2% | 24.2% online only

oHealth & Public Affairs


oNursing


oBusiness

55.2% | (<.01 blended)

oArts and Humanities

52.6% | 40% online only

Colleges over 25% online SCH

oUndergrad Studies


oGraduate Studies


o Education


o Sciences


Academic Year 2017-18

6

100,000

300,000

500,000

700,000

900,000

1,100,000

1,300,000

1,500,000

1,700,000

02-03 03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 12-13 13-14 14-15 15-16 16-17

F2F OTHER VIDEO BLENDED (WEB) ONLINE (WEB)

UCF Today:66,000+ Students

7

100,000

300,000

500,000

700,000

900,000

1,100,000

1,300,000

1,500,000

1,700,000

02-03 03-04 04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 12-13 13-14 14-15 15-16 16-17

F2F

Without Online Learning:44,000 Students

8

Online Exclusive Headcount by Semester

9

6,546

9,916

6,971 6,957

10,389

7,1607,444

11,110

7,873 7,675

11,947

9,125 8,945

14,177

9,98710,495

14,709

10,654

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

Spr 2012

Sum 2012

Fall 2012

Spr 2013

Sum 2013

Fall 2013

Spr 2014

Sum 2014

Fall 2014

Spr 2015

Sum 2015

Fall 2015

Spr 2016

Sum 2016

Fall 2016

Spr 2017

Sum 2017

Fall 2017

Online % and Speed to Graduation (FTIC Full-time)

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

0% (face-to-face only)

1-20%

21-40%

41-60%

61-80%

81-99%

100% (online only)

Time to Degree (2014-15)

Time to Degree (Years)

N=0

N=5

N=30

N=381

N=1,556

N=2,155

N=231

10

88 91 88 88 90 8791

95 92 9195

8990 92 91 90 91 89

0

10

20

30

40

50

60

70

80

90

100

Spring 15 Sum 15 Fall 15 Spring 16 Sum 16 Fall 16

F2F (n=710,463) Blended (n=99,111) Fully Online (n=234,676)

Student Success(A, B, or C grade)

11

4 3 4 4 3 43 2 3 3 2 34 3 4 4 3 40

10

20

30

40

50

60

70

80

90

100

Spring 15 Sum 15 Fall 15 Spring 16 Sum 16 Fall 16

F2F (n=710,463) Blended (n=99,111) Fully Online (n=234,676)

Student Withdrawal

12

Blended Learning 57%

Fully Online 55%

Face-to-Face 53%

Video (fully online) 47%

Video (blended) 45%

N = 1,431,907

– Dziuban & Moskal, 2017

Overall “Excellent” Student Ratings

13

Data Context

14

Data Usage

• Descriptive:

• Diagnostic:

• Predictive:

• Prescriptive:

15

Courtesy of Joel Hartman

Data Usage

• Descriptive: what happened?

• Diagnostic: why did it happen?

• Predictive: what will happen?

• Prescriptive: how can we make it happen?

16


Data Usage





17


Data Usage





18


A Shift in Perspective

• We used to think of students as cohorts and our data about them was historical, e.g.:

• Last fall’s freshman class

• This term’s transfer students

• The problem with this approach is that by the time we can see the data, the damage is already done

19


Predictive Analytics

• Predictive analytics is the practice of extracting information from existing data in order to predict future outcomes and trends

• Using predictive analytics allows us to intervene with students on an individual basis before a problem occurs

20


21

Actionable Insight

TransactionalData

TransactionalData

TransactionalData


22

Actionable Insight

CRM

LMSERP


23

Course Term Year Degree

STUDENT SUCCESS


Predictive Analytics Toolkit

• UCF-generated analyses

• EAB SSC-Campus

• UCF-developed Student Dashboard

24


EAB SSC-Campus

25

UCF Student Performance Dashboard

https://lti.online.ucf.edu/spd-standalone/26

https://lti.online.ucf.edu/spd-standalone/

Fall 2016 Survey Results (n=96)

• The majority of respondents were juniors (54%) or seniors (24%), expecting As (41%) or Bs (49%)

• 41% of students never used the dashboard, but 25% used it daily.

• Of the very few students (n=28) who offered open ended responses as to why they did not use the dashboard, the overwhelming majority indicated a lack of awareness regarding the dashboard rather than dislike of the dashboard.

• The dashboard was reported as being easy or very easy to use by 79% of respondents (n=52), with the ‘assignments due soon display’ and ‘current score display’ being the most utilized and most helpful features.

• The ‘goals display’ was rated as the least helpful feature by 20% of the respondents.

27

Fall 2016 Survey Results (n=96)

• Most (72%) of the respondents felt the dashboard was somewhat or very accurate, although 15% were unsure.

• Students expressed concerns over grades being displayed as artificially low during the beginning of the semester as a lack of grades input made it appear as if they were failing.

• Students liked the accessibility and visualization of the dashboard, especially for keeping up to date with assignments and course performance.

• 57% of students who used the dashboard believed it motivated them to change their behavior either a little or a great deal in regards to the course.

28

Lessons Learned & Next Steps

• Communication is key• Even with prompting, students did not know it was

there

• Pilot launched mid-semester—not optimal timing

• Technical issues

• Re-architecture and relaunch pilot

• New UI/UX

• Tool tips / Pop up info

29

Other Relevant Initiatives

30

COHPA Competency Dashboard

• Track program competencies for accreditation and compliance

• Competencies mapped in a custom Peoplesoftapplication

• Faculty map competencies to assignments within Canvas

31

COHPA Competency Dashboard

• Track program competencies for accreditation and compliance

• Competencies mapped in a custom Peoplesoftapplication

• Faculty map competencies to assignments within Canvas

32

Notes:

• This tab is only visible and accessible to the course instructor

“Zero Probation”

33

Grade ranges in a course

Notes:

• Instructor can get a bird’s eye view of students in different grade ranges


34

Number of students with grade

Notes:

• Instructor can look for changes in student counts in each grade range


35

Students in selected grade range

Notes:

• Select specific students or the entire group


36

Message to be sent to selected students

Notes:

• Instructor can send a custom message


37

Notes:

• Instructor selects a student in a group


38

Notes:

• Messages are sent to selected students in Webcourses (Canvas)

• Will show up in conversations inbox


39

College Algebra: Change In

Student Progress Over Time

40

Kn

ow

led

ge C

ove

red

Kn

ow

led

ge C

ove

red

Days Days

Finished course early Started late and finished on time

Math Sequence: Adaptive

Learning Across Courses

41

Psychology: Mean Module Scores By Course Success

0

2

4

6

8

10

1 2 3 4 5 6 7 8

Mea

n

Module

Success(n=258)

Non-success(n=20)

42

Students’ Adaptive Learning Behaviors

44

Data Mining Partnership

• Examine Canvas LMS data to determine correlations with student success

• Student behaviors

• Faculty behaviors

• Usage of certain tools/features

• Etc.

45

Thomas [email protected]

@tbcavanagh

digitallearning.ucf.edu

Is That a Check Engine Light? The UCF Student ......2015 Sum 2015 Fall 2015 Spr 2016 Sum 2016 Fall...

Documents

Transcript of Is That a Check Engine Light? The UCF Student ......2015 Sum 2015 Fall 2015 Spr 2016 Sum 2016 Fall...