John Whitmer, Ed.D.Academic Technology Services

California State University, Office of the Chancellor

San Diego State UniversityApril 12, 2013

The Achievement Gap in Online Courses through a Learning Analytics Lens

Motivating Questions …

1. Do our current uses of academic technologies (such as online learning) decrease or exacerbate the achievement gap?

2. If we don’t believe our current uses serve these students, how do we know? What can we do about it?

Agenda

1. Context: CSU Achievement Gap & Conceptual Framework

2. Recent Conventional Research in Online Courses

3. Research using Learning Analytics & Course Redesign

4. Next Steps & Discussion

1. CONTEXT: ACHIEVEMENT GAP & CONCEPTUAL FRAMEWORK

Increasing Access to Higher Education in the U.S.

Table adapted from data in NCES Digest of Educational Statistics (2011)

  1976Percent of Enrollment

(1976)2010

Percent of Enrollment

(2010)

Percent Increase

All Students 10,986 21,016 91%White 9,076 83% 12,723 61% 40%Asian/Pacific Islander 198 2% 1,282 6% 548%

URM Students 1,493 14% 5,977 28% 300%American Indian 76 1% 196 1% 158%

Black 1,033 9% 3,039 14% 194%Hispanic 384 3% 2,741 13% 614%

Increased Access to Higher Education

All Students White Asian/Pacific Islander URM Students American Indian Black Hispanic0%

100%

200%

300%

400%

500%

600%

700%

Enrollment Increase by Race/Ethnicity in Higher Education (1976-2010)

Table adapted from data in NCES Digest of Educational Statistics (2011)

CSU Achievement Gap

By 2015, the CSU will improve graduation rates by 8 percentage points systemwide and halve the achievement gap. – Baseline 6-Year Graduation Rate: 46%– Target 6-Year Graduation Rate: 54%

– Baseline Achievement Gap: 11%– Target Achievement Gap: 5.5%

2

No Significant Difference: Framing Academic Technology

[academic technologies] are “mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition”. (Clark, 1983)

Image courtesy bsabarnowl @ Flickr

Index of studies: http://www.nosignificantdifference.org/

2. RECENT RESEARCH W/HYBRID & ONLINE COURSES

Adaptability to Online Learning: Differences Across Types of Students and Academic Subject Areas (2013: Xu, Jaggers)

Compares persistence and grade between online and f2f courses

Compares same student Washington Community and

Technical Colleges (2 yr) Studied 500,000 course

enrollments (10% online), 40,000 individuals

Data 2004-2009

Overall Findings

Table adapted from Xiu & Jaggers, 2013

Major Finding by Population

Overall Course ResultOnline GPA Average 2.77F2F GPA Average 2.98Entire Population -0.215Effect by Subject -0.267

Select Populations ResultBlack -0.394Males -0.288Academic Preparedness (F2F GPA<3.0 First Term) -0.314Age < 25 -0.300Cohort Effect (Courses w/+75% online at-risk students v. less than 25%) -0.359

Table adapted from Xiu & Jaggers, 2013

Major Finding by Subject

Overall Course ResultOnline GPA Average 2.77F2F GPA Average 2.98Entire Population -0.215Effect by Subject -0.267

Table adapted from Xiu & Jaggers, 2013

Select Subjects ResultsEnglish -0.394Applied Knowledge -0.322Social Science -0.308

Online Course

F2F CourseStudent

Grade

Grade

Treatments

Traditional Experimental Design

Image courtesy bsabarnowl @ Flickr

What’s your experience with Online or Hybrid Course Design?

Who has designed a fully online or hybrid course? (raise hands)

Of those who have, how many think it’s harder to create online/hybrid materials than to create face to face activities? (keep hands raised)

Peering into the blue box

Online CourseCours

e Desig

n

Faculty Course Development

Support (Reassigned

time, incentives, etc.) Student

use of online

materials

Specialist Support

(Instructional designers,

etc.)

Faculty &

Student

Training

Do we need *students* to adapt …

or

Do we need to change how *we* approach creating & evaluating

our technology-enhanced instructional materials?

3. LEARNING ANALYTICS & COURSE REDESIGN

Learner Analytics

“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)

Case Study: Intro to Religious Studies

• Undergraduate, introductory, high demand

• Redesigned to hybrid delivery format through “academy eLearning program”

• Enrollment: 373 students (54% increase on largest section)

• Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits)

• Bimodal outcomes:• 10% increase on final exam• 7% & 11% increase in DWF

• Why? Can’t tell with aggregated data

54 F’s

LMS Use Variables

Administrative Activities (calendar, announcements)

Assessment Activities (quiz, homework, assignments, grade center)

Content Activities (web hits, PDF, content pages)

Engagement Activities (discussion, mail)

Student Characteristic Variables Enrollment Status First in Family to Attend

College Gender HS GPA Major-College Pell Eligible URM and Pell-Eligibility

Interaction Under-Represented

Minority URM and Gender

Interaction

Correlation: Student Char. w/Final Grade

Scatterplot of HS GPA vs. Course

Grade

Predict the trend

LMS use and final grade is _______ compared to student characteristics and final grade:

a) 50% smaller

b) 25% smaller

c) the same

d) 200% larger

e) 400% larger

Predict the trend

LMS use and final grade is _______ compared to student characteristics and final grade:

a) 50% smaller

b) 25% smaller

c) the same

d) 200% larger

e) 400% larger

Correlation LMS Use w/Final Grade

Scatterplot of Assessment Activity

Hits vs. Course Grade

Combined Variables Regression Final Grade by LMS Use & Student Characteristic Variables

LMS Use

Variables

25% (r2=0.25)

Explanation of change in final grade

Student Characteristic

Variables

+10%(r2=0.35)

Explanation of change in final grade

>

Correlation LMS & Student Characteristic Variables w/Final Grade

At-Risk Students: “Over-Working Gap”

Activities by Pell and Gradegrade / pelleligible

A B+ C C-

Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible

0K

5K

10K

15K

20K

25K

30K

35K

Value

Content

Content

Engage

Engage

Assess

Assess

Admin

Admin

Content

Content

Engage

Engage

Assess

Assess

Admin

Content

Content

Engage

Engage

Assess

Assess

Content

Content Engage

Engage

Assess

Assess

Admin

Admin

Measure Names

Admin

Assess

Engage

Content

Extra effort in content-related activities

Next Generation Learning Analytics

Graphic Courtesy Sasha Dietrichson, X-Ray Research SRL

COURSE REDESIGN Flagship: Program in Course Redesign, led by Carol Twig

(1999-2004)– Pew funded 30 grants, $8.8M budget to redesign courses

for improved outcomes & lower costs (institutionalization)

Result: 25 of 30 courses reported increased learning outcomes, 5 no change (not worse!)

– 17 reduction DWF (10-20%)– Cost reduction 20-77%, $3M annual savings

Adopted (with modifications by CSU, SDSU, Chico State)– Chico State evaluations:

http://www.csuchico.edu/academy/outcomes/index.shtml

Sample Improvements DWF (drop-failure-withdrawal) rates at Drexel were consistently reduced 10-12

percent in the redesigned course.

At OSU, withdrawals were reduced by 3 percent, failures by 4 percent and incompletes by 1 percent. As a result, 248 more students successfully completed the course compared to the traditional course.

At TCC, students in redesigned sections had a 68.4 percent success rate compared to 60.7 percent for traditional sections. Success rates were higher for all groups of students regardless of ethnicity, gender, disability, or original placement. The overall success rate for all composition students was 62 percent for the 2002-2003 year compared to 56 percent for the 1999-2000 year prior to redesign.

In the traditional course at USM, faculty-taught sections typically retained about 75 percent of students while adjunct- and TA-taught sections retained 85 percent. In the redesign, the retention rate was 87 percent. The rate of D and F grades dropped from 37 percent in the traditional course to 27 percent in the redesigned course. DFW rates dropped from 26 percent in the traditional course to 22 percent in the redesign.

Source: Program in Course Redesign Round III: Lessons Learned (http://www.thencat.org/PCR/R3Lessons.html)

Underserved Student Experiences More comfort, higher

participation in online forums

Appreciate ability to anonymously rewind / repeat / review materials

English language learners decreased social anxiety

Source:  Twigg, C. (2005). Increasing Success for Underserved Students: Redesigning Introductory Courses. URL: http://www.thencat.org/Monographs/IncSuccess.htm

Proven PCR Techniques for Underserved Students

Interactive online tutorials (not PPT decks) Continuous assessment / feedback Increased student interaction Individualized, on-demand support Undergraduate learning assistants Structural supports that encourage student

engagement / progress

4. DISCUSSION

Discussion

Do you think there is an achievement gap at SDSU for under-served students in online / hybrid / tech enhanced courses? What evidence do you have for those beliefs?

What is SDSU doing about any existing achievement gap w/academic technology?

What supports are in place or could be developed?

Feedback? Questions?

John Whitmer jwhitmer@calstate.eduTwitter: johncwhitmer

Learning Analytics resources: http://www.johnwhitmer.net