Tweaking the pilotA Case Study from DVMT 100 at Frostburg State University

Dr. Megan E. Bradley

DVMT 100

• 3 credits, does not count toward graduation or GPA*

• Must take if need MATH 102 (College Algebra) or MATH 106 (Algebra with Calculus – Business majors)

• About 450 students per year• 1078% increase since inception in 1985• No budget increase

Course Issues

• Failure rate with gender gap in DVMT 100:• 41% failure rate overall• 44% rate for males; 35% rate for females

• Failure rate in next math course:COURSE DVMT DWF RATE

NON-DVMT DWF RATE

DIFFERENCE

MATH 102 56% 39% 16%

MATH 106 43% 33% 11%

Course Issues

• Staffing issues• Relied solely on undergraduate students to

teach

• Course Drift• Delivery: ½ sections all face-to-face (f2f); other

½ all computer lab• Different textbook, syllabus, point system• No system for checking reliability of grading

What we did

• Emporium Model done right• 1 large lecture• Rest of time in lab

• Hired new staff member to serve as lead instructor• Undergraduates became ULAs, shifting role to lab

assistant

• Added material to help students in next math class

Pilot – spring ‘11• Traditional lecture

• all face-to-face (f2f) classes• no online work• taught by trained undergraduates• point system for course grade• 1 final exam but could have earned other points with previous

assignments to make final exam not have much weight

• Redesign• Lecture 1x/week by instructor & lab 4x/week with trained

Undergraduate Learning Assistants (ULAs) using ASAlgebra by Plato

• 3 modules & corresponding exams

• Mastery learning – retake exams until passed• Pass course by passing all 3 modules with 80% or higher

• Extra credit for attending and doing online homework & evaluates

assessment

• Pass/Fail rates

• Scores on “core questions” • Questions that show up on the redesign module

exams & the final exams for the traditional sections

• Focus groups

Pilot results

• Pass/fail• Historical failure rate: :

41%

• Redesign failure rate: 47.2% which was significantly worse than…

• Traditional failure rate: 22.6%

• Males failed more than females

Pilot results

• Core questions• Difficult to use final grades due to different

grading systems• Considering all core questions, a one-way

ANOVA of Type of Classroom (2: Redesign versus traditional) by Core Qs (All) was significant, F = 37.429, p = .000, eta2 = .327.

• Redesign students (X = 87.98%) performed significantly better than traditional students (X = 63.14%).

Pilot results

• Core questions• Below is a breakdown of core questions per module, • Students from the redesign section scored significantly

higher than traditional sections for all three modules:

• M1: Redesign (X = 86.20%) > traditional (X=83.66%)

• M2: Redesign (X = 84.90%) > traditional (X=74.07%)

• M3: Redesign (X = 90.85%) > traditional (X=59.05%)

Pilot results

• Regression indicated which of course activities significantly related to student grade on core questions.• Attendance: correlated but weak• Online homework: correlated but weak• Online evaluates: strongly correlated

• Homework & evaluates: needed 80% to pass and move on• Evaluates: Often only had 4 questions so needed

to get perfect score.

Additional results

• We examined students’ time on task and when they were using software.

• Reviewed focus group suggestions.

• Compared student performance on certain items in traditional sections.

• Created hypotheses and tested them out as best as we could.

• Reassessed the team

Issues & tweaks

1. Students compared DVMT 100 sections.

2. Redesign students did not effectively use their lab time wisely.

3. Redesign students did not have enough deadlines – 1x/module, night before exam.

4. Students fell behind next module while retaking previous module exam.

1. Fall 2011 – full implementation.

2. Changed lab to 2x/wk and used technology to block other sites.

3. Created several deadlines with last deadline before test review day.

4. Added retake week after Mod1.

Issues & tweaks

1. The grading system in the redesign confused students.

2. Redesign students found and exploited a loophole about retaking modules next semester.

3. Lab assistants were scattered across different labs.

4. No pedagogy to address gender gap.

1. Revised to be based on weights that required and rewarded important course aspects.

2. Modified retaking of modules.

3. Assigned lab assistants.

4. Created Train Your Brain Program

Issues & tweaks

1. Failure rate on first version of module exam was very poor:

• Mod1 = 27% passed

• Mod2 = 20% passed

• Mod3 = 17% passed

1. Implemented PreModule Exam

• Earn 85% or higher – no need to take Module exam

• Reward studying & doing well

•Fall 2011 pass/fail rate• 20.3% failure rate overall• 19.7% rate for males; 21.3% rate for females• Gender analyses NOT statistically significant.

Full implementation results

• Remember this?• Failure rate with gender gap:

• 41% failure rate overall• 44% rate for males; 35% rate for females• Pilot redesign failure rate: 47.2%

W FX F NC P0

10

20

30

40

50

60

70

80

90

GRADE

GRADE

Why Stop There?

assessment

• Score on pre-test in next math class• Non-DVMT• Redesign DVMT• Traditional DVMT

Pre-Test in Next Math Course

Math Level Student Type Pre-Test Score

ML 3, 4  Eligible for Pre-Cal or Calculus 63%

ML 2   Eligible for College Algebra Repeating the Course

67%

ML 2  Eligible for College AlgebraFirst Time Students

62%

ML 1   DVMT 100 Redesign Graduates 60%

ML 1 Traditional DVMT 100 Graduates 51%

Why Stop There?

assessment

• Final grades in next math class• Who passed with a C or higher?

ALL ST

UDENTS

NON-DVM

T

TRAD

ITIO

NAL D

VMT

DVMT

REDES

IGN

0%10%20%30%40%50%60%70%80%

0.56

% PASSED

% PASSED BY GROUP

ALL ST

UDENTS

NON-DVM

T

TRAD

ITIO

NAL D

VMT

DVMT

REDES

IGN

0%10%20%30%40%50%60%70%80%

56%

35%

% PASSED

% PASSED BY GROUP

ALL ST

UDENTS

NON-DVM

T

TRAD

ITIO

NAL D

VMT

DVMT

REDES

IGN

0%10%20%30%40%50%60%70%80%

56%

35%

24%

% PASSED

% PASSED BY GROUP

ALL ST

UDENTS

NON-DVM

T

TRAD

ITIO

NAL D

VMT

DVMT

REDES

IGN

0%10%20%30%40%50%60%70%80%

56%

35%

24%

41%

% PASSED

% PASSED BY GROUP

HIS

TORIC

AL

NO

N-D

VM

T

NO

N-D

VM

TH

ISTO

RIC

AL

DVM

TD

VM

T RED

ESIG

N

0%

20%

40%

60%

80%

100%

0.380.56

FAILURE RATE MATH 102

FAILURE RATE FOR MATH 102

HIS

TORIC

AL

NO

N-D

VM

T

NO

N-D

VM

TH

ISTO

RIC

AL

DVM

TD

VM

T RED

ESIG

N

0%

20%

40%

60%

80%

100%

38%53% 56%

42%

FAILURE RATE MATH 102

FAILURE RATE FOR MATH 102

Impact of changes

• Deadlines = large % students completed deadlines

• Weights & lab changes = better attendance and time on task

• Train Your Brain = no gender gap, better performance overall

Impact of changes

1. Failure rate on first version of module exam was very poor:

• Mod1 = 27% passed

• Mod2 = 20% passed

• Mod3 = 17% passed

1. PreModule Exam results• Module 1

• Premod: 36% passed

• Version 1: 72% passed

• Module 2

• Premod: 16.2% passed

• Version 1: 60% passed

• Module 3

• Premod: 18.3% passed

• Version 1: 53% passed

Overall recommendations

• Look, look, look.

• Add structure.

• Improve based on evidence (from pilot, from other redesigns, from published research)

• Add psychology• Provide incentives• Spacing effect• Practice effect• Mastery learning

Your reward