A familiar(ity) problem: Assessing the impact of prerequisites and content

familiarity on student learning

Brian Sato

University of California, Irvine

UC System: Teaching Professor(Lecturer SOE)

• Tenure-track

• Expectations of quality:

– Teaching– Professional

Development

– Service

60%

30%

10%

Typical STEM curriculum

Bio 1 Bio 2 Bio 3

Bio 1

Bio 2

Why do we have prerequisites?

Student Perceptions of Prerequisites

Positive Attributes Fraction of Interviews

Background knowledge 89.3%

Acts as a safety net for students 35.7%

Responsible for future success 25.0%

Contributes to interest in subject material 21.4%

Positively impacts how instructors teach 14.2%

Improves student behaviors 7.1%

Improves overall quality of students 7.1%

Scheduling 7.1%

Student Perceptions of Prerequisites

Negative Attributes Fraction of Interviews

Scheduling 51.7%

Waste of student’s time or money 37.9%

Not used as intended by faculty 31.0%

Student’s are prepared without the prerequisite 17.2%

Not used as intended by students 13.8%

Why do we have prerequisites?

• Teach students content/skills necessary for success in future courses

• Students need time to mature to succeed in later courses

• Logistically easier to schedule future courses

Potential benefits are rarely assessed

How does your campus assess prereqs?

• Prereq = Exam/Course Grades

• Prereq course grade ~ Later course grade

Is prerequisite completion beneficial?

It depends!

Prerequisite Later scenario

assessed

Does prereq impact

later success?

Reference

Premed

Anatomy/Histolog

y

Anatomy/Histology

(Medical School) Yes

Forester et al.

2002

Premed Organic

Chemistry

Biochemistry in

Chiropractic SchoolYes

McRae. 2010

General Ed

Courses

ETS Major Fields Test

for BusinessYes

Ritchie et al. 2011

Organic

Chemistry

Course

Intro Biochemistry

Course No

Wright, et al. 2009

# of Business

School Prereqs

MBA Program GPANo

Christensen et al.

2011

Intro Business

Courses

Intermediate

Business CoursesNo

Jones et al. 2013

• Examine concept overlap on exam questions

Novel Means to Assess Value of Prerequisites - Familiarity

Genetics (Prerequisite)

Mol Bio

• Very Familiar – Students should be able to answer the Mol Bio exam question based on the Genetics prereq

• Familiar – This was discussed in the Genetics prereq

• Not Familiar – This was not discussed in the Genetics prereq

• Examine concept overlap – Familiarity

Novel Means to Assess Value of Prerequisites

Hypothesis: Students will perform better on MolBio exam questions covering more familiar concepts (VF > F > NF)

Genetics (Prerequisite)

Mol Bio

How to assign question familiarity?

Genetics Lecture Slides Genetics Instructor Genetics Students

Using:

How to examine the impact of familiarity on exam performance?

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10

Assign Familiarity (Genetics)

Mol Bio Exam

Questions

How to examine the impact of familiarity on exam performance?

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10

VF F NF

Calculate % correct for questions in each category

85% 75% 65%

Assign Familiarity (Genetics)

Mol Bio Exam

Questions

Do the familiarity assignment methods agree with each other?

0

20

40

60

80 Agree

Slightly Agree

Disagree

Methods Compared

Students & Slides

Students & Instructor

Slides & Instructor

Perc

enta

ge o

f Ex

am Q

ues

tio

ns

Exam Performance and Concept Familiarity

Familiarity Designation: Genetics Lecture Slides

*** p < 0.001

Shaffer et al. 2016

*** ***

VF F NF0

20

40

60

80

100

Perc

ent

Co

rrec

t

Familiarity Designation: Genetics Instructor

Shaffer et al. 2016

VF F NF0

20

40

60

80

100

Exam Performance and Concept FamiliarityPe

rcen

t C

orr

ect

Familiarity Designation: Genetics Students

Shaffer et al. 2016

VF F NF0

20

40

60

80

100

Exam Performance and Concept FamiliarityPe

rcen

t C

orr

ect

• Hypothesis:

VF > F > NF

Exam Performance and Concept Familiarity

At best, students do better on VF questions

But how can we control for differences in question type?

• Bloom’s taxonomy

Similar conclusions when controlling for Bloom’s level

Shaffer et al. 2016*p < 0.05 **p < 0.01 *** p < 0.001

Conclusions

• At best VF > F and VF > NF, but not F > NF

• Similar conclusions with:

• Human Physiology (prereq)/Human Anatomy

• Microbiology Lecture (prereq)/Lab

So what do we take away from this?

• Breadth vs. Depth

• Important to assess prerequisites

• Need for increased communication between faculty teaching related courses

Transforming Courses – Breadth vs. Depth

• Topics covered:– DNA/RNA/protein structure– DNA Replication– DNA Repair– Transcription– Prokaryotic Txn Regulation– Eukaryotic Txn Regulation – Chromatin, Txn Factors– mRNA processing, export– Non-coding RNAs– Genetic Code– Translation – Initiation, Elongation, Termination– Viruses

Mol Bio

Instructors A/B/C

Instructors D/E/F

Instructors G/H/I

Goal: Revamp the course to re-align the 3 sections

Transforming Courses – Breadth vs. Depth

Mol Bio

Instructors A/B/C

Instructors D/E/F

Instructors G/H/I

Goal: Revamp the course to re-align the 3 sections

Transformation of Mol Bio

• High Structure Format

– Increased active learning in lecture

• Clickers

• Student group work

– Problem-based discussion sections

– Emphasis on work outside of class

• Online textbook quizzes before each lecture

• Weekly online HWs

– Increased testing

• 3 “midterms”

• 1 cumulative final

Decrease content “covering”Increase depth of coverage

Impact of Course Transformation

• 19 question end of quarter student assessment

10

20

30

40

50

0

Perc

ent

Co

rrec

t

Low Structure Course

High Structure Course

*** p < 0.001

***

What next?

• Expand prerequisite analysis to additional courses/disciplines

• Would this work in your discipline?

• Can you think of a pair of courses (prereq/later course) to analyze?

Acknowledgements

• Justin Shaffer

• Pavan Kadandale

• Usman Alam

• Samantha Dacanay

• Jennifer Dang

• Amanda Lee

• Michael Dennin

• Sarah Eichhorn

Questions?