A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar...
Transcript of A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar...
A familiar(ity) problem: Assessing the impact of prerequisites and content
familiarity on student learning
Brian Sato
University of California, Irvine
UC System: Teaching Professor(Lecturer SOE)
• Tenure-track
• Expectations of quality:
– Teaching– Professional
Development
– Service
60%
30%
10%
Typical STEM curriculum
Bio 1 Bio 2 Bio 3
Bio 1
Bio 2
Why do we have prerequisites?
Student Perceptions of Prerequisites
Positive Attributes Fraction of Interviews
Background knowledge 89.3%
Acts as a safety net for students 35.7%
Responsible for future success 25.0%
Contributes to interest in subject material 21.4%
Positively impacts how instructors teach 14.2%
Improves student behaviors 7.1%
Improves overall quality of students 7.1%
Scheduling 7.1%
Student Perceptions of Prerequisites
Negative Attributes Fraction of Interviews
Scheduling 51.7%
Waste of student’s time or money 37.9%
Not used as intended by faculty 31.0%
Student’s are prepared without the prerequisite 17.2%
Not used as intended by students 13.8%
Why do we have prerequisites?
• Teach students content/skills necessary for success in future courses
• Students need time to mature to succeed in later courses
• Logistically easier to schedule future courses
Potential benefits are rarely assessed
How does your campus assess prereqs?
• Prereq = Exam/Course Grades
• Prereq course grade ~ Later course grade
Is prerequisite completion beneficial?
It depends!
Prerequisite Later scenario
assessed
Does prereq impact
later success?
Reference
Premed
Anatomy/Histolog
y
Anatomy/Histology
(Medical School) Yes
Forester et al.
2002
Premed Organic
Chemistry
Biochemistry in
Chiropractic SchoolYes
McRae. 2010
General Ed
Courses
ETS Major Fields Test
for BusinessYes
Ritchie et al. 2011
Organic
Chemistry
Course
Intro Biochemistry
Course No
Wright, et al. 2009
# of Business
School Prereqs
MBA Program GPANo
Christensen et al.
2011
Intro Business
Courses
Intermediate
Business CoursesNo
Jones et al. 2013
• Examine concept overlap on exam questions
Novel Means to Assess Value of Prerequisites - Familiarity
Genetics (Prerequisite)
Mol Bio
• Very Familiar – Students should be able to answer the Mol Bio exam question based on the Genetics prereq
• Familiar – This was discussed in the Genetics prereq
• Not Familiar – This was not discussed in the Genetics prereq
• Examine concept overlap – Familiarity
Novel Means to Assess Value of Prerequisites
Hypothesis: Students will perform better on MolBio exam questions covering more familiar concepts (VF > F > NF)
Genetics (Prerequisite)
Mol Bio
How to assign question familiarity?
Genetics Lecture Slides Genetics Instructor Genetics Students
Using:
How to examine the impact of familiarity on exam performance?
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10
Assign Familiarity (Genetics)
Mol Bio Exam
Questions
How to examine the impact of familiarity on exam performance?
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10
VF F NF
Calculate % correct for questions in each category
85% 75% 65%
Assign Familiarity (Genetics)
Mol Bio Exam
Questions
Do the familiarity assignment methods agree with each other?
0
20
40
60
80 Agree
Slightly Agree
Disagree
Methods Compared
Students & Slides
Students & Instructor
Slides & Instructor
Perc
enta
ge o
f Ex
am Q
ues
tio
ns
Exam Performance and Concept Familiarity
Familiarity Designation: Genetics Lecture Slides
*** p < 0.001
Shaffer et al. 2016
*** ***
VF F NF0
20
40
60
80
100
Perc
ent
Co
rrec
t
Familiarity Designation: Genetics Instructor
Shaffer et al. 2016
VF F NF0
20
40
60
80
100
Exam Performance and Concept FamiliarityPe
rcen
t C
orr
ect
Familiarity Designation: Genetics Students
Shaffer et al. 2016
VF F NF0
20
40
60
80
100
Exam Performance and Concept FamiliarityPe
rcen
t C
orr
ect
• Hypothesis:
VF > F > NF
Exam Performance and Concept Familiarity
At best, students do better on VF questions
But how can we control for differences in question type?
• Bloom’s taxonomy
Similar conclusions when controlling for Bloom’s level
Shaffer et al. 2016*p < 0.05 **p < 0.01 *** p < 0.001
Conclusions
• At best VF > F and VF > NF, but not F > NF
• Similar conclusions with:
• Human Physiology (prereq)/Human Anatomy
• Microbiology Lecture (prereq)/Lab
So what do we take away from this?
• Breadth vs. Depth
• Important to assess prerequisites
• Need for increased communication between faculty teaching related courses
Transforming Courses – Breadth vs. Depth
• Topics covered:– DNA/RNA/protein structure– DNA Replication– DNA Repair– Transcription– Prokaryotic Txn Regulation– Eukaryotic Txn Regulation – Chromatin, Txn Factors– mRNA processing, export– Non-coding RNAs– Genetic Code– Translation – Initiation, Elongation, Termination– Viruses
Mol Bio
Instructors A/B/C
Instructors D/E/F
Instructors G/H/I
Goal: Revamp the course to re-align the 3 sections
Transforming Courses – Breadth vs. Depth
Mol Bio
Instructors A/B/C
Instructors D/E/F
Instructors G/H/I
Goal: Revamp the course to re-align the 3 sections
Transformation of Mol Bio
• High Structure Format
– Increased active learning in lecture
• Clickers
• Student group work
– Problem-based discussion sections
– Emphasis on work outside of class
• Online textbook quizzes before each lecture
• Weekly online HWs
– Increased testing
• 3 “midterms”
• 1 cumulative final
Decrease content “covering”Increase depth of coverage
Impact of Course Transformation
• 19 question end of quarter student assessment
10
20
30
40
50
0
Perc
ent
Co
rrec
t
Low Structure Course
High Structure Course
*** p < 0.001
***
What next?
• Expand prerequisite analysis to additional courses/disciplines
• Would this work in your discipline?
• Can you think of a pair of courses (prereq/later course) to analyze?
Acknowledgements
• Justin Shaffer
• Pavan Kadandale
• Usman Alam
• Samantha Dacanay
• Jennifer Dang
• Amanda Lee
• Michael Dennin
• Sarah Eichhorn
Questions?