Assessing the Impact of Peer Instructor Generated Formative Self-Assessments
Transcript of Assessing the Impact of Peer Instructor Generated Formative Self-Assessments
1ExamSoft Assessment Conference | #ASSESS2015
ExamSoft Assessment Conference
#ASSESS2015
2ExamSoft Assessment Conference | #ASSESS2015
Assessing the Impact of Peer Instructor Generated Formative
Self-AssessmentsPresenters:
Melinda Lull Ph.D., Jennifer Mathews Ph.D. Wegmans School of Pharmacy
St. John Fisher College, Rochester, NY
Jane Marie Souza, Ph.D.University of Rochester, Rochester, NY
3ExamSoft Assessment Conference | #ASSESS2015
Learning Objectives for the Session:
• Demonstrate the process of enrolling peer instructors as exam item writers.
• Demonstrate the use of the internal comments box as a means of providing feedback to peer instructors.
• Describe the process for analyzing ExamSoft data in the comparison of quiz and exam performance.
4ExamSoft Assessment Conference | #ASSESS2015
Why use self-assessments?
• Self-assessments increase course performance. (Stewart et.al., 2014; West and Sadoski, 2011)
• Self-assessments increase self-directed learning. (Nicol et.al., 2006)
• Students achieve better long-term retention of material. (Nevid and Mahon, 2009)
• Students get more practice with the format of in-class exams.
5ExamSoft Assessment Conference | #ASSESS2015
Why use peer instructors?
• Peer instructors increase student performance. (Santee and Garavalia, 2006)
• Students may feel more comfortable with peers than faculty. (Carrol, 1996)
• Peer instructors themselves benefit from reviewing material. (Haist et.al., 1997; Haist et.al., 1998)
6ExamSoft Assessment Conference | #ASSESS2015
Tra
inin
g FacultyExamSoft Infrastructure
Enrolling Peer Instructors
StudentsQuestion Writing
ExamSoft
Imp
lem
enta
tion
Question WritingQuestion Editing
Feedback
Launching Quizzes
Ass
ess
me
nt
Utility
ImpactStudents
Peer Instructors
7ExamSoft Assessment Conference | #ASSESS2015
• ExamSoft Infrastructure– Peer instruction requires separate folders
• Questions
• Assessments
• Course
Training FacultyExamSoft Infrastructure
Enrolling Peer Instructors
8ExamSoft Assessment Conference | #ASSESS2015
• Enrolling Peer Instructors– Peer Instructor “Faculty” Account
• E.g., [email protected]
– With limited permissions• Create exam items only• Only access tutoring courses question folder• Can only save, not approve, questions
Training FacultyExamSoft Infrastructure
Enrolling Peer Instructors
9ExamSoft Assessment Conference | #ASSESS2015
• Provide peer instructors with:– Detailed ExamSoft instructions– Resources on how to write good questions
with examples
• Considerations:– Have them provide rationales for all questions– Limit question types (e.g., essays)
Training StudentsQuestion Writing
ExamSoft
10ExamSoft Assessment Conference | #ASSESS2015
• This is a learning process for peer instructors too!
Implementation Question Writing
Question Editing
Feedback
11ExamSoft Assessment Conference | #ASSESS2015
• Faculty will ultimately be in control of:– Reviewing all questions– Assembling quizzes– Launching quizzes
• Aim for consistency – Across quizzes, courses, and with in-class exams
• Considerations:– Posting multiple versions or allow for multiple downloads– Enable secure quiz and review– Make passwords easily available (e.g., post on course
management site)– Ensure adequate time to access
Implementation Launching Quizzes
12ExamSoft Assessment Conference | #ASSESS2015
Implementation Launching Quizzes
13ExamSoft Assessment Conference | #ASSESS2015
• Data Available from ExamSoft– Number of exam takers– Aggregate performance on quizzes and
individual questions– Individual student performance on individual
questions– Category performance*
AssessmentUtility
Impact
14ExamSoft Assessment Conference | #ASSESS2015
Assessment Utility
Number per semester 9-12
# quizzes per exam 2-3
Average # of students taking each assessment (% of class)
1st attempt: 53.9 (67.1%)
2nd attempt: 24.2 (30.2%)
% taking at least one quiz 95%
Average # of quizzes taken per student 8.5
Average of 6 semesters worth of data in 5 different courses
15ExamSoft Assessment Conference | #ASSESS2015
• Comparisons using data mined from ExamSoft– Exam performance vs. quiz performance– Exam vs. quiz performance broken down by
course learning objective– Exam performance of quiz takers vs. non-quiz
takers
Assessment ImpactStudents
Peer Instructors
16ExamSoft Assessment Conference | #ASSESS2015
Assessment ImpactStudents
Peer Instructors
* p<0.05** p< 0.001
Example: Quiz vs. Exam Performance by Exam
By Learning OutcomeBy Exam
17ExamSoft Assessment Conference | #ASSESS2015
Assessment ImpactStudents
Peer Instructors
Student Perceptions of Tutoring Quizzes (n=37)
Question Average Score*
% SA/A
I feel that the tutoring quizzes are a valuable resource. 4.65 95%
I feel that taking the tutoring quizzes increased my confidence going into the exam.
4.43 92%
I feel that taking the tutoring quizzes increased my performance on the exam.
4.27 81%
I feel that the tutoring quizzes accurately reflected the material that was on the exams.
4.24 92%
I recommend continuing to offer tutoring quizzes in the future.
4.76 98%
Based on a Likert Scale of 1-5 (1= Strongly Disagree; 5= Strongly Agree); SA= Strongly Agree; A= Agree
18ExamSoft Assessment Conference | #ASSESS2015
Assessment ImpactStudents
Peer Instructors
Peer Tutor Perceptions of Tutoring Quizzes (n=8)
Question Average Score* % SA/A
Online tutoring quizzes were a valuable resource for the students I was instructing.
4.63 100%
Writing online tutoring quiz questions increased my knowledge of the course topics covered.
4.38 88%
Learning to write quiz questions was a valuable learning experience.
4.63 100%
Based on a Likert Scale of 1-5 (1= Strongly Disagree; 5= Strongly Agree); SA= Strongly Agree; A= Agree
19ExamSoft Assessment Conference | #ASSESS2015
Use of Self-Assessments in Pharmaceutical Science Courses
• Started with a pilot in 1 pharmacology course with 2 peer instructors
• Now implemented in 7 courses throughout the pharmaceutical sciences curriculum– 16 peer instructors involved
• All guided by same guidelines and using same resources and instructions
20ExamSoft Assessment Conference | #ASSESS2015
Conclusions
• Creating the infrastructure for self-assessments in ExamSoft is similar to that of in-class assessments, and only requires a few minor modifications.
• Feedback demonstrates that students perceive a benefit from self-assessments.
• Quiz and exam performance indicate that students benefitted from self-assessments.
• Peer instructors perceived a benefit from creating self-assessments.
• Faculty perceive a benefit and more courses are utilizing self-assessments and peer instructors.
21ExamSoft Assessment Conference | #ASSESS2015
References• Carroll M. Peer tutoring: can medical students teach biochemistry. Biochem
Educ. 1996; 24:13-15• Haist SA, Wilson JF, Fosson SE, Brigham NL. Are fourth-year medical students
effective teachers of the physical examination to first-year medical students. J Gen Intern Med. 1997; 12:177-181
• Haist SA, Wilson JF, Brigham NL, Fosson SE, Blue AV. Comparing fourth-year medical students with faculty in the teaching of physical examination skills to first year students. Acad Med 1998; 3: 190-200
• Nevid JS, Mahon K. Mastery quizzing as a signaling device to cue attention to lecture material. Teach Psychol. 2009; 36(1): 29-32
• Nicol DJ, Facfarlane-Dick D. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Stud High Educ. 2006; 31(2): 199-218
• Santee J and Garavalia L. Peer tutoring programs in health professions schools. AJPE 2006; 70: Article 70
• Stewart D, Panus P, Hagemeier N, Thigpen J, Brooks L. Pharmacy student self-testing as a predictor of examination performance. AJPE 2014; 78(2): Article 32
• West C and Sadoski M. Do study strategies predict academic performance in medical school? Med Educ. 2011; 45(7): 696-703
22ExamSoft Assessment Conference | #ASSESS2015
THANK YOU!
CONTACT US: Mindy Lull: [email protected]
Jennifer Mathews: [email protected] Marie Souza: [email protected]