Mid-Term Evaluation Course and Instructor evaluation Jennie Dorman.
Course Evaluation
description
Transcript of Course Evaluation
www.sakaiproject.org 1
Course Evaluation
Amitava ‘Babi’ Mitra, MIT
Maneesha Aggarwal & Robert Cartolano,
Columbia University
William Plymale and Aaron Zeckoski, Virginia Tech
www.sakaiproject.org 2
Agenda
• Introduction
• Review Common Ground
• Features at Columbia and Virginia Tech
• Next Steps
• Q & A
www.sakaiproject.org 3
Introduction• How did this begin ?
– Late Feb 05 began exploring who was using/planning to use online course eval within SEPP
– March 05, SEPP Course Eval Working Group set up
• What’s the WG been doing since then ?– Decided to use Columbia's existing feature set as a baseline to start with– CU and VT prepared a detailed functional layout and work flow,
and posted it in April so that we can clearly show how the existing systems might operate. MIT, e.g., has studied this and responded with their own needs
– CU and VT have made sandboxes available
• Why are we doing this ?– Getting to the common ground - the baseline that most of us can agree
on
www.sakaiproject.org 4
What is Course Evaluation?• Administration of survey to students across one or more
courses– May span course, department, school, university– Hierarchy, permissions, access levels are important
• Provide results to University, School, Department, Faculty, Students to evaluate:– Instructors– Curriculum– Academic Program (Program Evaluation)– Department, School, University (accreditation)
www.sakaiproject.org 5
Course Evaluation
• Evaluation has historical aspects at each school• There are different motivations for evaluation• System must be flexible enough to deal with
different needs, but…• There are common features that we have
identified among our working group members.
www.sakaiproject.org 6
Common Features --- from the WG’s discussions• Anonymity - student privacy must be preserved• Evaluation administrator - someone with appropriate
authority must create and administer evaluation.• Course Selection - must be very flexible
– As little as 1 course…– Or the entire department, program, school, etc.
www.sakaiproject.org 7
More Common Features --- from the WG’s discussions
• Ability to combine core and specific questions– Course-specific, instructor-specific, etc.
• Requires extensive reporting capabilities– Administrator, Instructors, Students, Public accessible
• Requires workflow and automation to make it easy, reduce burden to administrators.
www.sakaiproject.org 8
Columbia - Current System– In production since Fall 2003– Built into CourseWorks, campus CMS– Flexible approach needed to support multiple
schools, multiple departments, etc.• Carrot vs. Stick• Open vs. Closed reports
• Many Reporting types; built-in reports and export data
www.sakaiproject.org 9
CU Current System (cont’d.) – E-mail tool very important; both auto and
manual– Aggregate data across semesters to support
accreditation review– Very popular, well-received by faculty,
students, and administrators• 87,825 evaluations administered to date• 54,480 evaluations completed• 157 evaluation templates created
www.sakaiproject.org 10
Columbia Live Demo
www.sakaiproject.org 11
--- Current System
www.sakaiproject.org 12
--- Current System
• Implemented - 1981• Different forms for different colleges
– Business, Architecture, Engineering
• Statistics– 500 class sections per term– 230,000 sheets per year
• Response Rate: 78% during past 3 years
www.sakaiproject.org 13
SEPP Contribution – VT / CU• Columbia University Partnership
– SEPP Conference – Summer, 2004 – Denver– Robert Cartolano / Maneesha Aggarwal
• Virginia Tech Development– Aaron Zeckoski– Kapil Ahuja / Justin Gawrilow
• SEPP Course Eval Discussion Group• VT / CU Evaluation System Summary• VT Evaluation System – “sandbox” release
– https://courseeval-dev.cc.vt.edu
www.sakaiproject.org 14
SEPP Contribution – VT / CU• Virginia Tech extensions
– Administrative hierarchy (Super, University, School, Dept, Instructor)
– Dynamic reporting (flexible formatting, trend analysis, sub-setting)
– Ability to incorporate existing data collected with VT’s current evaluation system.
– Oracle database tuning
www.sakaiproject.org 15
--- Pilot System
• VT Pilot release – May 2005– Columbia System with VT extensions – Very limited response to VT pilot study due to late
release.– VT pilot study will resume Fall 2005
www.sakaiproject.org 16
--- Pilot System• Faculty training will be incorporated into FDI
• Change to online evaluation system needs approval by Deans and Department Heads
• Online system will provide better access for Institutional Research and the administration
• Online system offers faculty flexibility in adding questions to their evaluations
www.sakaiproject.org 17
--- Demo
www.sakaiproject.org 18
--- Current scenario• Institutional subject evaluation evaluates between 650-700 subjects each
term, about 2/3 of which are undergraduate subjects and 1/3 graduate.• This is out of approximately 1700 subjects with students enrolled and 3000
total subjects offered for credit each term.• Evaluations are end-of-term only, on 2 paper forms separately for
Science/Engineering subjects and for Humanities, Arts, and Social Science subjects, as chosen by the department. Quantitative data are captured and analyzed, written comments are available to each department from originals.
• Three departments, Electrical Engineering and Computer Science, Management, and Aeronautics and Astronautics (online) have their own forms and systems for subject evaluations.
www.sakaiproject.org 19
Needs --- Must havesMust haves for a new system (to replicate current system):• Stand alone system (with the capability of being tied to the Course Management
System)• Anonymity, confidentiality, security• Overall institutional administrator, plus levels of school/departmental administrators• At least 2 different forms (1 for Science/Engineering subjects and 1 for Humanities, Arts,
and Social Sciences subjects) with preference for many• Some questions consistent across all schools with flexibility within school/dept./subject.
Flexible questions would need to be administered at the local level but available to all.• Rating both subject and instructors on the same form• Access to individualized reports for each person evaluated • Assigning roles to instructors (lecturer, recitation instructor, etc.)• Integration with other MIT systems for data downloads (student, subject, and instructor
information) and appropriate access (e.g., only to students registered in subject)• A 7 point rating scale, with flexibility to change that if needed• Blocking access to data/reports until after grades are turned in• Analysis and reporting at a centralized point• Student access to appropriate report information through a centralized site
www.sakaiproject.org 20
Needs --- NecessaryNecessary in any new system:• Searching data/reports on instructor name• Assigning rank/status to instructors (prof./assoc. prof./assist.
prof./lecturer/grad. student, etc.)• Individual and other reports that include overall subject, department, school,
and institutional data• Longitudinal data/reports• Analysis and reporting at local points in addition to centralized one, with
capability to print subject/department/school level with 1 click• Linking to evaluation results by department and by subject• Linking to evaluation results from subject listings/selection (i.e., our on-line
catalog)* Integration with course management system with necessary security• Having students go to a single site for all their subjects to be evaluated
www.sakaiproject.org 21
Needs --- Nice to haveNice to have capability for:• Evaluating sections of a subject (with the capacity for
students to enter instructor's name)• Instructors' photos together with their name• Mid-term evaluations• Evaluations that only faculty would have access to• Capturing, editing, and distributing students'
comments• Having a single email sent to a student listing all
subjects to be evaluated by that student
www.sakaiproject.org 22
What’s Next• Create a functional specification for ‘Course Eval
ver 1.0’ built upon Columbia base and Virginia Tech extensions
• Develop Java-based version based on Sakai framework
• Run as standalone or within Sakai• Consider using SAMigo as assessment engine,
but probably not in version Course Eval ver 1.0
www.sakaiproject.org 23
Open Discussion
• Questions?
• See Demonstrations at 5:30 today
• Join Evaluation Working Group– Contacts:
• Robert Cartolano [email protected]
• Amitava ‘Babi’ Mitra [email protected]