KwantlenUNIVERSITY COLLEGE
1 Introduction of New Members
2 Agenda confirmation
3 Minutes of January 22 meeting
4 Business Arising from Minutes
5 ChairsReport
6 Discussion and Development of Program Review Indicators
7 Additional Items
1
ii
m
8 Next Meeting
9 Adjournment
PROGRAM EVALUATION COMMITTEEMEETING
Feb 19 2003 215 415 pm Room B 206Feb 20 2003 130 430 pm Room G 2110
Agenda February 19
Agenda February 20
1 Agenda confirmation
2 utes ofFeb 19 meeting
3 Finalire PRC Member
4 PRC Statement of Purpose draft
5 Continue Discussion and Development ofProgram Review Indicators Database
6 Additional Items
i
ii
111
7 Next Meeting Time and Agenda refer to List of Ongoing Issues8 Adjournment
1
NAME PRESENT ABSENT
Androsiuk Maryf o 1
Brooks Ianf
Chambers Susan
Cheema Chaumkaur jDavis Bob
Dunbar Sophie
Goedbloed Dana
Hamilton Kevin
MacNamara Deb
McGillivray JudithMetzger Karen
Mott Maxine
Nanson Derek
Penhorwood Jan
Richard Paul9
Robertson CarolynSchultz Dianne
Tritchew Panteli
rS t
GUESTS
KwantlenUNIVERSITY COLLEGE
Program Review Committee
Attendance Date
NAME PRESENT ABSENT
Androsiuk Mary r
Brooks Ian
Chambers Susan
Cheema Chaumkaur
Davis Bob
Dunbar Sophie
GoedbloedDana
Hamilton Kevin
MacNamara Deb
McGillivray Judith
Metzger Karen
Mott Maxine
Nanson Derek
Penhorwood Jan
Richard Paul
Robertson CarolynSchultz Dianne
Tritchew Panteli
cWJJj 744Lccx fsSZ
GUESTS
KwantlenUNIVERSITY COLLEGE
Attendance Date 20
Program Review Committee
KwantlenUNIVERSITY COLLEGE
Present
Regrets MrAtithoslukc5ue Chambers
Chaumkaur CheemaDeborah MacNamara
Judith McGillivrayJan Penhorwood
Guests Arthur Coren
4 Business Arising from MinutesNone
PROGRAM REVIEW COMMITTEE MEETING
January 22 2003 215 pmSurrey Campus Boardroom
MINUTES
Ian Brooks
Bob Davis
Sophie DunbarDana Goedbloed
Kevin Hamilton
Sooz Klinkhamer altemateMaxine Molt
Derek Nanson
Paul Richard
Carolyn RobertsonDianne Schultz
Panteli Tritchew
1 The Chair called the meeting to order and introduced the new members
2 Agenda ConfirmationMoved by Kevin Hamilton seconded by Derek Nanson to approve the agenda
3 Approval of Minutes of Meeting Dec 10 2002The minutes of the Dec 10 2002 meeting were approved with the followingamendment Paul Richard had been in attendance
5 ChairsReportACTION Karen to send memo re location for upcoming meetings
The Chair reported that he is in the process of establishing representation on thecommittee from Admissions from among program assistants or deans areas andlabs
Program Review CommitteeMinutes of Meeting Jan 22 2003Page 2
The Chair announced that he has contacted students in a 3year psychology courseon Program Evaluation PSYC 3960 to see if they may be interested in joining thecommittee
a Education Council UpdateEducation Council approved the terms of reference and purpose of the committeeMembership for faculty representation from Library Counselling and CoopEducation still needs clarification
b Communication with Stakeholders
The Chair met with chairs of social sciences departments this monthwho agreedto talk with their faculty members about program advisory committees theytypically rely on articulation for external approval The Chair plans to meet withTrades and Community Health coordinators
The committee explored the idea of setting an additional meeting during readingweek There was general agreement to meet Thursday Feb 20 from 130430pm
Timelines We plan to identify sample indicators in February and March thenfinalize Kwantlensversion of indicators The next step is to identify coreindicators for first and second level reviews
6 PRC Terms of ReferenceMoved by Dana Goedbloed seconded by Derek Nanson to table item to the nextmeeting
MOTION CARRIED
7 Sample Program Review from Office of Institutional ReviewOffice of Institution Research presented a sample or pilot Program Review usingBusiness Management Diploma applicants for the 1999 year as a trial program
a OIR used their internal Data Warehouse as a source of data and ran a sampleLevel One review using the following indicatorsi Number of Program Applicants
ii Number of Accepted Applicants
iii Number of Applicants Who Registered at KUC in any course
iv Number of Applicants Who Registered in a program specific course
v Number of Credentials awarded to the 1999 cohort to date
b The OIR pilot review highlighted a number of challenges for KUCTheressteady attrition from of students who apply to of students who areaccepted to of students who actually register We dont know how manystudents are accepted but dont register because they cant get seats
The study showed that a very low percentage of students who entered the programgraduated within two years but program attrition may not be an entirely usefulindicator for openaccess nonlockstep programs at KUC
Program Review CommitteeMinutes of Meeting Jan 22 2003Page 3
It is difficult to assess how long it takes for program students to graduate because
vi students do not need to declare their intended major when they register atKUC or each term that they register
vii students do not receive a credential unless they apply for it and somestudents who finish their 60credit credential and go on to degree studiesdont bother
viii according to OIR Banner credential data is only activated ifwhen studentsapply so Banner might show an undercount if we tried to query how manystudents simply completed the credential and didnt apply to graduate
ix students can take their studies part time and have a very successfulKwantlen experience but may take many semesters to complete theirstudies and
x students can change the program or their major partway through theprogram and end up over achieving on the number of credits they earn for aparticular credential
c Challenges identified from OIR presentation to PRCi technical challenges with Banner both in terms of what kind of data is
available and the extractability of the data PEC will be inviting someone fromthe registrarsoffice to discuss this once weve got a good idea of what weneed
ii cohort tracking challenges currently students dont need to declare theirmajors either upon registration or on a termbyterm basis
iii Perception challenges in terms of attrition as an indicator since students mayhave an eminently successful experience at KUC but simply choose to studyparttime and take several years to complete their degree
Core indicator and benchmark challenges how to develop common indicators andbenchmarks or if its possible to develop common indicators and benchmarks forboth openaccess and selectiveentry programs
Identify what information will be required and request changes to Banner toaccommodate that or other software adjustments as required in order to accomplishthe goals of meaningful program reviews May need to find new ways to collect therequired data
Identify what it means for students to have a successful experience and at the sametime fulfill Ministry requirements and meet Kwantlensneeds
Students may register in openaccess program courses not knowing what their finalgoals are and may switch to completely different area before graduating or leaving
Question Can BANNER keep track of no of students included in the attrition poolwho wanted to register but couldntbecause of lack of availabilityActivity reports can be run through Admissions to show the number of people whoapplied for a program and how many times they attempted it and who dropped the
Program Review CommitteeMinutes of Meeting Jan 22 2003Page 4
program For example 300 to 500 students may have attempted to register 1500 timesamong themBanner may be able to identify students who are eligible for a credential though theydont apply for the credential
8 PEC Statement of Purpose DraftTabled to next meeting
9 Additional ItemsNone
10 Next Meeting Wed Feb 19 2003 Room B206Additional Meeting Thurs Feb 20 130 430 pm Surrey Boardroom G 2110
11 Motion to adjourn by Maxine Mott seconded by Sooz Klinkhamer at 415 pm
PT EDCO REPORTS 1FOR FEBRUARY 03 MEETING OF EDCO
1 Program Review Committee met on Jan 22 03
2 Office of Institution Research presented a sample or pilot Program Review usingBusiness Management Diploma applicants for the 1999 year as a trial program
3 OIR used their internal Data Warehouse as a source of data and ran a sample LevelOne review using the following indicators1 Number of Program Applicants2 Number of Accepted Applicants3 Number of Applicants Who Registered at KUC in any course4 Number of Applicants Who Registered in a program specific course5 Number of Credentials awarded to the 1999 cohort to date
4 The OIR pilot review highlighted a number of challenges for KUC
Theressteady attrition from of students who apply to of students who areaccepted to of students who actually register We dont know how many studentsare accepted but dont register because they cant get seats
The study showed a very low age of students who entered the program graduatedwithin two years but program attrition may not be an entirely useful indicator foropen access non lockstep programs at KUC
It is difficult to assess how long it takes for program students to graduatebecause
a students do not need to declare their intended major when they register at KUCor each term that they register
b students do not receive a credential unless they apply for it and some studentswho finish their 60credit credential and go on to degree studies dont bother
c According to OIR Banner credential data is only activated ifwhen studentsapply so Banner might show an undercount if we tried to query how manystudents simply completed the credential and didnt apply to graduate
d students can take their studies part time and have a very successful Kwantlenexperience but still take a longtime to complete their studies
e students can change the program or their major partway through the program andend up over achieving on the number of credits they eam for a particularcredential
Challenges identified from OIR presentation to PRCa technical challenges with Banner both in terms of what kind of data is available
and the extractability of the data PEC will be inviting someone from theregistrarsoffice to discuss this once weve got a good idea of what we need
PT EDCO REPORTS 1b cohort tracking challenges as long as students dont need to declare their
majors either upon registration or on a termbyterm basisc Perception challenges in terms of attrition as an indicator since students may
have an eminently successful experience at KUC but simply choose to study parttime and take several years to complete their degree
d Core indicator and benchmark challenges how to develop commonindicators and benchmarks or if its possible to develop common indicators andbenchmarks for both openaccess and selective entry programs
The PRC will be looking at that these challenges in the next few months
PROGRAM EVALUATION INDICATORS WORKSHEET
INTRODUCTION TO INDICATORS SELECTION PROCESS
This worksheet contains a list database of indicators that could potentially be used forProgram Reviews at Kwantlen University College
Over the course of several meetings well use this worksheet to filter out and selectLevel One Core Indicators
Stage One Clarification and Comment
Review possible indicators clarify meaning add additional indicators state concerns andcollate comments Please jot down any comments that are meaningful to you not allcomments will necessarily be recorded in the minutes
Homework Solicit feedback from constituents as per your own comfort level andnumerically rank indicators prior to Stage Two
Stage Two Collation of Indicators Ranking
Roundtable numerical ranking of the Indicators
Members not able to attend can submit their rankings electronically or by hardcopy
The chair will collate and tabulate the results prior to Stage Three and sources for datawill be identified with the Office of Institutional Research
Stage Three Review of Ranking
The indicators as numerically ranked by the committee will be reviewed
Motions for approval and selection of 1st Level Review Core Indicators number to bedetermined debate and vote on motions
Developedfor PRC Meetings ofebsmy19 and2C1 1
PROGRAM EVALUATION INDICATORS WORKSHEET
DcvefoprdforPIT 9Katingsof February19and206 2
INDICATOR COMMENTS word as you see tit RANKING
1 Courses Success Rate
comparative assessment withother institutions
2 Courses Success Rate
comparative assessment withother programs
3 Program and course revisionscontinuous evaluation cycleefficient and current
4 Program Withdrawal Rate
5 Course Withdrawal Rate
6 Program Attrition Rate
7 Course Attrition Rate
8 Course Failure Rate
9 Graduation Rate comparativeassessment with other
institutions
PROGRAM EVALUATION INDICATORS WORKSHEET
DcvefoprdforPIT 9Katingsof February19and206 2
PROGRAM EVALUATION 1 DICATORS WORKSHEET
Developedfor PACMutingr ofFebruary 19 and 20ie 3
INDICATOR COMMENTS record as you see fit RANKING
10 Graduation Rate comparativeassessment with other programs
11 Graduation Speed
12 Admission Rate to followingstudy levels
13 Instructor availability
14 Program objectives relevanceto labour market demand jobs
15 Program objectives relevanceto society community needs
16 Program duplication withnearby institutions
17 Program prerequisites
18 Program vs courseprerequisites
19 Program breadth liberaleducation
PROGRAM EVALUATION 1 DICATORS WORKSHEET
Developedfor PACMutingr ofFebruary 19 and 20ie 3
PROGRAM EVALUATION ICATORS WORKSHEET
DevelopedforPlFMeeting oflebruary 19 aid20 4
INDICATOR COMMENTS record asyouseefit RANKING
20 Links between teaching modesor activities classes labstutorials online etc with
program and course objectives21 Support staff availability lab
assistants tutors etc22 Support staff professional
development and currency23 Program recognition by
industry24 Program recognition by
accrediting bodies Programaccreditation
25 Links between teaching modesor activities classes labstutorials online etc withstudent demographicscharacteristics
26 Links between teaching modesor activities classes labstutorials online etc withprogram and course objectives
27 Student diagnostics preentryin progress exit exams etc
PROGRAM EVALUATION ICATORS WORKSHEET
DevelopedforPlFMeeting oflebruary 19 aid20 4
PROGRAM EVALUATION INDICATORS WORKSHEET
DevelopedforI Meetingsoftbruary19 and 20 5
INDICATOR COMMENTS record as you see fit RANKING
28 Student selection entrancecriteria
29 Instructor evaluations
30 Instructor availability
31 Instructor professionaldevelopment and currency
32 Library resources as perprogram needs
33 Laboratory resources spaceequipment material as perprogram needs
34 Classroom availability spaceand appropriateness for programneeds
35 Computer availability andappropriateness for programneeds
36 Course outlines updated
37
PROGRAM EVALUATION INDICATORS WORKSHEET
DevelopedforI Meetingsoftbruary19 and 20 5
PROGRAM EVALUATION I ICATORS WORKSHEET
38
39
40
41
42
43
44
45
46
47
48
INDICATOR COMMENTS record as you see fit RANKING
DevelopedforP4FMuting ofFebruary 19 and20s 6
PROGRAM EVALUATION INDICATORS WORKSHEET
INTRODUCTION TO INDICATORS SELECTION PROCESS
This worksheet contains a list database of indicators that could potentially be used forProgram Reviews at Kwantlen University College
Over the course of several meetings well use this worksheet to filter out and selectLevel One Core Indicators
Stage One Clarification and Comment
Review possible indicators clarify meaning add additional indicators state concerns andcollate comments Please jot down any comments that are meaningful to you not allcomments will necessarily be recorded in the minutes
Homework Solicit feedback from constituents as per your own comfort level andnumerically rank indicators prior to Stage Two
Stage Two Collation of Indicators Ranking
Roundtable numerical ranking of the Indicators
Members not able to attend can submit their rankings electronically or by hardcopy
The chair will collate and tabulate the results prior to Stage Three and sources for datawill be identified with the Office of Institutional Research
Stage Three Review of Ranking
The indicators as numerically ranked by the committee will be reviewed
Motions for approval and selection of 1st Level Review Core Indicators number to bedetermined debate and vote on motions
Developerffor2ltC Meetings ofFebruary 19 and20 1
PROGRAM EVALUATIONOICATORS WORKSHEET
Devefoytdfor fllMeetings ofFebruary 19 ant 2L1h 2
INDICATOR COMMENTS ordas you seefit RANKING1 Courses Success Rate
comparative assessment withother institutions
2 Courses Success Rate
comparative assessment withother programs
3 Program and course revisionscontinuous evaluation cycleefficient and current
4 Program Withdrawal Rate
5 Course Withdrawal Rate
6 Program Attrition Rate
7 Course Attrition Rate
8 Course Failure Rate
9 Graduation Rate comparativeassessment with other
institutions
PROGRAM EVALUATIONOICATORS WORKSHEET
Devefoytdfor fllMeetings ofFebruary 19 ant 2L1h 2
PROGRAM EVALUATION IRDICATORS WORKSHEET
DeUdopulfor TrAg Meetings ofFebruary 19 and20 3
J
INDICATOR COMMENTS record as you see fit RANKING
10 Graduation Rate comparativeassessment with other programs
11 Graduation Speed
12 Admission Rate to followingstudy levels
13 Instructor availability
14 Program objectives relevanceto labour market demand jobs
15 Program objectives relevanceto society community needs
16 Program duplication withnearby institutions
17 Program prerequisites
18 Program vs courseprerequisites
19 Program breadth liberaleducation
PROGRAM EVALUATION IRDICATORS WORKSHEET
DeUdopulfor TrAg Meetings ofFebruary 19 and20 3
J
PROGRAM EVALUATION INDICATORS WORKSHEET
Devdopr1jorPlFMeetings aJjebruary and 20 4
INDICATOR COMMENTS record as you see fit RANKING
20 Links between teaching modesor activities classes labstutorials online etc withprogram and course objectives
21 Support staff availability labassistants tutors etc
22 Support staff professionaldevelopment and currency
23 Program recognition byindustry
24 Program recognition byaccrediting bodies Programaccreditation
25 Links between teaching modesor activities classes Tabstutorials online etc withstudent demographicscharacteristics
26 Links between teaching modesor activities classes labstutorials online etc withprogram and course objectives
27 Student diagnostics preentryin progress exit exams etc
PROGRAM EVALUATION INDICATORS WORKSHEET
Devdopr1jorPlFMeetings aJjebruary and 20 4
PROGRAMEVALUATION INDICATORS WORKSHEET
Devefopdfor Mg Meetings ofFebruary 19 and2YF 5
J
INDICATOR COMMENTS record as you see fit RANKING
28 Student selection entrancecriteria
29 Instructor evaluations
30 Instructor availability
31 Instructor professionaldevelopment and currency
32 Library resources as perprogram needs
33 Laboratory resources spaceequipment material as perprogram needs
34 Classroom availability spaceand appropriateness for programneeds
35 Computer availability andappropriateness for programneeds
36 Course outlines updated
37
PROGRAMEVALUATION INDICATORS WORKSHEET
Devefopdfor Mg Meetings ofFebruary 19 and2YF 5
J
PROGRAM EVALUATION INDICATORS WORKSHEET
38
39
40
41
42
43
44
45
46
47
48
INDICATOR COMMENTS record as you see fit RANKING
DevelopedforP1JMeetings ofFebruary 19 au 202 6
TERMS OF REFERENCE FOR PROGRAM REVIEW COMMITTEE
Mandate
The Program Review Committee is a sub committee of the Education Council Thecommitteesmandate is to assist Kwantlen University College to
perform regular and consistent reviews of Educational Programs and Offerings thatare under the jurisdiction of Education Council
adopt a two tiered evaluation process
develop policies and procedures to coordinate and monitor program and educationaloffering evaluations
PanttTritncewChair Program Evaluation Committee 2
TERMS OF REFERENCE FOR PROGRAM REVIEW COMMITTEE
Functions
The functions of the Program Review Committee are to
1 develop guidelines and procedures for the regular and consistent review ofeducational programs and offerings that are under the jurisdiction of EducationCouncil at Kwantlen University College
2 develop indicators to be used in program and educational offering reviews atKwantlen University College
3 develop the process and principles by which benchmarks could be used in programand educational offering reviews at Kwantlen University College
4 develop a scheduling procedure for the regular review of educational programs andofferings programs at Kwantlen University College
5 develop procedures to document the results and recommendations of programreviews To Education Council
6 ensure that program and educational offering reviews at Kwantlen University Collegeare consistent with and meet the requirements of Education Council
7 ensure that program and educational offering reviews at Kwantlen University Collegeare consistent with and meet the requirements of the Ministry of AdvancedEducation
8 periodically review the committeesTerms of Reference and make revisions andrecommendations as it deems necessary or as directed by Education Council
9 periodically review KwantlensProgram Review Policy and make revisions andrecommendations as it deems necessary or as directed by Education Council
10 distribute the minutes reports and recommendations of the committee to variouscollege bodies and stakeholders as appropriate
PanuiritrlewChair Program Evaluation Committee 3
TERMS OF REFERENCE FOR PROGRAM REVIEW COMMITTEE
1 one faculty member to be selected by eacthofrhe University Co eges divisions andone faculty member to be selected tom each o 1r3 ary CipSJ ing andCooperative Education 1
2 two deans to be selecteffb the Vice Pr ident ring and Provost
3 one member of admutistrat appointed m and by the Office of the Registrar
4 staff membersV A
5 students3
6 the Vice President Claming and Provost andL
7 the chair or vic chair of Education Council ex officioSelection of Chair
The chair of the committee shall be a faculty member selected for a twoyear termnormally beginning in September The vice chair shall be nominated by and from withinthe committee
Membership
The composition of the PRC shall consist of
Quorum and Voting
A quorum of the committee shall consist of 50 percent plus one of the voting members
All members of the committee shall be voting members
When a member is unable to attend a meeting a designated alternate may attend themeeting to speak on their behalf with the same voting rights as the regular member
PanuSTriuhew Chair Program Evaluation Committee 4
Purpose of Program Review at Kwantlen University College
KUC REQUIREMENTS
Currently Kwantlen University College has a Program Review Policy in place Policy B12This policy lists the following objective for Program Review at Kwantlen University College
Source apIptazakwantlenbccafPrimary Objectives of Program Review
1 to review the programsrelevance to the communities Kwantlen serves
2 to ensure the programsconsistency with the university collegesmission goals policyphilosophy standards and plans
3 to determine external and intemal satisfaction with the programsstructure content anddelivery and reasons for dissatisfaction if any
4 to project the level of student demand for the program for the next five years
5 to determine the strengths and weaknesses of the program and
6 to identify specific issues or concerns regarding achievement of the programsgoals andobjectives
Other review objectives may be added as part of any given program review
The following page uses Kwantlenscurrent policy as a starting point to develop a Statement ofPurpose for Program Evaluation at Kwantlen University College
Purpose of Program Review at Kwantlen University College
The purpose of a program review at Kwantlen University College shall be primarily formative
Program reviews shall be conducted in order for Kwantlen to
I meet reporting and accountability requirements of the Ministry of Advanced Education
2 meet reporting and accountability requirements of KwantlensEducation Council and Boardof Govemors
3 review the programsrelevance to the communities that Kwantlen serves
4 ensure that programs are consistent with the university collegesmission goalsphilosophy standards and plans
5 determine external and internal satisfaction with the programss cture content anddelivery and reasons for dissatisfaction if any
6 project the level of student demand for the program
7 determine the strengths and weaknesses of the program
8 identify specific issues or conce a arding achievement ogd s goals andobjectives
9 review student achievement usin
10 identify areas for improvement
I 1 set specific tar e for improvement
12 describe yr addressing tareking performancean ting targets
to be used in striegic planning
13 establish
14 provide progr
15 develop plans forcture
ongoing prog
d local perfomianc e
hortterm or longterms revisions to program content or
Ives may be added as part of any given program review
LIST OF ONGOING ISSUES FOR PROGRAM REVIEW COMM JANUARY 22
How do we define the relationship between formative and summative
What is the relationship between program planning program evaluation and budgeting
How will program evaluation reports be disseminated Eg what parts will go to differentconstituents including students
What do we do with results of program review reports
How do we link program review process with institutional values
What kind of process body or mechanism needs to be inplace at KUC to coordinate track andmonitor the implementation of recommendations that arise from either regular or indepth ProgramReviews
How do we align review with accreditation
How do we fit input from Program Advisory Committees into Evaluation process
How do we fit input from Program Accreditation into the Evaluation process
How do we define benchmarks
Should we develop benchmarks
How do we develop benchmarks
How long after a program is implanted should it be reviewed
How do we develop the Program Evaluation Rotation SequenceSchedule
How often is the indepth review feasible
Should the PRC develop different evaluation frameworks for each different credential as well asdifferent evaluation frameworks for the twotiers
Of the 129 different educational offerings some of them are duplicated as programs for exampleAccounting offers credentials and courses that serve other programs On the other hand English offerscourses that serve other credential programs but does not offer its own credential Should the nonprogram offerings Typically UniversityTransfer departments have an evaluation frameworkthat is different than credential programs
How do we ensure sufficient stakeholder input to ensure ownership and participation
How do we develop a culture for evaluation at KUC
Given that we have multiple models for review how do we customize our review process coreindicators for all programs plus unique indicators for individual programs
Can we open up access to SORS data there may be a need for prod sessions
What kinds of resources will KUC be able to commit to ongoing program evaluation CET supportfinancial staff etc
How do we create incentives program evaluation participation
Pantel Tritckew Chair Program Evaluation Committee 1
Obtain an SLT commitment to create a separate fund budget line item for PRC so that programs canuse the results of their evaluation to apply for funding specific improvements incentive forevaluation
LIST OF ONGOING SUGGESTIONS FOR PROGRAM REVIEW COMMITTEE
Top Related