Reviewing Course Evaluation Data: Perception vs reality

Peter D Munn and

Sheila D Scutter

Introduction

• Graduate Course Evaluation Questionnaire (GCEQ) provided to all graduates Australia wide– Comprises 25 Likert scale statements and

two free response items– Six key areas of graduates’ experience

• Clear goals and standards• Appropriate workload• Appropriate assessment• Good teaching• Generic skills• Overall satisfaction

Introduction

• University of South Australia uses good teaching, generic skills and overall satisfaction as key performance indicators of perceived teaching quality

• Areas of concern with Nursing program GCEQ scores.– Assumed to be due to course content – A comprehensive review process to

understand specific areas of concern in nursing.

Nursing program

• Three year undergraduate program• Offered in 2 city and 1 regional campus• Offered in internal and external mode• Approximately 1000 students • Many part time students• Access via Year 12, STAT, bridging

programs• Special entry test for ATSI students• Very high participation of equity

groups.

Method• Graduate Course Evaluation

Questionnaire(GCEQ)– GCEQ data for the nursing programs

across the city and regional campuses were analysed by external and internal mode of study

– Thematic analysis on comments on the ‘best aspects’ and ‘areas most in need of improvement’ responses analysed by mode of study

– Individual item scores reviewed

• Student experience questionnaire (SEQ)– Distributed online to all currently enrolled

students– Feedback on aspects of academic life,

resources and services– Sixteen Likert scale items and two open

response items concerning course and program quality

– Responses compared by mode of study– Thematic analysis of comments by mode

of study

• Course Evaluation Instrument– Each course in the University is evaluated

every time it is offered– Course Evaluation Questionaire (CEI),

online instrument developed by the University

– CEI contains 10 core Likert-scale questions concerning course quality

– Additional items may be added, this analysis concentrated on the 10 core items

• Focus Groups with StudentsThree focus groups were conducted:– Two with interstate, rural and metropolitan

South Australian students– One with indigenous students and their

learning support coordinator

• Focus Groups with Staff– Eight staff representing both campuses

Results

GCEQ Scores, Good Teaching

ResultsGCEQ data• Concerns over good teaching, overall

satisfaction and generic skills;

• Low response rates may be a source of bias

ResultsStudent Evaluation Questionnaire• Responses to most statements were positive

with a mean score over 4 (maximum 5); • Lower scores received for items relating to:

– Support provided by teaching staff;– Timeframe for return of assignments;– Knowing what is expected in assignments;– Consistency in marking.

ResultsCEI data• Limited response by students made

interpretation of data difficult;• Feedback on courses ranged from very good

to very poor;• Main areas of concern:

– I felt there was a genuine interest in my learning needs and progress;

– The workload for this course was reasonable given my other study commitments;

– I have received feedback that is constructive and helpful.

Focus Groups: GCEQ/SEQ analysis• Course Materials and Content

– Little comment– Valued flexibility– Often arrived late– External students felt “second best– Revised courses in 2004 to update content

• Communication between students and academic staff– A major concern of external students– Interpreted as lack of interest by academics– External students felt disadvantaged compared

to internal students.– Staff concerns about students accessing email– Geographical support groups disestablished

• Learning Support Service Availability– Requirements for assignment writing– Particular concern for indigenous students– Available online but prefer alternative delivery– Need for support/study groups– Difficulty accessing learning advisers.

• Teaching and Learning Issues– Time not spent productively in workshops– Clinical experiences very positive– Staff/student ratio low– Lack of training for staff in external delivery– Student preparation for the online environment– High percentage for exams in science based courses

• Assignments and Assessment– Consistency of requirements within courses– Turnaround times major concern– Clarity of expectations.– Large number of students affecting

turnaround

Discussion• GCEQ scores indicated concerns about overall

satisfaction, good teaching and generic skills– The instrument– Response rates– Timing– Interpreting the data

• Students are not given the opportunity to provide many areas of this feedback in GCEQ.

• Where is was possible to provide comment in the GCEQ, it was not reflected in item scores.

DiscussionKey concerns of students identified from

thematic analysis and focus groups:

• Timely, consistent and useful feedback;• Communication with academic staff;• Learning support;• Practical workshops/professional placements• Lack of preparation for study

RecommendationsAgreed practice model to be implemented

– Provide up to date study guide– Early arrival of course materials– Fortnightly email communication – Respond to emails and calls in 24 hours– Contribute to discussion board weekly– Harvard reference material– Geographical location maps– Information packages for Maths and

English support

Agreed Practice model…Key points sheet for assignments and marking• Moderation of assignment marking• Staff development for external mode• Two week turnaournd for first assignment• Feedback before next assignment due• Promote course evaluation• Respond to student feedback

• Initiatives around GCEQ response rates and distribution.

• Extension of preparatory courses (Sciences)• Review nursing workshops• Review assessment

Conclusion

• GCEQ scores alone do not provide direction for program improvement.

• Thorough analysis of course and program feedback is necessary to understand problems and to develop appropriate changes to programs.