Student evaluations of teaching: Perceptions determining teacher behaviours

Sarah Stein, Jo Kennedy, Trudy Harris, Stuart Terry, Lynley Deaker, Dorothy Spiller

Presentation at the Higher Education Research and Development Society of Australasia (HERDSA) conference, Hobart, 2012.

2

The project

Title: “Unlocking the Impact of Tertiary Teachers’ Perceptions of Student Evaluations of Teaching”

Ako Aotearoa (National Centre for Tertiary Teaching Excellence) National Project Fund grant ($150k)

2010 - 2011

3

Project partners

from left:top: Stuart, Sarah, Lynleyfront: Jo, Dorothy, Trudy

Otago Polytechnic

• Stuart Terry

University of Waikato

• Dorothy Spiller• Trudy Harris

University of Otago

• Sarah Stein• Jo Kennedy• Lynley Deaker

4

Issues academics’ hostility towards student evaluations

academics are resigned to the notion of student evaluations (Beran & Rokosh, 2009)

=/= improvements in teaching (Kember, Leung & Kwan, 2002) or serious engagement for development (Beran & Rokosh, 2009; Burden, 2008)

two (competing?) purposes for student evaluations:1. audit (monitoring, gauging teaching effectiveness)2. development of teaching and courses

claims that the two purposes are complementary (e.g., Ramsden, 1992)

BUT perceptions are very important

5

Issues interrelated factors (contextual, philosophical, practical and personal)

influence academics’ perceptions of evaluations and their use of them: validity and reliability are still cited, despite the evidence (e.g., Benton &

Cashin, 2012; Marsh, 1987; Theall & Franklin, 2001)

institutional expectations and community norms (e.g., Nasser & Fresko, 2002)

limitations of student judgement (Aleamoni, 1981)

quality of the institution’s evaluation/appraisal instruments (e.g., Ballantyne, Borthwick & Packer, 2000; Penny & Coe, 2004)

institutional ownership and use of evaluations (e.g., Edström, 2008; McKeachie, 1997; Nasser & Fresko, 2002)

individual teacher’s teaching beliefs (Hendry, Lyon & Henderson-Smart, 2007) and emotions (e.g., Moore & Kuol, 2005)

6

Issues

Perceptions do matter

“it doesn’t matter much what the institution’s intended purpose is [or what research evidence suggests]. What is important is what the individual teachers perceive to be

the purpose”

(Edström, 2008, p. 100, emphasis in original)

7

Research questions How do current formal student evaluation/appraisal

processes and practices influence teachers’ thinking and behaviours in relation to student learning at all stages of the teaching and learning cycle?

What are the perceptions that tertiary teachers hold about student evaluation/appraisal?

What factors (causes, influences) affect these views?

How do tertiary teachers engage with student evaluations/appraisals?

8

Design & Methods overarching interpretivist research approach

(Erikson, 1998)

combination of quantitative and qualitative data: online questionnaire semi-structured interviews

key ideas and issues identified through the literature review contributed to the development of the data gathering tools and to the analysis of the resulting data

9

Online questionnaire Likert-scale and open response questions

4 parts: Section A - explored current practices (Q1-8) Section B - explored perceptions of student

evaluation data and influence on practice (Q9-22) Section C - demographic information (Q23-32) Section D - interview availability (Q33)

2,426 teaching staff invited 1,065 responses received (44%)

10

Semi-structured interviews 20 from each institution, purposively selected (Patton, 1990)

from volunteers (Q33)

range of academic discipline, career stage, seniority

core interview questions based on the key themes identified in the questionnaire responses

teaching and learning beliefs students’ capacity to make judgements personal, emotional factors other factors e.g., timing, promotion, engagement with

evaluation

11

Analysis

Questionnaire comments & interviews thematic analysis - involved searching

for themes, using a constant comparative technique (Dye, Schatz, Rosenberg & Coleman, 2000; Silverman, 2001)

Likert scale questionnaire questions ANOVA, Kruskal–Wallis test

12

Findings: Questionnaire data

widespread recognition in the questionnaire that collecting evaluation/appraisal data was worthwhile

(Q17 Do you personally think it is worthwhile to gather student evaluation/appraisal data about teaching and courses/papers?)

OU WU OP0102030405060708090

100

freq

uenc

y of

res

pons

es

(%)

very worthwhile (1&2)

middle ground (3)

not at all worthwhile (4&5)

combined institutionsby institution

13

Findings: Questionnaire data

over half the respondents find their centralised system effective at gathering useful/meaningful data for them (1 or 2 rating). Sixteen percent, on average, found the centralised system not effective (4 or 5 rating)

(Q19 How effective is your institution’s centralised evaluation/appraisal system in gathering useful/meaningful student data for you?)

very ef -fective (1&2)

middle ground

(3)

not ef -fective (4&5)

OU WU OP0102030405060708090

100

freq

uenc

y of

re

spon

ses

(%)

combined institutions by institution

14

Findings: Questionnaire comment dataQ18 Please explain your answer to Q17 enhancing role of evaluations

[Evaluation] is great way to learn about what is good and what is bad - hard to be totally objective about your methods etc and students are great at honesty in this forum!

(Q18 sub theme 1b, OP, lecturer teaching position, 0-5 years’ tertiary teaching experience, permanent, Sciences)

limiting role of evaluations

What is not good in our system is the standard format which is unsuited to so many diverse courses.

(Q18 sub-theme 2c, WU, lecturer teaching position, 11-15 years’ tertiary teaching experience, continuing, Humanities)

I do not however, approve of the institution's tendency to use them [evaluations] as weapons against staff. A heavy-handed hierarchical approach from academics with little knowledge of the course or the students and even less interest in teaching or its context, is counter-productive.

(Q18 sub theme 2a, OU, lecturer teaching position, 16-20 years’ tertiary teaching experience, permanent, Humanities)

15

Findings: Questionnaire comment & interview data mistrust about students’ reliability

You should treat them [student evaluations] with a pinch of salt. Students don’t have a long term perspective. Personality is a big factor. (OU, interview)

I used to believe that [students can make judgements about teaching], but now I no longer believe that. I think in terms of how…students are believing they are buying a qualification..it’s more like purchasing their degree. (WU, interview)

Students have bullied staff and they use evaluations as an opportunity to dump on staff. (OP, interview)

preferences for other forms of student feedback

The rating questions are rather useless but perhaps useful for a promotion committee to make broad judgements. That is their sole value, nothing else. The reason for that is that they do not specifically tell you what is wrong or what is right. The comments do that best. Also, as mentioned earlier, the statistical rigour in many of these appraisals would make a real statistician seriously question their meaning.

(Q18 sub-theme 2a, WU, lecturer teaching position, 11-15 years’ tertiary teaching experience, continuing,

Sciences)

16

Questionnaire comment & interview data

suspicion of institutional use of student evaluations It is valuable, but only if taken in context! One negative comment and 100

positive ones is a very good result. However, management have a tendency to focus on that one comment. Often there are reasons other than the quality of the teaching for negative evaluations.

(Q18 sub-theme 2a, OP, senior teaching position, 6-10 years’ tertiary teaching experience, permanent, Health Sciences)

lack of faith in the process and instrument The institution tries to do too much with this limited data. (OU, interview)

The student evaluations instrument is unreliable, contrived and manipulated. (OU, interview)

suspicion of manipulation by colleagues People are more careful to choose questions that are more likely to yield a

positive response. (OU, interview)

17

Findings: Summary Teachers are generally positively disposed towards the student evaluations,

although not well informed about student evaluation purposes and effective use;

Perceptions of student evaluations seem to be connected to: their expressed teaching beliefs and emotions; their concerns with the quality of student evaluation instruments; misgivings about students' competency to judge; disenchantment with a student evaluation system that can be manipulated

easily by academics; lack of institutional support for, and recognition of, teaching; and their preoccupation with research.

(Context and personal experience determined the extent of these views.) Evaluation tends to be seen as an individual and private activity. Many teachers (predominantly university) have grave reservations about

institutional reliance on a single evaluation instrument to measure the quality and effectiveness of teaching/courses.

18

Implications for practice & change: Making connections or ‘closing the loop’For institutions: ensure that there is a clear alignment between institutional

vision/policy statements and processes of implementation;

recognise and acknowledge that student evaluation is first and foremost about development, and therefore the developmental and auditing purposes of student evaluation should be clarified within that frame;

be aware that expectations about roles and responsibilities in evaluation/appraisal can be ambiguous, and so connections among performance, evaluation and reward need to be clearly understood by all (teaching and administration/personnel staff and students).

19

Implications for practice & change: Making connections or ‘closing the loop’For evaluation systems:

recognise and acknowledge the that staff perceptions about evaluation vary and provide appropriate support and resources, to address teacher expectations and needs, without compromising institutional intents and purposes

include processes and practices (including an ongoing professional development strategy) that target both developmental and auditing purposes, while recognising the complementarity of the purposes

recognise that multiple forms of evaluation are more likely to represent a ‘well-rounded’ description of teaching and courses.

20

Questions? More information?Sarah SteinHigher Education Development Centre (HEDC)University of OtagoDunedin, New Zealandsarah.stein@otago.ac.nz

Look for the full and summary reports, soon to be on the Ako Aotearoa website