Post on 29-Mar-2015
Satisfied – but students want more feedback: Why did I get 37%?
1
University of ExeterLearning and Teaching Conference, May 22nd 2008
Professor Brenda SmithHigher Education Academy
Aims…
• Emphasis assessment for learning rather than of
learning
• Encourage the active engagement of students in their
learning
• Enhance assessment and feedback process for both
staff and students
Road Map of the Presentation
• Myths about assessment
• Burgess Report – a case for change
• Assessment methods – have we got it right?
• NSS tables in relation to Exeter
• Focus on feedback
John Cowan, former Director, OU, Scotland
Assessment is the engine that
drives learning
5
1.Certification
3.Learning
4.Sustainability
2.QualityAssurance
Why do we assess?
What’s wrong with assessment?
The evidence:• QAA Subject Reviews• National Student Survey• Rising concern about cheating and plagiarism• Student complaintsSome issues:• Too much summative and insufficient formative
assessment• Insufficient effective feedback that comes too late• Variations in practice• Unscholarly practices (especially with numbers)• Myths and traditions that inhibit change
Myths…
• Students benefit from jumping assessment hurdles unaided
• Experienced tutors apply consistent assessment standards
• Students don’t want feedback on exams
• That it is possible to distinguish the quality of work to a precision of one percentage point
• Moving back to exams will solve ‘cheating’ problems
• Double marking will ensure fairness and reliability
• Assessment standards are consistent across an institution
The Burgess Report
The Burgess ReportA case for change
• Present system cannot capture achievement in some key areas of interest to students and employers
• Emphasis on the top two degree classes wrongly reinforces an impression that a Lower Second or Third Class Degree is not an achievement
• Institutional methods for calculating the degree classification could be clearer
The Burgess ReportA case for change
• A summative system – is at odds with lifelong learning
• There is a need for greater emphasis on additional information that is currently contained in the European Diploma Supplement and academic transcript (an official record of a learner’s programme of study, grades achieved and credit gained)
• Institutional practices – complex and inconsistent
Complexity, variety and inconsistency of classification assessment regulations
(SACWG) surveyed 35 HEIs
25 used percentage marks; eight used grades; and two used both percentages and grades
18 institutions aggregated or averaged percentage marks to arrive at the honours degree classification; 4 used a ‘profiling’ approach; and 13 used both or either
SACWG: Student Assessment and Classification Working Group
Complexity, variety and inconsistency ofclassification assessment regulations
Variations in degree classification can be caused by:
Differing institutions’ award algorithms
‘Dropping’ a no of modules when calculating the degree classification
A shift towards course work and away from formal examinations
Institutions diverged on their approaches to borderline performances
Variations in the weightings given to the penultimate and final years of full-time study
Ref: SACWG and Mantz Yorke et al
Relative weightings given to results at Levels 2 and 3
Ratio, Level 3:level 2 Frequency
4:1 1
3:1 4
70:30 3
2:1 4
3:2 2
4:3 1
1:1 5
Ref: Mantz Yorke et al
A single index of performance, whether it is agrade-point average or an honours degree classification, tells nothing on its own aboutthe trajectory of a student’s development. It does not differentiate between the student who progresses steadily over the course of a programme and the student who starts slowly but in the final year produces work of an outstanding standard
Issues involving student trajectory
The student voice - what they would like…
More advice& support
Relevant &real-life topics
Choice of assessment
topicsVariety of assessment
methods
Feedback to help me
learn
For the assessment load to be spread out
Clear criteria
MESA Project
I never realised that assessment was
for learning
Department AYear 1
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Year 2
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Year 3 / Final Year
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Department B
Year 1
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Year 2
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Year 3 / Final Year
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Department C
Year 2
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Year 1
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Year 3 / Final Year
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Department D
Year 1
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Year 2
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Year 3 / Final Year
100
80
60
40
20
0
20
40
60
80
100
exam coursework
Assessment Methods at Exeter?
University of Hertfordshire analysed assessment methods across the institution
Tasks• Multiple choice• Open book• IT based• Interactive• Group element• Role play
Medium• Oral• Numeric• Diagram/
Pictorial
Cognitive skills• Analytic• Evaluative• Skills focus• Primary research• Theory focus
Time-span• Reflective• Process/Periodic• Portfolio
Who assesses?
Self assessPeer assessSelf set element
Work-related
Practice focusCase Study
University of Hertfordshire
National Student Survey(Full-time students UK)
The Assessment and feedback scale 2006 2007
1. The criteria used in marking have been made clear in advance
67 68
2. Assessment arrangements and marking have been fair
73 72
3. Feedback on my work has been prompt
51 51
4. I have received detailed comments on my work
55 54
5. Feedback on my work has helped me clarify things I did not understand
51 51
Video Man Friday
• Please watch the video and ask yourself are you always clear about the criteria you set for students?
• How do you ensure that they understand what is required of them?
• Could you give students practice in setting and then marking against criteria?
Comparison of courses within ‘x’ University in terms of students’ use of feedback
Assessment Experience Questionnaire
(AEQ items)http://www.open.ac.uk/science/fdtl/overview.htm
‘Best’ course
‘Worst’ course
%
strongly agree
% strongly
agree
The feedback helps me to understand things better 36% 6%
The feedback shows me how to do better next time 31% 4%
The feedback prompts me to go back over material covered earlier in the course
13% 1%
% agree
stronglyor
agree
I do not use the feedback for revising 17% 44%
Everyone can make a difference..
What are the positive features of your module/course?
How do you:• Identify them?• Disseminate them within departments?• Disseminate across the institution?• Disseminate externally?• Draw on effective practices in learning and teaching
from external sources?• Embed that practice?
Dissemination
1. For awareness
2. For understanding
3. For action.
By ‘action’ here we mean some change of
practice inspired by, and making use of, the
materials or processes produced by your initiative
4. Evaluate the success of your dissemination
Conditions necessary for innovations to improve Learning, Teaching & Assessment
• Implement in student focused ways
• Departments that value teaching
• Workload that are not too high
• Effective leadership of learning, teaching &
assessment
• Perceived recognition and rewards
Recommendations in relation to support, adaption & implementation
1. Recognise in workload allocation formulae
2. Recognise scholarship of teaching outcomes – publications etc
3. Provide small scale grants
4. Provide sources of institutional support and advice
5. Encourage development of cross institutional networks & events
6. For academic developers to work alongside ‘project teams’
7. Ensure time release as appropriate
Ref: Carrick Institute, Australia
Managing the Assessment process
FEEDBACK TO HELP STUDENTS TO LEARN
Why feedback?
• What kind of feedback do learners need?
• How much is enough?
• How often is often enough?
• How can we make feedback efficient and effective?
• How can we ensure feedback is taken seriously?
• Should feedback be one way?
• Can students give each other feedback?
Why give feedback?
• Increase motivation and interest (Give choice)
• Improve performance
• Eliminate misconceptions
• Develop independence
• Raise confidence
• Create trust
• Individualise learning
• Develop a learning community
Paradoxes of feedback
• Teachers complain about it
• Students ignore it
• They get it too late
• When they get it it’s not useful
Questions…..
• How many hours do you spend giving feedback?
• At what time in the semester do you give most feedback?
• Could you give more feedback upfront?
• Could feedback be spread out throughout the year?
• Could you encourage peer feedback?
• Could you give feedback in different ways (MP3)?
Staff comments on feedback and marking
“They often don’t even collect their essays…..Some of them are more ready to argue about their mark than they are to collect their essay”
“They came up to me afterwards…and they said well we didn’t really understand what you meant by evaluate. And I kind of thought well isn’t it obvious, but the more I think about it I wonder if it is actually obvious to students, what the word means”
Ref:www.assessmentplus.net
45
Do we try and trip our students up?
Evaluation of tutor feedback
om
ission
Use
of E
nglish
Ask
for
clarifi
catio
n
Erro
r
Giv
e cla
rifica
tion
Pra
ise
Enco
ura
gem
ent
Feed fo
rward-iv
e
30%
20%
17%
11.5%
10.5%
3.1%
1.5%
0%0%
OU/SHU Improving the Effectiveness of Formative Assessment in Science
A study (Carless 2006) reported a substantial difference in the perceptions of tutors and their students on feedback on assessed work
A study at the University of Sydney, ran focus groups with 5,000 students to gain an in-depth understanding of student perceptions of feedback
3 dimensions emerged
Differing perceptions (most recognisable if written)
Impact of feedback – timeliness, significance & 1st year experience (difference between University and School)
Credibility of feedback – this was related to the students’ perceptions of the lecturers themselves
Do you know what your students think about feedback?
Students comments
They are not very specific when they mark. I got “some good evaluation”, what does that mean?
You put so much effort into this work that you really want feedback and to know they have read it properly. If you just get a mark you think, “I’ve worked for days and that’s it?”
I think it would be helpful as well if when we had our feedback we could relate it to the assessment criteria
Ref:www.assessmentplus.net
49
Do these comments aid learning?
A Excellent
C- Fine
C/C- Interesting
C Opening lousy - main part OK
C- Your writing is appalling
D Terrible - didn’t you learn anything at school about essay writing!
Comments difficult to understand…
This essay is not sufficiently analytical
This report is not logically structured
However we also need to focus our efforts on strengthening the skills of self-assessment
Students need to be active in this process
We need to use language that they can understand
How might we use students in this process?
Involve the studentsFeedback as a dialogue
For Student Completion
1. These are areas of my work that I think are good for the following reasons
2. Please comment on the following areas of work
3. What I want to improve or do differently next time
4. The mark I think this piece of work deserves is
For Staff Completion
Order of giving feedback
1. Good news – what was done well
2. Enhancement news – what still needs to improve
3. Options – what can be done to improve
4. Plans – what the learner intends to do
5. Commitments – what both parties agree to do, how, to what standard and by when?
Formative feedback - a challengeto course teams
• Is the feedback motivating and does it aid learning?
• Do students use feedback to improve future performance?
• Do you and the student track the
changes?
• How can you improve the need for
students to want feedback and use it as feed forward?
• Can you give credit to students
who use their feedback?
How can we engage with our students?
• Engage from day 1• Rethink induction• Get students involved in setting and marking
against the criteria• Encourage peer feedback & peer assessment• Rethink the course committee agenda• Ask their views
The one minute paper
Structured feedback session…..
Structured Feedback Session
This is an activity to give an individual and collective voice to your experiences of assessment and feedback
TASK• Individually think of all the good things about your
experiences of assessment and feedback• Write each one on a separate post-it note• In your group put all your post-its on a flip chart and say
them out loud as you post them up• Now collectively agree on the top 3 aspects of your
course in relation to assessment and feedback that rate the highest
Structured Feedback Session
TASK• Individually think of all aspects of your assessment
and feedback experiences that could be enhanced• Write each one on a separate post-it note• In your group put all your post-its on a flip chart and
say them out loud as you post them up• Now collectively agree on the top 3 enhancements
that could be improved • Taking those top 3 aspects lists some ideas that
could be taken forward
Applying the seven principles of assessment
Seven principles of good feedback practice
1. Facilitates the development of self-assessment (reflection) in learning
2. Encourages teacher and peer dialogue around learning
3. Helps clarify what good performance is (goals, criteria, standards expected)
4. Provides opportunities to close the gap between current and desired performance
(SENLEF Project)
Seven principles of good feedback practice
5. Delivers high quality information to students about their learning
6. Encourages positive motivational beliefs and self-esteem
7. Provides information to teachers that can be used to help shape the teaching
Ref: Juwah, Macfarlane-Dick, Matthew, Nicol, Ross & Smith (2004)
A vision of students
Do league tables give us the whole picture?
• Does our position on the league tables guarantee a
really positive learning experience for all our
students?
• How might we collectively work towards an
enhanced student experience?
Financial indicators Job prospects Academic indicators Ease of entry and demand Student life indicators
1 Lampeter, University of Wales2 Heythrop College, University of London3 Stirling University4 School of Pharmacy, University of London5 University of St Andrews6 St George's, University of London7 Harper Adams University College8 Bishop Grosseteste University College9 King's College, London10 University of Edinburgh
113 University of Exeter
Congratulations
EXETER is a top 20 UK University, according to all four of the main higher education league tables
BUT does that guarantee that the student experience is enhanced?
HOW can you make that student learning experience even better?
Reflections …..