TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)
-
Upload
testa-winch -
Category
Education
-
view
168 -
download
1
Transcript of TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)
![Page 1: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/1.jpg)
Programme Assessment: Insights about student learning from TESTA
Dr Tansy Jessop, TESTA Project LeaderPresentation to the SDG Course Leaders
University of West of Scotland12 May 2014
![Page 2: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/2.jpg)
1)Assessment drives what students pay attention to, and defines the actual curriculum (Ramsden 1992).
2)Feedback is significant (Hattie, 2009; Black and Wiliam, 1998)
3)Programme is central to influencing change.
TESTA premises
![Page 3: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/3.jpg)
Thinking about modules
modulus (Latin): small measure
“interchangeable units”
“standardised units”
“sections for easy constructions”
“a self-contained unit”
![Page 4: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/4.jpg)
How well does IKEA 101 packaging work for Sociology 101?
Furniture Bite-sized Self-contained Interchangeable Quick and instantaneous Standardised Comes with written
instructions Consumption
Student Learning Long and complicated Interconnected Distinctive Slow, needs deliberation Varied, differentiated Tacit, unfathomable,
abstract Production
![Page 5: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/5.jpg)
HEA funded research project (2009-12) Seven programmes in four partner
universities Maps programme-wide assessment Engages with Quality Assurance processes Diagnosis – intervention – cure
What is TESTA?Transforming the Experience of Students through Assessment
![Page 6: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/6.jpg)
TESTA ‘Cathedrals Group’ Universities
![Page 7: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/7.jpg)
Edinburgh Edinburgh
Napier
Greenwich
Canterbury Christchurch
Glasgow
Lady Irwin College University of Delhi
University of West ScotlandSheffield Hallam
![Page 8: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/8.jpg)
TESTA
“…is a way of thinking about assessment and feedback”
Graham Gibbs
![Page 9: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/9.jpg)
Time-on-task Challenging and high expectations Students need to understand goals and
standards Prompt feedback Detailed, high quality, developmental
feedback Dialogic cycles of feedback Deep learning – beyond factual recall
Based on assessment principles
![Page 10: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/10.jpg)
TESTA Research Methods(Drawing on Gibbs and Dunbar-Goddet, 2008,2009)
ASSESSMENT EXPERIENCEQUESTIONNAIRE
FOCUS GROUPS
PROGRAMME AUDIT
Programme Team Meeting
Case Study
![Page 11: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/11.jpg)
Number of assessment tasks Summative/formative Variety Proportion of exams Oral feedback Written feedback Speed of return of feedback Specificity of criteria, aims and learning
outcomes.
Audit in a nutshell
![Page 12: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/12.jpg)
Quantity of Effort Coverage of content and knowledge Clear goals and standards Quantity and Quality of Feedback Use of feedback Appropriate assessment Learning from exams Deep and surface learning
Assessment Experience Questionnaire
![Page 13: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/13.jpg)
Focus Groups
Student voice and narrativeExplanation Corroboration & contradictionCompelling evidence with the
stats
![Page 14: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/14.jpg)
Case Study X: what’s going on?
Mainly full-time lecturers Plenty of varieties of assessment, no exams Reasonable amount of formative assessment (14 x) 33 summative assessments Masses of written feedback on assignments (15,000 words) Learning outcomes and criteria clearly specified….looks like a ‘model’ assessment environment
But students: Don’t put in a lot of effort and distribute their effort across few topics Don’t think there is a lot of feedback or that it very useful, and don’t
make use of it Don’t think it is at all clear what the goals and standards are …are unhappy
![Page 15: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/15.jpg)
Case Study Y: what’s going on?
35 summative assessments No formative assessment specified in documents Learning outcomes and criteria wordy and woolly Marking by global, tacit, professional judgements Teaching staff mainly part-time and hourly paid….looks like a problematic assessment environment
But students: Put in a lot of effort and distribute their effort across
topics Have a very clear idea of goals and standards Are self-regulating and have a good idea of how to close
the gap
![Page 16: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/16.jpg)
Two paradigms…
![Page 17: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/17.jpg)
Transmission Model
![Page 18: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/18.jpg)
Social Constructivist model
![Page 19: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/19.jpg)
AuditAEQFocus Groups
Research Methods
![Page 20: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/20.jpg)
1) Any interesting patterns?2) Anything particularly striking?3) Any dangling questions, curiosities,
scepticisms?4) Any predictions, hunches, thoughts
about what other data might throw up?
Task 1: Audit data
![Page 21: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/21.jpg)
Audit Variables Humanities
(Mean)
Sciences(Mean)
Professional
(Mean)Total number of assessments
54 67 42
Number of summative assessments
42 43 32
Number of formative assessments
12 31 10
Varieties of assessment (n) 11 15 14Proportion of examinations 14.3% 30.6% 15%Time to return feedback 23 days 20 days 23 daysAmount of oral feedback 3hrs
17mins4hrs
56mins10hrs
33minsAmount of written feedback 7,382
words3,615 words
7,040 words
Discipline Effects
![Page 22: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/22.jpg)
a) Fill in an AEQ from the vantage point of being a student in the past, or one of your own students
b) Cluster the questions into scales using the Scales Exercise.
c) What’s a ‘good’ score?d) Any thoughts, issues, comments,
questions?
Task 2: The AEQ
![Page 23: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/23.jpg)
In pairs, explore programme audit and AEQ data from one programme.
Triangulation: Audit and AEQ data
![Page 24: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/24.jpg)
In pairs/groups, read through quotes from student focus group data on a particular theme.
What problems does the data imply? What solutions might a programme develop
to address some of these challenges? A3 sheets provided to tease out challenges
and solutions.
Focus Group data
![Page 25: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/25.jpg)
Challenges Solutions
Student voice data
![Page 26: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/26.jpg)
If there weren’t loads of other assessments, I’d do it.
If there are no actual consequences of not doing it, most students are going to sit in the bar.
I would probably work for tasks, but for a lot of people, if it’s not going to count towards your degree, why bother?
The lecturers do formative assessment but we don’t get any feedback on it.
Theme 1: Formative is a great idea but…
![Page 27: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/27.jpg)
We could do with more assessments over the course of the year to make sure that people are actually doing stuff.
We get too much of this end or half way through the term essay type things. Continual assessments would be so much better.
So you could have a great time doing nothing until like a month before Christmas and you’d suddenly panic. I prefer steady deadlines, there’s a gradual move forward, rather than bam!
Theme 2: Assessment isn’t driving and distributing student effort
![Page 28: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/28.jpg)
The feedback is generally focused on the module.
It’s difficult because your assignments are so detached from the next one you do for that subject. They don’t relate to each other.
Because it’s at the end of the module, it doesn’t feed into our future work.
You’ll get really detailed, really commenting feedback from one tutor and the next tutor will just say ‘Well done’.
Theme 3: Feedback is disjointed and modular
![Page 29: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/29.jpg)
The assessment guidelines is a formal document so the language is quite complex and I’ve had to read it a good few times to kind of understand what they are saying.
Assessment criteria can make you take a really narrow approach.
They read the essay and then they get a general impression, then they pluck a mark from the air.
It’s a shot in the dark. We’ve got two tutors – one marks completely
differently to the other and it’s pot luck which one you get.
Theme 4: Students are not clear about goals and standards
![Page 30: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/30.jpg)
1. Too much summative; too little formative2. Too wide a variety of assessment3. Lack of time on task4. Inconsistent marking standards5. ‘Ticking’ modules off6. Poor feedback: too little and too slow7. Lack of oral feedback; lack of dialogue about
standards8. Instrumental reproduction of materials for marks
Main findings
![Page 31: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/31.jpg)
1. Students and staff can’t do more of both.2. Reductions in summative – how many is
enough?3. Increase in formative – and make sure it is
valued and required.4. Debunking the myth of two summative per
module.5. Articulating rationale with students, lecturers,
senior managers and QA managers.
1. Summative-formative issues
![Page 32: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/32.jpg)
The case of the under-performing engineers (Graham, Strathclyde)The case of the cunning (but not litigious) lawyers (Graham, somewhere)The case of the silent seminar (Winchester)The case of the lost accountants (Winchester)The case of the disengaged Media students (Winchester)
1. Examples of ramping up formative
![Page 33: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/33.jpg)
The case of low effort on Media StudiesThe case of bunching on the BA Primary
2. Examples of improving ‘time on task’
![Page 34: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/34.jpg)
The case of the closed door (Psychology)The case of the one-off in History (Bath Spa)The case of the Sports Psychologist (Winchester)The conversation gambit
3. Engaging students in reflection through improving feedback
![Page 35: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/35.jpg)
The case of the maverick History lecturer (a dove)
The case of the highly individualistic creative writing markers
4. Internalising goals and standards
![Page 36: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/36.jpg)
Programmatic Assessment Design
Feedback Practice
Paper processes to people talking
Changes
![Page 37: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/37.jpg)
Improvements in NSS scores on A&F – from bottom quartile in 2009 to top quartile in 2013
Three programmes with 100% satisfaction ratings post TESTA
All TESTA programmes have some movement upwards on A&F scores
Programme teams are talking about A&F and pedagogy
Periodic review processes are changing for the better.
Impacts
![Page 38: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/38.jpg)
www.testa.ac.uk
![Page 39: TESTA, Presentation to the SDG Course Leaders, University of West of Scotland (May 2014)](https://reader036.fdocuments.net/reader036/viewer/2022070518/58e9898d1a28aba6498b6319/html5/thumbnails/39.jpg)
Gibbs, G. & Simpson, C. (2004) Conditions under which assessment supports students' learning. Learning and Teaching in Higher Education. 1(1): 3-31.Gibbs, G. & Dunbar-Goddet, H. (2009). Characterising programme-level assessment environments that support learning. Assessment & Evaluation in Higher Education. 34,4: 481-489.Hattie, J. (2007) The Power of Feedback. Review of Educational Research. 77(1) 81-112.Jessop, T. and Maleckar, B. (in press). The Influence of disciplinary assessment patterns on student learning: a comparative study. Studies in Higher Education.Jessop, T. , El Hakim, Y. and Gibbs, G. (2014) The whole is greater than the sum of its parts: a large-scale study of students’ learning in response to different assessment patterns. Assessment and Evaluation in Higher Education. 39(1) 73-88.Jessop, T, McNab, N & Gubby, L. (2012) Mind the gap: An analysis of how quality assurance processes influence programme assessment patterns. Active Learning in Higher Education. 13(3). 143-154.Nicol, D. (2010) From monologue to dialogue: improving written feedback processes in mass higher education, Assessment & Evaluation in Higher Education, 35: 5, 501 – 517Sadler, D.R. (1989) Formative assessment and the design of instructional systems, Instructional Science, 18, 119-144.
References