Efficiency of assessment: initial thinking from the ITEAM project
description
Transcript of Efficiency of assessment: initial thinking from the ITEAM project
![Page 1: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/1.jpg)
Efficiency of assessment: initial thinking from the ITEAM project
Mark Russell, Liz Gormley-Fleming and Karen Robins
The ITEAM project.University of Hertfordshire
![Page 2: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/2.jpg)
The ITEAM project
• Strand A project
• A (reasonably) tightly defined focus– EVS– QMP– On line assessment and feedback system– Dashboard
![Page 3: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/3.jpg)
A matrix
Educationally effective
Reso
urce
effi
cien
cy o
r (1
/t)
![Page 4: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/4.jpg)
A matrix
Educationally effective
Reso
urce
effi
cien
cy o
r (1
/t)
![Page 5: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/5.jpg)
What about you?
Think about one of your assessments …
1. Where would you locate it on the grid?
2. How might you nudge it to be more educationally effective and more resource efficient?
…Now, next year, the year after …Thoughts in the chat box - please
![Page 6: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/6.jpg)
The ESCAPE assessment for learning themes …
Good assessment for learning:…Engages students with assessment criteria…Supports personalised learning…Focuses on student development…Ensures feedback leads to improvement…Stimulates dialogue
![Page 7: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/7.jpg)
The ESCAPE assessment for learning themes …
Good assessment for learning:…Engages students with assessment criteria…Supports personalised learning…Focuses on student development…Ensures feedback leads to improvement…Stimulates dialogue…Considers staff and student effort
![Page 8: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/8.jpg)
What about some sample assessments?
![Page 9: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/9.jpg)
Essay - Summative (2000 words)
Students
Activity Number
Activities Time (minutes) 20 60 120
1 Module aims and learning outcomes 10 10 10 102 Writing assignment brief & marking
criteria50 50 50 50
3 Essay title. 30 30 30 304 Team agreement 20 20 20 205 External examiner comments 20 20 20 206 Re drafting assignment. 10 10 10 107 Student preparation (specific) 60 60 60 608 Tutorial support-drafts & feedback* 20 per student. 400 1200 24009 Submission-setting up site, guidance 20 20 20 2010 Marking* 60 per 3 students 400 1200 240011 Feedback* 30 per 3 students. 200 600 1200
12 Module report 30 30 30 3013 Moderation* 60 per 3 students. 400 1200 2400
14 External examiner 30 30 30 3015 Assessment office*-inputting results 2 per student. 40 120 24016 Exam Board 10 10 10 1017 Student tutorial post assessment
process*15 per student. 300 900 1800
Total time in minutes 2030 5510 10730 Time in minutes per student 101.5 91.8 89.4
![Page 10: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/10.jpg)
![Page 11: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/11.jpg)
EVS test - 1 hour MCQs, 25 questions Students
Activity Number Activity Time (minutes) 20 60 120
1 Module aims and LO’s 10 10 10 102 Create and type up 25MCQ questions 15 per question 375 375 3753 Internal Moderation 20 20 20 204 External moderation 20 20 20 205 Update questions 20 20 20 206 Create practice test 375 375 375 3757 Student preparation 120 120 120 1208 Preparing environment for assessment* 15 15 15 159 Run MCQ test 60 60 60 6010 Marking - creating EVS report 10 10 10 1011 Assessment office and inputting of grades* 1 per student 20 60 12012 Providing feedback* 15 15 15 1513 Module report 30 30 30 3014 External examiner moderation 30 30 30 3015 Module board 20 20 20 2016 Post assessment process tutorial support* 15 per student 300 900 180017 Student tutorial post assessment process* 120 120 120 120 Total in minutes 1560 2200 3160 Time in minutes per student 78 37 26
![Page 12: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/12.jpg)
OSCE-summative.
Students
Activity Number
Activities Time (minutes) 20 60 120
1 Module aims and LO’s 5 5 5 52 Case scenario’s 120 / scenario 120 120 1203 Team meeting 120 120 120 1204 External examiner 20 20 20 205 Uploading assessment information 30 30 30 306 Student preparation 120 120 120 1207 Environment* 450 450 450 4508 OSCE* (role of Academic) - 6hrs*10
academics3600 3600 10800 21600
9 Marking & feedback* 10 per student 200 600 120010 Moderation* 5 per student 100 300 60011 Module report 30 30 30 3012 External examiner 20 20 20 2013 Assessment office* 5 per student. 100 300 60014 Module board 10 10 10 1015 Post assessment process feedback* 15 per student. 300 900 1800
Total time in minutes 5225 13825 26725
Time in minutes per student
261.3 230.4 222.7
![Page 13: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/13.jpg)
Essay OSCE Group Presentation Exam EVS 0
50
100
150
200
250
300
Assessment time per student (mins)
20 students60 students120 students
![Page 14: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/14.jpg)
Essay OSCE Group Presentation Exam EVS 0
50
100
150
200
250
300
Assessment time per student (mins)
20 students60 students120 students
12% change
66% change34% change
![Page 15: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/15.jpg)
And you …?What sort of things are you doing / (need to do) to help your staff make informed
decisions about choosing an assessment type for effectiveness and
efficiency?Thoughts in the chat box - please
![Page 16: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/16.jpg)
Developing a tool
![Page 17: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/17.jpg)
Plotting the data
Numbers of students
Tim
e (m
inut
es)
![Page 18: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/18.jpg)
Diverging time scales
Numbers of students
Tim
e (m
inut
es)
Assessm
ent type A
Assessment type B
![Page 19: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/19.jpg)
Overlapping time scales
![Page 20: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/20.jpg)
Not prescriptive!
![Page 21: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/21.jpg)
Not prescriptive!
![Page 22: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/22.jpg)
And so …In what ways might these tools (and approaches) help or hinder your work?
Thoughts in the chat box - please
![Page 23: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/23.jpg)
Does (say) 20 hours of our effort yield the same learning payoff?
![Page 24: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/24.jpg)
Does (say) 20 hours of our effort yield the same learning payoff?
An example with EVS
![Page 25: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/25.jpg)
![Page 26: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/26.jpg)
![Page 27: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/27.jpg)
![Page 28: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/28.jpg)
The ESACPE assessment for learning themes …
Good assessment for learning:…Engages students with assessment criteria…Supports personalised learning…Focuses on student development…Ensures feedback leads to improvement…Stimulates dialogue…Considers staff and student effort
![Page 29: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/29.jpg)
So let’s always use the least time consuming?
![Page 30: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/30.jpg)
So let’s always use the least time consuming?
Alas, it’s
a little (a lot) m
ore complex and
messy th
an that.
![Page 31: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/31.jpg)
31
The purposes of assessment are important too!
SummativeHigh StakesMeasure learning
FormativeNo / Low StakesMotivate learning
Educative Auditing
![Page 32: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/32.jpg)
32
The purposes of assessment are important too!
SummativeHigh StakesMeasure learning
FormativeNo / Low StakesMotivate learning
Feedback rich?Linked (in-process) tasks?Regular tasks?Error does not lead to punishment
Need for feedback is questionable (MBR)One-off tasksEnd of process tasks?
Educative Auditing
![Page 33: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/33.jpg)
Assessment patterns
Beyond the single assessment task
![Page 34: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/34.jpg)
Some alternatives…
![Page 35: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/35.jpg)
And many more …
![Page 36: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/36.jpg)
Programme view of assessment
![Page 37: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/37.jpg)
Programme view of assessment
![Page 38: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/38.jpg)
Programme view of assessment
![Page 39: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/39.jpg)
Far reaching implications• Our work (as teachers) extends beyond our
classroom. How might we measure this?
• Performance on singular assessment?• Success in Module?• Success in Year?• Success in Programme?
• Institutional reputation?
![Page 40: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/40.jpg)
Conclusion – to date
• Numbers ‘might’ be countable• Numbers ‘might’ be meaningful (process
improvement, stimulating dialogue, moving in the ‘right direction’)
![Page 41: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/41.jpg)
A possible conclusion – to date
• Numbers ‘might’ be countable• Numbers ‘might’ be meaningful (process
improvement, stimulating dialogue, moving in the ‘right direction’)
But…
![Page 42: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/42.jpg)
A possible conclusion – to date
• Numbers ‘might’ be countable• Numbers ‘might’ be meaningful (process
improvement, stimulating dialogue, moving in the ‘right direction’)
But…
![Page 43: Efficiency of assessment: initial thinking from the ITEAM project](https://reader036.fdocuments.net/reader036/viewer/2022062816/56816391550346895dd48600/html5/thumbnails/43.jpg)
Some questions for you to ponder…What is the difference in learning payoff between a 4000 word and a 2000 word essay ?
Is a single 4000 word essay more beneficial than (say) two 1500 word essays?
What are the resource implications of the above variants?
What are the various processes of assessment doing?
Where might you best place your efforts to reduce the resource burden while keeping the effectiveness of the assessment?
In what ways are you facilitating meaningful discussions about both resource efficiency and educational effectiveness?