James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own...
-
Upload
simon-bates -
Category
Education
-
view
1.093 -
download
0
description
Transcript of James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own...
![Page 1: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/1.jpg)
James Bond, internetmemes and partyingpenguins
(or, what happens when students write their own assessment content)
Simon BatesPearson Strategies for Success WorkshopToronto, May 2013
![Page 2: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/2.jpg)
2
![Page 3: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/3.jpg)
3
![Page 4: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/4.jpg)
![Page 5: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/5.jpg)
![Page 6: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/6.jpg)
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
![Page 7: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/7.jpg)
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
![Page 8: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/8.jpg)
![Page 9: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/9.jpg)
![Page 10: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/10.jpg)
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
![Page 11: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/11.jpg)
The University of EdinburghEdinburgh, Scotland
5th July, 2010
Paul Denny
PeerWisebridging the gap between online learning and social media
Department of Computer ScienceThe University of AucklandNew Zealand
![Page 12: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/12.jpg)
• Web-based Multiple Choice Question repository built by students
• Students:– develop new questions with
associated explanations– answer existing questions and rate
them for quality and difficulty– take part in discussions– can follow other authors
peerwise.cs.auckland.ac.nz
![Page 13: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/13.jpg)
>100,000>100,000
student contributors
>500,000>500,000unique questions
>10,000,000>10,000,000answers
![Page 14: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/14.jpg)
14
As a question author…..
![Page 15: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/15.jpg)
15
![Page 16: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/16.jpg)
16
![Page 17: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/17.jpg)
![Page 18: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/18.jpg)
![Page 19: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/19.jpg)
19
As a question answerer …..
![Page 20: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/20.jpg)
20
![Page 21: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/21.jpg)
21
![Page 22: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/22.jpg)
22
![Page 23: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/23.jpg)
23
![Page 24: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/24.jpg)
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
![Page 25: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/25.jpg)
Timeline
2010-11: UoE pilot study
2011-12: Multi-institution, multi-course
2012-13: UBC PHYS 101
Coursera MOOC
25
![Page 26: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/26.jpg)
Pilot year (2010-11) – replace single handin
PeerWise was introduced
in workshop sessions
in Week 5
Students worked through
structured example task
and devised own Qs in groups.
All these resources are available online (see final slide)
26
![Page 27: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/27.jpg)
An assessment was set for the end of
Week 6:
Minimum requirements:
• Write one question• Answer 5• Comment on & rate 3
Contributed ~3% to course assessment
27
![Page 28: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/28.jpg)
28
![Page 29: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/29.jpg)
We were deliberately
hands off.
• No moderation
• No corrections
• No interventions at all
But we did observe…..
29
![Page 30: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/30.jpg)
• JISC project – SGC4L
![Page 31: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/31.jpg)
N (students) ~800N (staff) ~10
![Page 32: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/32.jpg)
![Page 33: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/33.jpg)
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
![Page 34: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/34.jpg)
Generally, students did
• Participate beyond minimum requirements
• Engage in community learning, correcting errors
• Create problems, not exercises
• Provide positive feedback
34
![Page 35: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/35.jpg)
35
![Page 36: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/36.jpg)
Generally, students did not
• Contribute trivial or irrelevant questions
• Obviously plagiarise
• Participate much beyond assessment periods
• Didn’t all leave it to the last minute
36
![Page 37: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/37.jpg)
• Phys 101 uptake graph showing midterm
![Page 38: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/38.jpg)
![Page 39: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/39.jpg)
39
![Page 40: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/40.jpg)
40
![Page 41: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/41.jpg)
![Page 42: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/42.jpg)
![Page 43: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/43.jpg)
![Page 44: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/44.jpg)
![Page 45: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/45.jpg)
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
![Page 46: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/46.jpg)
Correlation with end of course outcomes
![Page 47: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/47.jpg)
Quartiles
Q4 – top 25%
Q3 – upper middle
Q2 – lower middle
Q1 – bottom 25%
22 students did not take the FCI
![Page 48: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/48.jpg)
1st year Physics N=172University of Edinburgh
![Page 49: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/49.jpg)
1st year Physics N=172University of Edinburgh
![Page 50: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/50.jpg)
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
![Page 51: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/51.jpg)
Comprehensive categorisation of >50% of repository for two successive academic years
Principal measures to define a ‘high quality question’
- cognitive level of question- explanation quality- other binary criteria
![Page 52: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/52.jpg)
Category Description 6 Create (synthesise ideas) 5 Assess 4 Analyse (multi-step) 3 Apply (1-step calcs.) 2 Understand 1 Remember
Cognitive level of question
![Page 53: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/53.jpg)
Explanation
0 – Missing
1 – Inadequate(e.g. wrong reasoning / answer, trivial, flippant, unhelpful)
2 – Minimal (e.g. correct answer, but with insufficient explanation or justification, aspects may be unclear)
3 – Good/Detailed(e.g. clear and sufficiently detailed exposition of correct method and answer)
4 – Excellent (e.g. Describes physics thoroughly, remarks on plausibility of answer, use of appropriate diagrams, perhaps explains reasoning for distractors)
![Page 54: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/54.jpg)
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
1 2 3 4 5 6
Taxonomic Category
Per
cen
tag
e o
f S
ub
mit
ted
Qu
esti
on
s
First semesterFirst semester N = 350 N = 350
Second semester N = 252Second semester N = 252
Results: Question level Physics 1A / 1B 2011
![Page 55: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/55.jpg)
Results: Explanation Physics 1A 2010 and 2011
![Page 56: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/56.jpg)
‘High quality’ question
1. At least 2/6 on cognitive level (“understand” and above)
2. At least 2/4 on explanation (“minimal” and above)
3. Clearly worded question (binary)
4. Feasible distractors (binary)
5. ‘Most likely’ correct (binary)
6. ‘Not obviously’ plagiarised (binary)
![Page 57: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/57.jpg)
Results: Physics 1A 2010 and 2011
2 successive years of the same course (N=150, 350)
•‘High quality’ questions: 78%, 79%
•Over 90% (most likely) correct, and 3/5 of those wrong were identified by students.
•69% (2010) and 55% (2011) rated 3 or 4 for explanations
•Only 2% (2010) and 4% (2011) rated 1/ 6 for taxonomic level.
![Page 58: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/58.jpg)
Literature
Bottomley & Denny Biochem and Mol Biol Educ. 39(5) 352-361 (2011)
•107 Year 2 biochem students •56 / 35 / 9 % of questions in lowest 3 levels.
Momsen et al CBE-Life Sci Educ 9, 436-440 (2010)
“9,713 assessment items submitted by 50 instructors in the United States reported that 93% of the questions asked on examinations in introductory biology courses were at the lowest two levels of the revised Bloom’s taxonomy”
![Page 59: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/59.jpg)
• High general standard of engagement and student-generated questions
• Relatively few basic knowledge questions
• Transferable across disciplines / institutions
• We hypothesise scaffolding activities are critical for high level cognitive engagement
Summary
![Page 60: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/60.jpg)
Acknowledgements
Physics 101 course team
Georg RiegerFiras MoosviEmily Altiere UBC CWSEI
@simonpbates
Ross GallowayJudy HardyKaron McBrideAlison KayKeith BruntonJonathan RiiseDanny Homer
Chemistry – Peter Kirsop
Biology – Heather McQueen
Physics – Morag Casey
Comp Sci – Paul Denny
![Page 61: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/61.jpg)
Community: http://www.PeerWise-Community.org
JISC-funded multi institution study:https://www.wiki.ed.ac.uk/display/SGC4L/Home
UoE Physics Pilot Study: AIP Conf. Proc. 1413, 359 http://dx.doi.org/10.1063/1.3680069
RSC overview article
http://www.rsc.org/Education/EiC/issues/2013January/student-generated-assessment.asp
UoE Physics scaffolding resources http://www2.ph.ed.ac.uk/elearning/projects/peerwise/
Resources
![Page 62: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/62.jpg)
Question quality analysis (1st year Physics University of Edinburgh)
Assessing the quality of a student-generated question repository, submitted to Phys Rev, ST Phys Educ Res.
Multi-institution, multi-course study
Student-generated content: Enhancing learning through sharing multiple-choice questions, submitted to
International Journal of Science Education
Scaffolding Student Learning via Online Peer Learning, submitted to International Journal of Science Education
Publications in preparation / review / press
![Page 63: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/63.jpg)
Cop
yrig
ht 2
01
3 G
raha
m F
ow
ell /
Th
e H
itma
n,
re-p
rodu
ced
with
per
mis
sio
n,
Edu
catio
n In
Che
mis
try,
V
ol 5
0 N
o 1
(20
13
)
![Page 64: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/64.jpg)
![Page 65: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/65.jpg)
Photo credits
Photo credits
Community: http://www.flickr.com/photos/kubina/471164507/
Screen grab from Mwensch ‘A vision of students today’ http://www.youtube.com/watch?v=dGCJ46vyR9o
65
![Page 66: James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content](https://reader036.fdocuments.net/reader036/viewer/2022062418/553b10cb550346b9328b465f/html5/thumbnails/66.jpg)