Computerised Peer-Assessment Phil Davies University of Glamorgan South Wales Lecturer getting out of doing marking Need for assessment? As tutors we are trying to separate the sheep from the goats via the assessment process. As tutors we are trying to separate the sheep from the goats via the assessment process. This can often be difficult with the time constraints imposed on tutors, in what Chanock describes as more goat-friendly times (Chanock, 2000). This can often be difficult with the time constraints imposed on tutors, in what Chanock describes as more goat-friendly times (Chanock, 2000). Problem . Feedback against time! Problem . Feedback against time! Defining Peer-Assessment In describing the teacher.. In describing the teacher.. A tall b******, so he was. A tall thin, mean b******, with a baldy head like a lightbulb. Hed make us mark each others work, then for every wrong mark we got, wed get a thump. That way he paused we were implicated in each others pain McCarthys Bar (Pete McCarthy, 2000,page 68) Why Peer-Assessment Working together to learn (and be assessed) Working together to learn (and be assessed) Overcome Perceptions Overcome Perceptions Staff > Lecturer getting easy life not marking Student> Lecturer getting easy life not marking Student Awareness of Benefits . IMPORTANT TANGIBILITY Student Awareness of Benefits . IMPORTANT TANGIBILITY Success of dependant upon scalability (computerisation).. Student Numbers have risen dramatically since 1991 without a concomitant increase in resources (Pond et al, 1995) Success of dependant upon scalability (computerisation).. Student Numbers have risen dramatically since 1991 without a concomitant increase in resources (Pond et al, 1995) Computerised Peer-Assessment CAP System (2000) CAP System (2000) Permits students to mark & comment the work of other students. (normally 6-8) Permits students to mark & comment the work of other students. (normally 6-8) Also initial self-assess stage (reflection) used as a standard of expectation Also initial self-assess stage (reflection) used as a standard of expectation Internet, not Web-based system (developed in Visual Basic / Access) Internet, not Web-based system (developed in Visual Basic / Access) Having done the marking, what next? Students should receive feedback Students should receive feedback What feedback? What feedback? Marks Comments Which is most important? Which is most important? To students or staff AUTOMATICALLYTHE MARKER.. ANONYMOUS What should the marker do? Reflect Read the essay again Read the essay again Take into account the essay owners comments Take into account the essay owners comments Further clarification (if it is needed, then is this a black mark against the marker?) Further clarification (if it is needed, then is this a black mark against the marker?) Try to appease the essay owner? Try to appease the essay owner? Modify mark based upon reflection? Modify mark based upon reflection? Give more constructive feedback Give more constructive feedback Exercise One In the top box what are the inherent advantages and disadvantages of peer- assessment In the top box what are the inherent advantages and disadvantages of peer- assessment Middle Box Mark (/10) and comment to support Middle Box Mark (/10) and comment to support Bottom Box Mark the marker /10 and comment to support Bottom Box Mark the marker /10 and comment to support Now give the work back! Now give the work back! Which of the above required higher order skills? Which of the above required higher order skills? Must be rewarded for doing the mark for marking process.. Based on quality How to judge? How to judge? Standard of expectation (self-assessment) Standard of expectation (self-assessment) Marking consistency Marking consistency Commenting, quality, measure against mark Commenting, quality, measure against mark Discussion Element Discussion Element Need for additional comments black mark? Need for additional comments black mark? Reaction to requests / further clarification Reaction to requests / further clarification How easy to get an automated mark for marking? Statistically fairly easy to create a mark for marking based upon marks Statistically fairly easy to create a mark for marking based upon marks Take into account high and low markers Take into account high and low markers Standard of expectation Standard of expectation Consistency judge against final mark awarded for an essay (compensated median) Consistency judge against final mark awarded for an essay (compensated median) What about the comments? What about the comments? Feedback Index Produce an index that reflects the quality of commenting Produce an index that reflects the quality of commenting Produce an average feedback index for an essay (also compensated?) Produce an average feedback index for an essay (also compensated?) Compare against marker in a similar manner to marks analysis Compare against marker in a similar manner to marks analysis Where does this feedback index come from and is it valid? Where does this feedback index come from and is it valid? The way to get the feedback index? Develop an application?? Develop an application?? C-Rater? Spelling Mistakes Spelling Mistakes Similar Meanings? Similar Meanings? That was cool Really Choc Really Good Essay Manually Manually Commonality!! In the 67 essays that were marked In the 67 essays that were marked Only 96 comments 44% positive and 56% negative Highly critical if something not explained properly Comments grouped into 10 categories Need to QUANTIFY these comments.. Feedback index Need to QUANTIFY these comments.. Feedback index Create a database holding positive & negative (by category) Create a database holding positive & negative (by category) General points of note Main criteria used for peer-marking Main criteria used for peer-marking Explanations, conclusions, references, examples Better students were critical (upper quartile) Better students were critical (upper quartile) Better understanding permitting criticism? Confidence? Hostility? Weaker student in cloud cuckoo land consistently Weaker student in cloud cuckoo land consistently Consistency of markers in marks & comments: 30-34, 35-39, 55-59; 65-69; Consistency of markers in marks & comments: 30-34, 35-39, 55-59; 65-69; Note OVER-Comment & Mark Note OVER-Comment & Mark Time Consuming? Can we formulate the marking process Can we formulate the marking process Take away need for quantification process of analyzing comments Take away need for quantification process of analyzing comments Is it still peer-assessment if the students are told what to say? Is it still peer-assessment if the students are told what to say? Exercise Two 1. I think youve missed out a big area of the research 2. Youve included a big chunk that you havent cited 3. There arent any examples given to help me understand 4. Grammatically it is not what it should be like 5. Your spelling is atroceious 6. You havent explained anything to me 7. Youve directly copied my notes as your answer to the question 8. Most of what youve said isnt about the question STUDENT FRED REFERENCES: Positive Negative . Personal Valuation 5, 3, 2, 1 3, 1, 2 Is it my job to teach students how to write essays, etc? Assessment MUST be directed at subject skills Assessment MUST be directed at subject skills Why bother writing essays, doing exam questions, etc. doesnt relate to needs or learning outcomes of subject Why bother writing essays, doing exam questions, etc. doesnt relate to needs or learning outcomes of subject Post HND N-tier Assess the essays of the final year (last year) Post HND N-tier Assess the essays of the final year (last year) Preparation/Research: Judge knowledge against last years results.. Both marks & comments Preparation/Research: Judge knowledge against last years results.. Both marks & comments Mistake!! Mistake!! MarksMarking DifferenceFeedback DifferenceMapping for MCQ & Essays 5