Phil Davies School of Computing University of Glamorgan “Super U”

11

Phil DaviesPhil DaviesSchool of ComputingSchool of Computing

University of Glamorgan “Super U”University of Glamorgan “Super U”

Peer-Assessment:Peer-Assessment: No marks required, just No marks required, just

feedback?feedback?Evaluating the Quality of Evaluating the Quality of

Computerized Peer-Feedback Computerized Peer-Feedback compared with Computerized compared with Computerized

Peer-Marking Peer-Marking

22

Definition of Peer-Assessment?Definition of Peer-Assessment?

In describing the teacher ..In describing the teacher ..

A tall b******, so he was. A tall thin, mean b******, A tall b******, so he was. A tall thin, mean b******, with a baldy head like a lightbulb. He’d make us with a baldy head like a lightbulb. He’d make us mark each other’s work, then for every wrong mark each other’s work, then for every wrong mark we got, we’d get a thump. That way – he mark we got, we’d get a thump. That way – he paused – ‘we were implicated in each other’s pain’paused – ‘we were implicated in each other’s pain’

McCarthy’s Bar McCarthy’s Bar (Pete McCarthy, 2000,page 68)(Pete McCarthy, 2000,page 68)

33

Why Peer-AssessmentWhy Peer-Assessment Community of Practice!!Community of Practice!! Working together to learn (and be assessed)Working together to learn (and be assessed) PerceptionsPerceptions

– Staff > Lecturer getting easy life not markingStaff > Lecturer getting easy life not marking– Student> Lecturer getting easy life not markingStudent> Lecturer getting easy life not marking

Student Awareness of Benefits …. IMPORTANT Student Awareness of Benefits …. IMPORTANT TANGIBILITYTANGIBILITY

Success dependant upon scalability Success dependant upon scalability (computerisation) .. “Student Numbers have risen (computerisation) .. “Student Numbers have risen dramatically since 1991 without a concomitant dramatically since 1991 without a concomitant increase in resources” (Pond et al, 1995)increase in resources” (Pond et al, 1995)

44

Computerised Peer-AssessmentComputerised Peer-Assessment

CAP SystemCAP System

Permits students to mark & comment the work of Permits students to mark & comment the work of other students . (normally 8)other students . (normally 8)

Also initial self-assess stage (reflection) … used Also initial self-assess stage (reflection) … used as a as a standard of expectationstandard of expectation

Internet, not Web-based system (developed in Internet, not Web-based system (developed in Visual Basic / Access)Visual Basic / Access)

66

Having done the marking, what Having done the marking, what next?next?

Students should receive feedbackStudents should receive feedback

What feedback?What feedback?– Marks Marks – CommentsComments

Which is most important?Which is most important?– To students or staffTo students or staff

88

AUTOMATICALLY EMAIL THE MARKER ..

ANONYMOUS

99

What should the marker do?What should the marker do?ReflectReflect

Look at essay againLook at essay again Take into account the essay owner’s Take into account the essay owner’s

commentscomments Further clarification (if it is needed, then is Further clarification (if it is needed, then is

this a ‘black mark’ against the marker?)this a ‘black mark’ against the marker?) Try to ‘appease’ the essay owner?Try to ‘appease’ the essay owner? Modify mark based upon reflection?Modify mark based upon reflection? Give more feedbackGive more feedback

1111

Must be rewarded for doing the Must be rewarded for doing the ‘mark for marking’ process .. Based ‘mark for marking’ process .. Based

on qualityon quality How to judge?How to judge?

Standard of expectation (self-assessment)Standard of expectation (self-assessment) Marking consistencyMarking consistency Commenting, quality, measure against markCommenting, quality, measure against mark Discussion ElementDiscussion Element Need for additional comments – black mark?Need for additional comments – black mark? Reaction to requests / further clarificationReaction to requests / further clarification

1212

Standard of ExpectationStandard of ExpectationSelf-Assess = 52%Self-Assess = 52%

Peer-Assessed as = 58%Peer-Assessed as = 58%ESSAYESSAY Compensated Compensated

Median MarkMedian MarkStudent Student

MarkMarkDifferenceDifference

(absolute)(absolute)

WW 65%65% 57%57% -8% (2)-8% (2)

XX 48%48% 36%36% -12% (6)-12% (6)

YY 53%53% 49%49% -4% (2)-4% (2)

ZZ 61%61% 59%59% -2% (4)-2% (4)

-6.5% (3.5)-6.5% (3.5)

1414

How easy to get mark for marking?How easy to get mark for marking?

Statistically fairly easy to create a mark for Statistically fairly easy to create a mark for marking based upon marksmarking based upon marks

Take into account high and low markersTake into account high and low markers Standard of expectationStandard of expectation Consistency … judge against final mark Consistency … judge against final mark

awarded for an essay (compensated awarded for an essay (compensated median)median)

What about the comments?What about the comments?

1515

Feedback IndexFeedback Index

Produce an index that reflects the quality of Produce an index that reflects the quality of commentingcommenting

Produce an average feedback index for an Produce an average feedback index for an essayessay

Compare against marker in a similar manner Compare against marker in a similar manner to marks analysisto marks analysis

Where does this feedback index come from Where does this feedback index come from and is it valid?and is it valid?

1616

The way to get the feedback index?The way to get the feedback index?

Develop an application??Develop an application??– C-Rater?C-Rater?

Spelling MistakesSpelling Mistakes Similar Meanings?Similar Meanings?

– That was coolThat was cool– Really ChocReally Choc– Really Good EssayReally Good Essay

ManuallyManually

1818

Commonality!!Commonality!!

In the 67 essays that were markedIn the 67 essays that were marked– Only 96 commentsOnly 96 comments– 44% positive and 56% negative44% positive and 56% negative– Highly criticalHighly critical if something not explained properly (21% of total if something not explained properly (21% of total

comments (of which 73% were negative)comments (of which 73% were negative) Better students were more critical than weaker students .. Better students were more critical than weaker students ..

– Better understanding permitting criticism?Better understanding permitting criticism?– Confidence?Confidence?– Hostility?Hostility?

Comments grouped into 10 categoriesComments grouped into 10 categories Need to QUANTIFY these comments .. Feedback indexNeed to QUANTIFY these comments .. Feedback index Create a database holding positive & negative (by Create a database holding positive & negative (by

category)category)

2525

Time Consuming?Time Consuming?

Can we formulate the marking processCan we formulate the marking process

Take away need for quantification process Take away need for quantification process of analyzing commentsof analyzing comments

Is it still peer-assessment if the students are Is it still peer-assessment if the students are told what to say?told what to say?

2727

STUDENT FRED

REFERENCES: Positive ………

Negative …….

Personal Valuation

5, 3, 2, 1

3, 1, 2

2828

Some points outstandingSome points outstanding What should students do if they identify What should students do if they identify

plagiarism?plagiarism? Is it ethical to get students to mark the work of Is it ethical to get students to mark the work of

their peers?their peers? Is a computerised solution valid for all?Is a computerised solution valid for all? At what age / level can we trust the use of peer At what age / level can we trust the use of peer

assessment?assessment? How do we assess the time required to perform How do we assess the time required to perform

the marking task?the marking task?

2929

Peer-Assessment: Peer-Assessment: No marks required, just No marks required, just

feedback?feedback?

Not there yetNot there yet Feedback index results are very positiveFeedback index results are very positive Require more evaluation before totally automated Require more evaluation before totally automated

system is possiblesystem is possible Getting there .. Removing subjectivity in markingGetting there .. Removing subjectivity in marking

Phil Davies School of Computing University of Glamorgan “Super U”

Documents

Transcript of Phil Davies School of Computing University of Glamorgan “Super U”