Tracking a Dose-Response Curve in Peer Feedback on

WritingA Work in Progress

Christina HendricksCo-Investigator: Jeremy Biesanz

University of British Columbia-Vancouver

SoTL Symposium, November 2015

Slides available here: http://is.gd/PFBwriting2015Slides licensed CC-BY 4.0

Literature on peer feedbackReceiving peer feedback improves writing (Paulus, 1999; Cho & Schunn, 2007; Cho & MacArthur, 2010; Crossman & Kite, 2012)

Giving peer feedback improves writing(Cho & Cho, 2011; Li, Liu & Steckelberg, 2010)

GAPS: Most studies look at revisions to a single essay, not changes across different essays

Draft 1

Draft 2

Draft 3

Essay 1 Essay 2 Essay 3 Essay 4 Essay …n

PFB

PFB

PFB

PFB PFB PFB

Few studies look at “dose-response curve”

Pilot study research questions1. How do students use peer comments given and

received for improving different essays rather than drafts of the same essay?

2. Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal?

3. Does the quality of peer comments improve over time?

• Interdisciplinary, full year course for first-years

• 18 credits (6 each in 1st year English, History, Philosophy)

• Students write 10-12 essays (1500-2000 words)

• Peer feedback tutorials every week (4 stdnts)

http://artsone.arts.ubc.ca

Toni Morrison, Wikimedia Commons, licensed CC BY-SA 2.0

Osamu Tezuka, public domain on Wikimedia Commons

Jane Austen, public domain on Wikimedia Commons

Friedrich Nietzsche, public domain, Wikimedia Commons

Data for pilot study 2013-2014

• 10 essays by 13 participants (130 essays)

• Comments by students in tutorial group (4 in group) on all essays (n=1219)

• Comments by instructor on all essays (n=3331)

• All essays and comments coded according to a common rubric

Coding RubricCategories(plus subcategories, for 11 options)

• Strength of argument• Organization• Insight• Style & Mechanics

Numerical value

1: Significant problem2: Moderate problem3: Positive comment/praise

E.g., STREV 2: could use more textual evidence to support your claims

Progress so farAll student and instructor comments coded60 of 130 essays coded -- (10 essays each by 6 of the 13 participants)

Inter-coder reliability

Fleiss’ Kappa Intra-class correlation

Student comments (n=141)

All categories: 0.61 (moderate)Most used categories: 0.8 (excellent)

0.96 (excellent)

Essays (n=60) 0.68 (adequate)

3 coders:• Daniel Munro & Kosta Prodanovich

(undergrads, former Arts One)• Jessica Stewart (author, editor)

LOOKING AT TRENDS IN COMMENTS OVER TIME

INSTRUCTORComments

-.28**

Strength

Style

Organiz.

Insight-.04*

Number of 2 comments in each categ.

STUDENT comments

Strength

Style

Organiz.

Insight

-.16**

Number of 2 comments in each categ.

INSTRUCTORComments .31***

Strength

Style

Organiz.

Insight

.08**

.19**

.11**

Number of 3 comments in each category

STUDENTComments

Strength

Style

Organiz.

Insight

Number of 3 comments in each categ.

Student/instructor agreement on comments

• Average numerical ratings for comments across all categories agree strongly between student and instructor (.48****)

• However, this agreement

goes down across essays (-.04**)o This is because student ratings increase

over time at only half the rate that instructor ratings do

*p < .05, **p< .01, ***p< .001, ****p < .0001

HOW DOES ESSAY QUALITY CHANGE OVER TIME?From this slide onwards, we are looking only at 60 essays coded so far, out of the set of 130 (essays from 6 of the 13 participants)

Essay quality

improves linearly

in 60 essays

Mean essay quality rating 4.12 out of 7, SD = .62

MORE COMPLEX ANALYSES

Cross-lagged panel design with auto-regressive structure

Essay QualityTime 1

Essay QualityTime 2

CommentsTime 1

CommentsTime 2

B

A

C

D

E… N

… N

Path A: Student comments

Essay QualityTime 1

Essay QualityTime 2

CommentsTime 1

CommentsTime 2

B

A

C

D

E… N

… N

Significant effects:• Ratings of 2 in Insight (-.53*)• Ratings of 3 in Strength (.25*)

Path C: instructor commentsEssay Quality

Time 1Essay Quality

Time 2

CommentsTime 1

CommentsTime 2

B

A

C

D

E… N

… N

Significant effects • Ratings of “2” in Strength (-.31**)• Ratings of “3” in Strength (.51***) and

Style/Mechanics (.34**)

Path C: student commentsEssay Quality

Time 1Essay Quality

Time 2

CommentsTime 1

CommentsTime 2

B

A

C

D

E… N

… N

Significant effects• Ratings of “2” in Style (.28*) and Insight

(.15*)• Ratings of “3” in Strength (.30*)

Path D: Student comments

Essay QualityTime 1

Essay QualityTime 2

CommentsTime 1

CommentsTime 2

B

A

C

D

E… N

… N

Significant effects:• Ratings of 2 in Organization (.16*)• Ratings of 3 in Style (.16*)

Research question 2How do students use peer comments given and received for improving different essays rather than drafts of the same essay?

o Not enough evidence yet to say much about path D

o Haven’t yet looked at differences in comments given vs. received

Research question 2Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal?

o No evidence yet that there is any change over time in path D

Research question 3Does the quality of peer comments improve over time?o No evidence yet that there is any change

over time in path A

Essay QualityTime 1

Essay QualityTime 2

CommentsTime 1

CommentsTime 2

B

A

C

D

E… N

… N

Some conclusions so far• Pilot study: is this sort of study

feasible for larger sample?o Yes, but probably more so if instructors

code essay quality rather than coders

• Facilitating easy collection of student & instructor comments is difficult

References• Cho, K., & MacArthur, C. (2010). Student revision with peer and

expert reviewing, Learning and Instruction. 20, 328-338.• Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving

comments. Instructional Science, 39, 629-643. • Cho, K. & Schunn, C. D. (2007). Scaffolded writing and rewriting

in the discipline: A web-based reciprocal peer review system. Computers & Education, 48, 409–426

• Crossman, J. M., & Kite, S. L. (2012). Facilitating improved writing among students through directed peer review, Active Learning in Higher Education, 13, 219-229.

• Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536.

• Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal of Second Language Writing, 8, 265-289.

Thank you! Christina Hendricks

University of British Columbia-Vancouver

Website: http://blogs.ubc.ca/christinahendricksBlog: http://blogs.ubc.ca/chendricksTwitter: @clhendricksbc

Slides available: http://is.gd/PFBwriting2015

Slides licensed CC-BY 4.0