Online peer assessment: an exploratory case study in a higher education civil engineering course
-
Upload
goncalo-cruz-matos -
Category
Education
-
view
461 -
download
0
description
Transcript of Online peer assessment: an exploratory case study in a higher education civil engineering course
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
Online peer assessment: an exploratory case study in
a higher education civil engineering course
Gordon GramsDep. Letters, Arts and Communication
UTAD - University of Trás-os-Montes e Alto DouroVila Real, [email protected]
Daniela PedrosaFaculty of Psychology and Educational Sciences
UC - University of CoimbraCoimbra, [email protected]
Gonçalo Cruz, Ana MaiaPro-Chancellery fo Innovation & Information Management
UTAD - University of Trás-os-Montes e Alto DouroVila Real, Portugal
{goncaloc, margaridam}@utad.pt
Caroline DominguezDep. Engineering
C-Made/UTAD - University of Trás-os-Montes e Alto DouroVila Real, Portugal
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study
II. Caracterization of the activity
III. Research methodology
IV. Evaluation
V. Discussion and limitations
VI. Conclusions and future work
Agenda
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study
II. Caracterization of the activity
III. Research methodology
IV. Evaluation
V. Discussion and limitations
VI. Conclusions and future work
Agenda
What we are
talking
about? What was our
aim?
What literature says?
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study – “What we are talking about?”
Civil engineering students were challenged to reinforce a set of cognitive and personal competencies, using a web-based collaborative writing environment and a peer assessment approach..
It’s all about learning...
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study – “What we are talking about?”
Civil engineering students were challenged to reinforce a set of cognitive and personal competencies, using a web-based collaborative writing environment and a peer assessment approach..
It’s all about teaching...
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study – “What we are talking about?”
Civil engineering students were challenged to reinforce a set of cognitive and personal competencies, using a web-based collaborative writing environment and a peer assessment approach..
It’s all about pedagogy...
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study – “What literature says?”
• Effective communication skills is considered to be one of the main goals of engineering curricula.
• Students’ resistance to peer review, but only a few conclusions on which factors lay behind that resistance..
• Little understanding on the impact of students’ perceptions of peer reviewed work.
• When feedback is given only by peers it seems to be less enforcing.
• Feedback provides guidelines when it is specific and clear.
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study – “What literature says?”
Adapted from Nelson, M. M., & Schunn, C. D. (2009). The nature of feedback: How different types of peer feedback affect writing performance. Instructional Science, 27(4), 375-401.
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study – “What was our aim?”
a. Examine students’ attitudes towards online peer assessment
b. Evaluate students’ written communication skills• The type of feedback implemented by student-reviewers.
• The overall students’ appreciation: activity tasks; web-based learning environment; learning support materials; and received feedback.• The related factors with students’ use and appreciation of received feedback.
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study
II. Caracterization of the activity
III. Research methodology
IV. Evaluation
V. Discussion and limitations
VI. Conclusions and future work
Agenda
The context and the related competencies
The activity
tasks
The learning support materials
What was the approach?
The web-
based
learning
environment
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The context and the related competencies
Midcourse level (3rd semester of a 6-semester program) of a Bachelor of Civil Engineering course, focusing only on one component of the syllabus of Management of Construction Enterprises Curricular Unit.Familiarize students in the exercise of the SWOT (Strengths, Weaknesses, Opportunities, and Threats) economic strategic analysis approach.
Students had to identify and characterize the opportunities and threats that construction companies (and other agents) faces according its economic, technological, legal, political and social-cultural environment.
Reinforce/develop writing skills, synthesis, analysis, critical thinking, autonomy, interpersonal communication, use of ICT and group work.
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The activity tasks
1st
week
•Select an economic article from one available source, dealing with recent news of an economic phenomenon;
•Perform a summary of the article;
•Analyze the article, identifying and classifying the type of variables;
•Evaluate them in terms of Opportunities and Threats;
•Give his/her own opinion on the reported news;
For the authors
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The activity tasks
2nd
week
•Read the document and give written feedback with suggestions for improvement
•Evaluate the colleague’s work on a 0 to 1 grading scale, taking into account evidence of the summary, the identification of all the variables of the economic environment, the conclusions in terms of opportunities and threats, and the elaboration of a well-grounded final individual commentary.
For the reviewers
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The activity tasks
2nd
week
•Made a revision of the work done by the author and the reviewer colleague.
•Evaluate both of them with the same grading scale (0 to 1).
For the teacher
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The activity tasks
3rd
week
•Make voluntary improvements to his/her original document.
For the authors
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The activity tasks
4th
week
•Give a final grade to the authors.
For the teacher
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The activity tasks
2 Cycles during a semester
The final grade was the average from the teacher’s final grades given to both assignments.
One third of the total grade of the curricular unit (in a grading scalefrom 0 to 20 values).
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The web-based learning environment
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The web-based learning environment
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The web-based learning environment
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The web-based learning environment
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The learning support materials
Guidelines
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The learning support materials
Management of the activity (1)
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
II. Caracterization of the activity – The learning support materials
Management of the activity (2)
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study
II. Caracterization of the activity
III. Research methodology
IV. Evaluation
V. Discussion and limitations
VI. Conclusions and future work
Agenda
Which instruments? For what data?Which steps?
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
III. Research methodology – “Which instruments? For what data?”
Two different instruments
1. Questionnaire
2. Students’ assignments and grades
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
III. Research methodology – “Which instruments? For what data?”
Two different instruments
1. Questionnaire
46 quantitative questions:
•12 were related to the characterization of students’ profile (sex, age, previous digital skills);
•17 with the pedagogical approach adopted (execution of the tasks, facility of use the online environment, time availability and support material);
•6 dealt with the students’ perception about the quality of the feedback (its value and impact to the writing improvement);
•10 on the acquired skills and the general satisfaction with the activity;
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
III. Research methodology – “Which instruments? For what data?”
Two different instruments
1. Questionnaire
4 qualitative open questions allowed students to state why they found (or not) important the suggestions made by the peer-reviewers and by the teacher, and why they used (or not) these suggestions to improve their own work.
After the data collection, a simple descriptive statistical analysis was made for the quantitative questions, limited to counting frequencies of the response modalities for each question.
For the qualitative questions, the nature of each response was verified through a structured content analysis grid.
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
III. Research methodology – “Which instruments? For what data?”
Two different instruments
2. Students’ assignments and grades
The type of feedback made by student-reviewers and by the teacher was analyzed in each document, using the feedback model adapted from Nelson & Schunn (2009). An analysis grid corresponding to a checklist of analyzed criteria was elaborated.
A total of 39 assignments (41%) of 81 were selected and analyzed. They all had feedback given, both by peers and by the teacher.
Finally, the average of the intermediate grades given by reviewers and teachers was calculated for additional information.
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
III. Research methodology – “Which steps?”
1. Sample characterization;
2. Quantitative and qualitative analysis of student perceptions;
3. Description of the type of feedback found in the final documents with a comparison of the grades given by student-reviewers and the teacher;
Three main steps
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study
II. Caracterization of the activity
III. Research methodology
IV. Evaluation
V. Discussion and limitations
VI. Conclusions and future work
Agenda
Use of the Google Docs environment
Characterizati
on of the
sample
Students’
perceptions
based on
quantitative
data
Type of
feedback and
grades given
by students
Students’ perceptions based on qualitative data
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Characterization of the sample
55% of the students were male.
Previously, most of the students used social networks and e-mail on a daily base (73% and 77%, respectively).
Only one student already knew about or had previously used Google Docs or a similar online environment.
95% of the students performed all the proposed mandatory tasks.
23 students from a total of 47(49%)...
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Use of the GoogleDocs environment
All of the students (100%) considered Google Docs to be a useful and efficient platform for the proposed tasks;
85% agreed that Google Docs made communication between colleagues and the teacher easier;
77% of the students evaluated the support documents for the activity (tables and guideline document) positively, allowing better organization, coordination and management of the activity;
55% considered the time available to perform the activity to be long enough;;
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Students’ perceptions based on quantitative data
You can see the detailed analysis here:
Cruz, G.; Dominguez, C.; Pedrosa, D.; Maia, A. (2012). A avaliação entre pares com recurso ao Google Docs: um estudo de caso exploratório num curso de Licenciatura em Engenharia Civil, in Livro de Atas - Conferência Ibérica em Inovação na Educação com TIC – Bragança.
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Students’ perceptions based on qualitative data
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Students’ perceptions based on qualitative data
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Students’ perceptions based on qualitative data
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Students’ perceptions based on qualitative data
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Students’ perceptions based on qualitative data
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Students’ perceptions based on qualitative data
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Type of feedback and grades given by students
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Type of feedback and grades given by students
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Type of feedback and grades given by students
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
IV. Evaluation – Type of feedback and grades given by students
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study
II. Caracterization of the activity
III. Research methodology
IV. Evaluation
V. Discussion and limitations
VI. Conclusions and future work
Agenda
What we can
draw?What we can improve?
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
V. Discussion and limitations – “What we can draw?”
The use of an online environment did not represent any difficulty for students, contrary to some literature. The support material prepared proved to be very important for the realization of the activity.
Students have a satisfactory level of written communication skills. However, some negative perceptions of feedback remain due to the lack of clarity in some cases, non specification of problems and their location, as well as no provision of solutions and explanations.
Thus, from a pedagogical point of view, it seems necessary to perform one or more training sessions with students on “what are the characteristics of a good feedback.” These in-class practice evaluation sessions can use a demonstration of good practices, models, key aspects, etc., with discussion on how to improve the arguments made .
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
V. Discussion and limitations – “What we can draw?”
Another factor which appears to influence the use (or non use) of peer or teacher feedback is the evaluation grades.
The small number of students who used the feedback to improve their work in the analyzed documents (= 3) and the high average (0.86 in a scale from 0 to 1) of the intermediate and final grades assigned by the teacher to each document leads us to consider the importance of testing the following question in future research:
How does the intermediate grade evaluation by the teacher influence the use of peer feedback by the student author for improving his/her work?
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
V. Discussion and limitations – “What we can draw?”
The final important factor in our analysis appears to be teacher status, specifically the overvaluation of teacher feedback compared with the reviewers,' as other studies already have demonstrated.
However, when we look at the grades assigned by the teacher and the peers, there was not a significant difference.
These results lead us to question the need to perform graded assessment of work at the intermediate level by both the teacher and peers (although this would require awareness and credibility of their own grading ability from the students, contrary to their current beliefs).
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
V. Discussion and limitations – “What we can improve?”
The large number of responses listed in the sub-category of “general perceptions” did not allow a good comprehension of the students' reasons for appreciating or using feedback. This results from a limitation of the research methodology, which was based exclusively in the use of a single instrument of data collection (in this case, the questionnaire). To overcome this problem, we suggest the use of other instruments and / or data collection methods such as personal or focus group interviews.
No data correlation between the instruments (questionnaire was anonymous).
No blind peer-review.
The adopted grading scale (students didn’t know what are the difference, for example, between a 0.7 grade to a 0.8).
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
I. Purpose of the study
II. Caracterization of the activity
III. Research methodology
IV. Evaluation
V. Discussion and limitations
VI. Conclusions and future work
Agenda
What’s the
final remarks?
What’s the next step?
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
VI. Conclusions and future work – “What’s the final remarks?”
Digital competencies do not represent any difficulty for the realization of the activity. However, good preparation and support guidelines are essential for its success.
Relevant factors underlying the students' appreciation and usefulness of received feedback, in particular by the peer reviewers, were identified: type of feedback, influence of the teacher status (versus peers), and grades.
The written communication skills seems to be quite developed and applied, although they can be improved at the level of summarization and provision of solutions. Thus, it is important to spend more time in developing skills on how to deliver good feedback, improving effective communication, for example by demonstrating successful models and case studies on feedback as well as the benefits of this type of activity.
The low rate of voluntary feedback leads us to question the influence of intermediate grading in the process, which could be better investigated in future research.
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
VI. Conclusions and future work – “What’s the next step?”
We are deepen the analysis on peer feedback, moving the research to other dimensions such as the effect of online peer assessment on critical thinking, and designing an research proposal will allow for follow up of each student’s results during the next academic semester.
We are actually doing that..
3 Groups of students: Grades + Comments; Only grades; Only comments;Cornell Critical Thinking Test (X) (1985) - to apply as a Pre- and Pos-testEnnis Critical Thinking Taxonomy (1985) - to measure the learning outcomesEnnis FRISCO Approach (1985) - to promote critical thinking competencies
SPEE Special Session "Talking about Teaching 2012"
(TAT'12)26 - 28 September 2012, Villach, Austria
Thanks for listeningHope to hear from you...
@GCruzMatos or @elearning_utad
Facebook.com/goncalocruzmatos
Msu Speiser