Student-Oriented Meta-Assessment in a Project-Based Systems Engineering Course 1 Niva Wengrowicz 1...
-
Upload
ursula-lang -
Category
Documents
-
view
219 -
download
3
Transcript of Student-Oriented Meta-Assessment in a Project-Based Systems Engineering Course 1 Niva Wengrowicz 1...
1
Student-Oriented Meta-Assessment in a Project-Based
Systems Engineering Course
Niva Wengrowicz1
Yehudit Judy Dori1,2
Dov Dori1,2
1-Technion, Israel Institute of Technology2-MIT, Massachusetts Institute of Technology
3
The Goal of Engineering Education
engineering education should promote among other things,
teamwork and project-based learning (PBL)
Modern engineering education programs aim to enrich students with the necessary
knowledge, skills, and attitudes for becoming successful young engineers
NSF determined that
4
Project-Based Learning (PBL)• The PBL approach emphasizes students' project
construction to represent what they learn.
• PBL is characterized by:– Active learning– No single correct solution, but rather
an open-ended driving challenge– Collaboration among peers• Our course is based on students’ projects of constructing
conceptual modeling
6
Conceptual Models
• In systems engineering, a conceptual model is necessary for:
– coherent thinking– sharing ideas– providing common ground for communication– solving problems jointly.
In our course we use two leading conceptual
modeling languages
UML and OPM
Model Quality Assessment
8
• Categories to evaluate students’ conceptual models:
– Model clarity and understandability (MCU) check if the model can be easily understood
– Model correctness (MCorr) evaluation of the model’s syntax, structure and behavior (logic) correctness.
– Model completeness (MComp) ensuring that the model contains all the requirements included in the scope .
– Model documentation (MDoc) appropriateness and contribution of the textual explanations related to the model to clarify the model’s constructing considerations
9
Peer & Meta Assessment
• Peer assessment is a method in which students are asked to evaluate each other's work.
• It helps students develop high order thinking skills including:– analyzing– generalizing– evaluating – critical thinking.
Meta assessment: Assessment of assessment
Evaluating the level of each student’s peer assessment
10
Peer & Meta Assessment
11
Research Goal
Overall research goalTo develop and assess a new approach for teaching and assessing undergraduate students in large PBL
systems engineering courses.
Current study goalTo develop and validate a supporting
Web-based tool for meta-assessment (assessment of peers’ assessment)
12
Research Method - Participants
About 130 students (half females) who participated in the course
Specification and Analysis of Information Systems at the Technion, Israel Institute of Technology.
The course objective is to familiarize 5th Semester Industrial Engineering and Information Systems Engineering undergraduate students with analysis, modeling, design, and assessment of systems in general and of information systems in particular.
13
Research Method - Setup
• PBL approach – reverse engineering of complex Web-based system, such as Gmail, Expedia, or eBay
– Realistic problems
– No single correct answer
• Groups of students were asked to specify, analyze
and model these complex system in both – OPM and
UML.
14
Research Method - Setup
• There were 23 groups of six (some of five) students divided into two teams of three (some of two).
• OPM and UML were taught every other week alternately.
GroupGroupGroupGroupGroupGroup
Team 1 Team 223
15
Research Method - Setup• crossover method –
– The first team started to model the system using OPM,while the second team started to model the same system using UML.
– In the middle of the semester, the teams swapped, and each team continue the model that the other team in the group had started.
G01 - Gmail
Team 1 Team 2
OPM
UML
UML
OPM
17
Research Method – Tools
Each student was tasked with assessing a two systems.
A dedicated Web-based peer assessment tool was developed
18
Research Method – Tools
Each student was tasked with assessing a two systems.
A dedicated Web-based peer assessment tool was developed
Students assessment was based on a list of criteria for each category.
19
Research Method - Tools
• The online assessment included:– a written assessment
– a corresponding grade in a scale of 1-10.
• To prevent students from giving 10 to all four models, the tool prevents giving any grade more than once to any one of the four models in each category
A dedicated Web-based peer assessment tool was developed
Research Method – Tools
• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment
• The inputs were:
− Student’s grading for each model
− Student’s explanations for the grading
− Model’s descriptive statistics based on other students grading
Research Method – Tools
• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment
• The inputs were:
− Student’s grading for each model
− Student’s explanations for the grading
− Model’s descriptive statistics based on other students grading
Research Method – Tools
• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment
• The inputs were:
− Student’s grading for each model
− Student’s explanations for the grading
− Model’s descriptive statistics based on other students grading
Research Method – Tools
• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment
• The outputs are:
− Grade for each categorywas given to the student based on his grades & explanation to all 4 models
− Explanation for the grading
Research Method – Tools
• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment
• The outputs are:
− Grade for each categorywas given to the student based on his grades & explanation to all 4 models
− Explanation for the grading
Research Method – Tools
• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment
• Evaluating scale:
−A list of available gradesand compatible comments.
26
Research Method – ToolsMeta-assessment grading scale • We created a grading scale
based on themes that had emerged from students' explanation combined with Bloom's revised taxonomy.
Research Method – ToolsMeta-assessment grading scale
Evaluation Evaluation definition Examples
Satisfactory = 60 Basic understanding – several basic summarized criteria with some diagram examples.
On overall the work is nice but some arrows and links cross each other (SD0, SD1) and makes it hard to read as well as the SD1.1 diagram included more than 5 processes and it is over loaded.
Good (-)= 75 Good understanding – several criteria with some diagram examples – mismatch between the explanation and grading.
Sometimes it's hard to read the diagrams because arrows cross each other (fig. 1.3, 1.5) and the caption on the lines is too small (fig. 1.1, 1.2). Additionally, some diagrams include unused objects (fig 1.4, 1.5)- adding unnecessary information can create confusion
• Three different aspects with examples that reinforce the main claim. •Not clear why this model was graded as 10?
Research Method – ToolsMeta-assessment grading scale
Evaluation Evaluation definition ExamplesExcellent = 100 Complex understanding – high
analyzing ability – several criteria with diagram examples – Including recommendations for improvement.
There is unfolding mixed with in zooming (Fig 2.3, 2.4), For better understanding- it should be separate.
… maybe I would use more tagged structural links for better explanation (Fig.6).
Complex understanding –ability to synthesis & conclude – several criteria with several different types of diagrams examples – drawing conclusions based on criteria (findings meaning).
A few mistakes that don't seem to indicate a big lack of understanding (one object is not connected to a process, XOR link missing-Fig.6), overall- the model seems to be correct.
Complex understanding – ability to evaluate & compare - several criteria with several different type of diagram examples - drawing conclusions based on comparing to the other works (grading explanation).
… (Fig 2.5,2.7). Overall- more mistakes then the other OPM team, BUT their diagrams are more detailed and informative, that uses correct states of object's and tagged structural links which makes it better in my opinion.
•Finding generalization with examples and conclusions•Critical thinking.
30
Research Method – ToolsMeta-assessment grading scale • We created a grading scale
based on themes that had emerged from students' explanation combined with Bloom's revised taxonomy.
• Reliability of the grading scale was tested by five judges.
• They were asked to evaluate several identical answers based on the grading scale.
Correlation between judges ranged from .78 to .93 and were significant (p<.01).
Research QuestionsFor validating the meta-assessment tool we asked four questions:
1 Is there a significant correlation between the students' meta-assessment grades and their prerequisite grades in software engineering course?
2 Is there any effect of the student's level of involvement in teamwork on student's meta-assessment grade?
3 Is there any gender effect on students' meta-assessment grades?
4 Is there any learning order effect on students' meta-assessment grades?
Criterion Validity
construct validity by group comparison
absence of bias
absence of bias
32
Results
The two grade variables were significantly correlated
r(128)=.69, p<.01.
This finding indicates empirical validity by criterion
The 1st research question was tested by a Pearson correlation test
Criterion Validity
33
Results
Main Effect n mean (sd) F(1,123) p
Teamwork involvement level
33% or lower 56 65.02 (18.06)18.65 P<.001
Greater than 33% 75 77.99 (16.15)
GenderMale 66 73.33 (19.02)
1.48 n.s.Female 65 71.54 (17.23)
Learning order (first modeling language)
UML 65 70.77 (18.77).22 n.s.
OPM 66 74.09 (17.41)
Meta assessment grade - means, standard deviation and the three way ANOVA main effect results
The two to four research questions were tested bythree-way ANOVA test
construct validity by group comparison
34
Results
Main Effect n mean (sd) F(1,123) p
Teamwork involvement level
33% or lower 56 65.02 (18.06)18.65 P<.001
Greater than 33% 75 77.99 (16.15)
GenderMale 66 73.33 (19.02)
1.48 n.s.Female 65 71.54 (17.23)
Learning order (first modeling language)
UML 65 70.77 (18.77).22 n.s.
OPM 66 74.09 (17.41)
Meta assessment grade - means, standard deviation and the three way ANOVA main effect results
The two to four research questions were tested bythree-way ANOVA test
construct validity by group comparison
absence of bias
35
Results
Main Effect n mean (sd) F(1,123) p
Teamwork involvement level
33% or lower 56 65.02 (18.06)18.65 P<.001
Greater than 33% 75 77.99 (16.15)
GenderMale 66 73.33 (19.02)
1.48 n.s.Female 65 71.54 (17.23)
Learning order (first modeling language)
UML 65 70.77 (18.77).22 n.s.
OPM 66 74.09 (17.41)
Meta assessment grade - means, standard deviation and the three way ANOVA main effect results
The two to four research questions were tested bythree-way ANOVA test
absence of bias
absence of bias
construct validity by group comparison
36
Conclusions
We developed and validated the meta-assessment tool
Content validation• three experts • correlation with other outcomes
structure validation• correlation between five judges (reliability of the scale)• group comparison (construct validity)
Meta assessment
tool
Absence of bias (reliability of the tool)• by gender• by order of using the modeling languages
37
Discussion• Meta assessment which is an extension for peer assessment
was found to be reliable and valid.
• Peer & meta assessment method enables to:– evaluate the individuals that take part in executing a team-based project. – promotes students' application of higher order thinking skills and
development of their metacognitive skills such as:
• Planning the execution of the task
• Allocating the optimal amount of time
• Monitoring and evaluating strategies
• Reflecting their project constructing
• Critical thinking
• Analyzing
• Synthesizing
• Evaluating the quality of each model
• Comparing between models
38
Further Study
We call for additional studies to:
– Validate the assessment tool in additional courses and countries
– Use NLP (natural language processing) methods to analyze large amount of peer assessments.
39
ACKNOWLEDGEMENT
This work was funded by:
– the EU FP7 Project #262044-VISIONAIR-Vision Advanced Infrastructure for Research
– the Gordon Center for System Engineering at the Technion
– the Technion Post-Doctoral Fellowships