Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally...
-
Upload
claire-morgan -
Category
Documents
-
view
218 -
download
1
Transcript of Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally...
![Page 1: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/1.jpg)
Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice
Sally Jordan and Janet HaresnapeFaculty of ScienceThe Open University, UK
Assessment in Higher Education Conference, June 2013
![Page 2: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/2.jpg)
This is a practice exchange…
• So please interrupt
Our plan
Previous practice
Drivers for change
What do we mean by formative thresholded assessment?
Evaluation and stumbling blocks
Early findings
![Page 3: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/3.jpg)
The UK Open University• Founded in 1969;• Supported distance learning;• 150,000 students, mostly studying part-time;• Undergraduate modules are completely open entry, so
students have a wide range of previous qualifications;• Normal age range from 18 to ??• 10,000 of our students have declared a disability of
some sort;• 25,000 of our students live outside the UK.
![Page 4: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/4.jpg)
Historic OU Science Faculty practice
• Tutor-marked assignments (TMAs) and sometimes interactive computer-marked assignments (iCMAs) combine together into overall continuous assessment score (OCAS);
• Examination and/or end-of-module assessment (EMA) gives overall examination score (OES).
![Page 5: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/5.jpg)
OCAS is integrated and interactive
![Page 6: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/6.jpg)
![Page 7: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/7.jpg)
![Page 8: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/8.jpg)
![Page 9: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/9.jpg)
![Page 10: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/10.jpg)
Continuous assessment• Has a useful pacing function;
• Students are concerned about the minutiae of the grading;
• Even though for most students their final grade is determined by their performance in the examination or end-of module assessment;
• A considerable amount of time and effort goes into producing new TMAs for each presentation of each module;
• And we don’t always get it right.
![Page 11: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/11.jpg)
Drivers for change• “when assessments serve both formative and
summative purposes… formative work will always be threatened due to the dominance of summative requirements” (Brearley & Cullen 2012 with echoes of Black & Wiliam 1998, Gibbs 2006, Snyder 1971)
• Lack of alignment of tutor and student understanding of our assessment strategies.
• Lack of alignment of tutor and student understanding of the purpose of continuous assessment.
• Saving time and money.
![Page 12: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/12.jpg)
Formative thresholded assessment – two models
• OCAS is formative, but students have to demonstrate engagement by scoring more than 30% in x out of y TMAs and iCMAs; final score is determined by OES alone.
• Students have to reach threshold (usually 40%) for OCAS (overall), but their module result is then determined on the strength of their OES alone.
![Page 13: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/13.jpg)
Risks
• Students will feel insufficiently prepared for the examination or end-of-module assessment;
• Students will not engage sufficiently in the formative thresholded OCAS and so will lose valuable formative opportunity;
• Assignments will be reused, so there may be more plagiarism. For formative assessment, does this matter?
DISCUSSION POINT – what do you think about our move to formative thresholded assessment?
![Page 14: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/14.jpg)
Evaluation
• A series of practitioner-led mini projects;• Strengths: we are practitioners and so close to the
student experience; we will help each other;• Dangers: we are practitioners so we have considerable
pressures of other work; we won’t deliver: we will have insufficient expertise.
DISCUSSION POINT: Does anyone have experience of research or evaluation carried out in this way?
![Page 15: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/15.jpg)
Evaluation• Quantitative and qualitative;• Janet – looking at behaviour of students who are taking
two modules, one with summative and one with formative thresholded assessment;
• Lynda – looking at two new modules with subtly different assessment strategies;
• iCMA usage can give a ‘signature’ of student engagement;
• TMA submission rates (before and after the change of assessment strategy);
• What do students think?• What do tutors think?
![Page 16: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/16.jpg)
Evaluation - complications• We need to compare current with historic data; we need
to compare current with historic student perceptions;
• Confounding variables: other changes to assessment strategy; changing student population; over-committed students.
![Page 17: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/17.jpg)
TMA submission rates
![Page 18: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/18.jpg)
How many TMAs did students omit
![Page 19: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/19.jpg)
Correlations between exam score and omitted TMAs
• 2011J – for students who submitted all TMAs, mean exam score = 57.5%; for those who did not submit all TMAs, mean exam score = 41.3%;
• 2012J – for students who submitted all TMAs, mean exam score = 53.4%; for those who did not submit all TMAs, mean exam score = 44.4%;
• Should this worry us? (DISCUSSION POINT)• Not a causal relationship.
![Page 20: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/20.jpg)
Engagement with iCMAs
![Page 21: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/21.jpg)
Engagement with iCMA feedback – same question, different mode of use
![Page 22: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/22.jpg)
Similar assignment, different students
![Page 23: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/23.jpg)
Outcomes from the evaluation• Overall findings (and thus our plans for the future) – still
too early to say; • A beneficial side effect is that we are thinking about our
assessment strategy for current and future modules in a more coherent manner.
![Page 24: Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open.](https://reader031.fdocuments.net/reader031/viewer/2022032203/56649e025503460f94aeca1c/html5/thumbnails/24.jpg)
Sally JordanSenior Lecturer and Staff Tutor
Deputy Associate Dean, AssessmentFaculty of Science
The Open University
blog: http://www.open.ac.uk/blogs/SallyJordan/