EPEARL An electronic portfolio. ePEARL Electronic Portfolio Encouraging Active Reflective Learning.
1 Encouraging reflective learning through formative assessment: a learning outcomes approach Arlëne...
-
Upload
micah-mabb -
Category
Documents
-
view
214 -
download
0
Transcript of 1 Encouraging reflective learning through formative assessment: a learning outcomes approach Arlëne...
1
Encouraging reflective learning through formative assessment: a learning outcomes approach
Arlëne G. HunterCentre for Open Learning in Maths, Science, Computing and Technology (COLMSCT), The Open University
2
Session overview
• brief overview of the project – development of an online formative assessment tool (SOFA)
• review perceptions of the value of the student online formative assessment as a reflective learning tool– methods of evaluation
– outcomes and methods of engagement
– preliminary conclusions
3
Background to project – the challenge
• new OU level 2 course on Earth System Science – academically course is conceptually challenging
– students need to draw on broad range of skills and prior knowledge (Biology, Chemistry, Earth Sciences, Maths, Physics)
– text co-published with external partner altering standard method of in-text distance learning support
Challenge• devise mechanism to support and inform all stakeholders
of academic progress throughout course
• SOFA: Student Online Formative Assessment
4
Book 1
seven SOFAs linked to Chpts 1-7
each SOFA – 10 questions, 2 levels
Revision SOFA – random selection – specific outcomes
3-stage feedback
end results learning outcomes based– overall result– per learning outcome
mixed format (different learning styles/ skills)
SOFA framework
complexity of learning increases between SOFAs
accessed from course website
5
6
Original goals/objectives• formative
– ‘safe’ learning environment to test depth of learning and application– enhance student awareness of personal strengths, weaknesses,
achievements and progression
• adaptive – match questions to learner competence [logistically challenging]
• integrated– embed SOFAs into assessment, course and programme strategies
via learning outcomes approach– involve all stakeholders in feedback process on academic
progression
• timely – offer ‘just in time’ and self-directed study options
7
Initial challenges and problems• student-course interaction an unknown entity
– how would they engage with an online formative assessment– what academic/ time demands would affect engagement
• question setting – what represents the ‘norm’– adaptivity = 10-15 × question = question bank– dealing with non-standard characters– coping with discursive questions and answers
• stepped learning– designing questions not activities
• time and resource for development
8
Question Types
Question type Method of interaction
assign scientific term free text; drag and drop
MCQ (1 or more answers) radial buttons
complete a table free text; drag and drop
link geological processes/ interactions
free text; drag and drop
label diagrams drag and drop
simple/complex calculations (1-2 steps; sig. figs; sci. not, units…)
numeric/text entry; use or find equation, graph, info…
interpret graphs, diagrams, equations
varied
multi-format combination
9
e.g. Assign scientific term
10
e.g. MCQ
11
e.g. Complete a table
12
e.g. Link geological processes
13
e.g. Label diagram
14
e.g. Two-step calculation
15
e.g. Interpret graphs
16
e.g. Multi-format question
17
Final formative feedback
18
Learning outcomes summary
19
Evaluation of SOFAs• Metadata collected by system (all users)
– number of users (62% in total; range 15-52%; ave 27%), timing and frequency of use per student
– time spent per question/ per assessment (ave ~70 mins)– all answers (analyse common errors ; difference in scores in chapters vs
revision SOFAs)
• Question 11 at end of each SOFA (all users)– quick feedback questions – perceived usefulness: test understanding, measure
progress, revision aid– free-text comments box
• generic questionnaire on Book 1 usage (whole community)– which (if any) SOFAs used– how did they help learning
• Success Case Method targeted interviews – specific subsection of community– exemplar success and non-success cases
20
Success Case Method (SCM) evaluation
• offers quick means of gaining quantitative and qualitative insight
• develop models of absolute success and absolute failure • meaning of success will vary between stakeholders
• failure does not equal absence of absolute success
• collect extreme robust (provable) stories to identify drivers behind experiences
• focus on “what, so what, now what?”
• how to make improvements
21
SCM process
• students and tutors invited to complete generic questionnaire
• gain two perspectives on perceived value of SOFAs
• 12 respondents selected for detailed interviews• ~5% of all SOFA users interviewed* • 9 potential success cases• 3 potential non-success cases
(*based on total of 270 users who accessed at least 1 SOFA between February – May 2007; ideally, 10% is preferred representative quota)
22
SCM Interview ‘buckets’
Success interview buckets
• what was used that worked?
• what results were achieved?
• what good did it do (what was its value)?
• what helped?
• suggestions?
Non success interview buckets• barriers?• suggestions?
23
SCM – success case summaryWhat was used that worked?• access via course website ‘Check your understanding...’
• repeat function and ‘changing questions’
• instant feedback on effectiveness of their learning approach
What results were achieved?• obtained ‘impartial’ insight into their strengths, weaknesses and
misunderstandings
• demonstrated ongoing progress
• instant pacing mechanism – review learning approach/materials
What was its value?• focussed learning and helped direct areas in need of ‘extra work’
• built confidence in personal abilities and progress
• provided ‘real’ measure of personal achievement
formative, integrated, adaptive
adaptive, timely, formative
formative, timely, integrated
24
SCM – success case summary cont/.What helped?• flexible, formative structure – instantly adaptable to own needs targeted
feedback on specific errors and omission
• positive affirmation of good work
• putting all learning (content and skills) into context
Suggestions• printable version of questions and final explanation for revision
BUT recognition that targeted feedback needs SOFA to be online
General benefits• consolidated and clarified learning
• identified gaps, misunderstandings, “unknown unknowns” before it was too late and offered advice on how to improve
• shift from seeing reflective feedback to using focussed feedback
formative, integrated
formative, adaptive, integrated, timely
formative, adaptable
25
SCM – non success case summaryBarriers – general issues• working online – prefer paper, more flexible, more familiar
• academic level – too high, so demoralising when wrong
• time pressures – too much to do in course; going online is time consuming
• accessing answers – prefer immediate access to read alongside questions
Barriers to value• expected to get all questions right first time – SAQ experience
• did not like being told ‘wrong’ or to ‘try again’
• too many clicks to get to the final answer
Suggestions• focus on prior background skills and knowledge – gap filling
• downloadable/ print-based – question and answer immediately accessible
• colourful, fun and tactile formative, adaptive
26
Preliminary outcomes• majority of students who tried them, liked the SOFAs and found
them helpful– but only reached on average 27% (total usage 62%); 15% did all chapter SOFAs– how could ‘remainder’ be encouraged to engage and continue to use
• individual students who did the SOFAs were repeating them– quantitative data indicated general improvement in application of learning (content and
skills) on repeats
• qualitative feedback demonstrates more confident learners – greater personal awareness of efficiency of learning approach– growing confidence in personal skills and sense of academic progress– awareness of personal strengths, weaknesses and learning needs – changing perception from reviewing feedback (i.e. what was wrong) to focussing on
feeding forward (i.e. what needs to be developed)
• majority (success and non-success cases) still wanted ‘hard copy’ version for revision– outcome of formative versus summative usage?– familiarity with paper over PC?
formative, timely
formative, adaptive, integrated, timely
? formative, adaptive
formative, adaptive
Dr Arlëne G. HunterTeaching FellowCentre for Open Learning in Maths, Science, Computing and Technology (COLMSCT)The Open UniversityWalton HallMilton KeynesEngland. MK7 6AA
28
Links – open access to SOFAsChapter specific SOFAs – Book 1
• https://students.open.ac.uk/openmark/s279-07.book1chapter1world/
• https://students.open.ac.uk/openmark/s279-07.book1chapter2world/
• https://students.open.ac.uk/openmark/s279-07.book1chapter3world/
• https://students.open.ac.uk/openmark/s279-07.book1chapter4world/
• https://students.open.ac.uk/openmark/s279-07.book1chapter5world/
• https://students.open.ac.uk/openmark/s279-07.book1chapter6world/
• https://students.open.ac.uk/openmark/s279-07.book1chapter7world/
Revision SOFAs – random selection of questions• https://students.open.ac.uk/openmark/s279-07.book1chapter8ran1world/• https://students.open.ac.uk/openmark/s279-07.book1chapter8ran2world/• https://students.open.ac.uk/openmark/s279-07.book1chapter8ran3world/• https://students.open.ac.uk/openmark/s279-07.book1chapter8ran4world/
29
Working with diagrams• https://students.open.ac.uk/openmark/s279-07.book1chapter8diagmodworld/ • https://students.open.ac.uk/openmark/s279-07.book1chapter8diagtaxingworld/
Understanding geological processes• https://students.open.ac.uk/openmark/s279-07.book1chapter8geoprocessmodworld/ • https://students.open.ac.uk/openmark/s279-07.book1chapter8geoprocesstaxingworld/ • https://students.open.ac.uk/openmark/s279-07.book1chapter8geotermsworld/
Using geological terms• https://students.open.ac.uk/openmark/s279-07.book1chapter8mathmodworld/ • https://students.open.ac.uk/openmark/s279-07.book1chapter8mathtaxingworld/
Summarising information with tables• https://students.open.ac.uk/openmark/s279-07.book1chapter8tablesworld/
Links – open access to SOFAs