Automatic feedback for motivation and self-regulation
Professor Denise Whitelock
The Open University, Walton Hall,
Milton Keynes MK7 6AA, UK
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
I hate marking but want the tasks and feedback to assist student learning
DMW UOC Open Seminar May 2013
The e-Assessment and automatic feedback Challenge
• Constructivist Learning – Push
• Institutional reliability and accountability – Pull
.
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
MCQs: Variation on a theme
DMW UOC Open Seminar May 2013
Example of LAPT Certainty-Based Marking, UK cabinet ministers demo exercise showing feedback, University College, Tony Gardner-Medwin
Drug Chart Errors and Omissions, Medicines Administration Assessment, Chesterfield Royal Hospital
MCQs: High Stakes Assessment
DMW UOC Open Seminar May 2013
Example of practice Thinking Skills Assessment" (TSA) question, Admissions Interview, Cambridge Assessment, Steve Lay
Example of practice Thinking Skills Assessment" (TSA) feedback, Admissions Interview, Cambridge Assessment, Steve Lay
Scaffolding and High Stakes assessment
• Math for Science
• Tutor less course
• Competency led
• No point to cheat
• Web home exam
• Invigilation technologies
DMW UOC Open Seminar May 2013
Self diagnosis
• Basic IT skills, first year med students (Sieber, 2009)
• Competency based testing
• Repeating tests for revision
• Enables remedial intervention
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
Characteristics Descriptor
Authentic Involving real-world knowledge and skills
Personalised Tailored to the knowledge, skills and interests of each student
Negotiated Agreed between the learner and the teacher
Engaging Involving the personal interests of the students
Recognise existing skills Willing to accredit the student’s existing work
Deep Assessing deep knowledge – not memorization
Problem oriented Original tasks requiring genuine problem solving skills
Collaboratively produced Produced in partnership with fellow students
Peer and self assessed Involving self reflection and peer review
Tool supported Encouraging the use of ICT
Elliott’s characteristics of Assessment 2.0 activities
Authentic Assessment: Building e-portfolios on a chef’s course
DMW UOC Open Seminar May 2013
food preparation for e-portfolio, Modern Apprenticeship in Hospitality and Catering,
West Suffolk College, Mike Mulvihill
Evidence of food preparation skill for e-portfolio, Modern Apprenticeship in Hospitality and Catering, West Suffolk College, Mike Mulvihill
Peer Assessment and the WebPA Tool
• Loughborough (Loddington et al, 2009)
• Self assess and peer assess with given criteria
• Group mark awarded by tutor
• Students rated:• More timely feedback
• Reflection
• Fair rewards for hard work
• Staff rated:• Time savings
• Administrative gains
• Automatic calculation
• Students have faith in the administrative system
DMW UOC Open Seminar May 2013
Mobile Technologies and Assessment
• MCQs ,PDAs Valdiva & Nussbaum(2009)
• Polls,instant surveys
• Simpson & Oliver (2007)
• Draper (2009) EVS
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
Gains from Formative Assessment
• Mean effect size on standardised tests between 0.4 to 0.7 (Black & Williams, 1998)
• Particularly effective for students who have not done well at school http://kn.open.ac.uk/document.cfm?docid=10817
• Can keep students to timescale and motivate them
• How can we support our students to become more reflective learners and engage in formative assessment tasks?
DMW UOC Open Seminar May 2013
Collaborative formative assessment with
Global Warming
DMW, Institute of Educational Technology, September 1997DMW, Institute of Educational Technology, September 1997
DMW UOC Open Seminar May 2013
Global Warming
DMW UOC Open Seminar May 2013
Next: ‘Yoked’ apps via BuddySpace
Student A
Student B(‘yoked’, butwithout full
screen sharingrequired!)
DMW UOC Open Seminar May 2013
Global Warming: Simlink Presentation
LISC: Aily Fowler
• Kent University ab-initio Spanish module
• Large student numbers
• Skills-based course
• Provision of sufficient formative assessment meant unmanageable marking loads
• Impossible to provide immediate feedback
• leading to fossilisation of errors
DMW UOC Open Seminar May 2013
The LISC solution: developed by Ali Fowler• A CALL system designed to enable students
to:• Independently
practise sentence translation
• Receive immediate (and robust) feedback on all errors
• Attend immediately to the feedback (before fossilisation can occur)
DMW UOC Open Seminar May 2013
How is the final mark arrived at in the LISC System?
• The two submissions are unequally weighted
• Best to give more weight to the first attempt
• since this ensures that students give careful consideration to the construction of their first answer
• but can improve their mark by refining the answer
• The marks ratio can vary (depending on assessment/feedback type)
• the more information given in the feedback, the lower the weight the second mark should carry
DMW UOC Open Seminar May 2013
Heuristics for the final mark
• If the ratio is skewed too far in favour of the first attempt…
• students are less inclined to try hard to correct non-perfect answers
• If the ratio is skewed too far in favour of the second attempt…
• students exhibit less care over the construction of their initial answer
DMW UOC Open Seminar May 2013
What about emotional support in the feedback?
• Difficult at times to receive written feedback
• Not just a cognitive response
• How can Bales help?
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
Coding into Categories
• Bales analysis
• Psychology 1950s
• Analyses talk
• Includes socio-emotive categories
• Flander’s (1970)categories inappropriate as also includes classroom control
DMW UOC Open Seminar May 2013
Bales Categories
• Four main groupings
• A. Positive reactions; agreeing and boosting the other person
• B. Directing/teaching
• C. Questions: requesting information, clarification etc
• D. Negative reactions: disagreement
DMW UOC Open Seminar May 2013
Coding the tutor comments
Categories Specific Examples
Positive Reactions
A1
A2
A3
1. Shows solidarity
2. Shows tension release
3. Shows agreement
Jokes, gives help, rewards others
Laughs, shows satisfaction
Understands, concurs, complies, passively accepts
Attempted Answers
B1
B2
B3
4. Gives suggestion
5. Gives opinion
6. Gives information
Directs, proposes, controls
Evaluates, analyses, expresses feelings or wishes
Orients, repeats, clarifies, confirms
Questions
C1
C2
C3
7. Asks for information
8. Asks for opinion
9. Asks for suggestion
Requests orientation, repetition, confirmation, clarification
Requests evaluation, analysis, expression of feeling or wishes
Requests directions, proposals
Negative Reactions
D1
D2
D3
10. Shows disagreement
11. Shows tension
12. Shows antagonism
Passively rejects, resorts to formality, withholds help
Asks for help, withdraws
Deflates others, defends or asserts self
Bales’ Interaction Process
DMW UOC Open Seminar May 2013
Identifying trends: H801
0 5 10 15 20 25
A Pass 1
A Pass 2
A Pass 3
A Pass 4
B Pass 1
B Pass 2
B Pass 3
B Pass 4
C Pass 1
C Pass 2
C Pass 3
C Pass 4
D Pass 1
D Pass 2
D Pass 3
D Pass 4
Ba
les'
In
tera
cti
on
al
Ca
teg
ori
es
at
ea
ch
Pa
ssL
leve
l
Number ofIincidences
Graph to show conflated Bale’s categories against mean number of incidences in H801 scripts
DMW UOC Open Seminar May 2013
Identifying trends: H801
5.96
17.13
5.73
1.61
A
B
C
D
Pie Chart to show the mean number of incidences per pass per conflated Bales' Interactional Category for all four levels of
pass in H801 scripts
Key:
A = Positive reactions
B = Responses
C = Questions
D = Negative reactions
DMW UOC Open Seminar May 2013
Identifying trends
A
B
C
D
A
B
C
D
A
B
C
D
Pie Charts to show the mean number of incidences per conflated Bales Interactional Category for ‘Pass 1’ and ‘Pass 4’ in the following courses:
Key:
A = Positive reactions C = Questions B = Responses D = Negative reactions
Pass 4
Pass 1
B820 S103
A
B
C
D
A
B
C
D
A
B
C
D
H801
DMW UOC Open Seminar May 2013
What is Open Mentor?
• “An open source mentoring tool for tutors”
• “Open source” = free and easy to use, and to embed in an institutions infrastructure and working practices
• “mentoring” = designed to help people learn how to give feedback effectively, through reflection and social networks
• “tutors” = primarily intended for teaching staff, but with clear applications for those involved in quality
DMW UOC Open Seminar May 2013
How Open Mentor handles comments
• “Good work”
• “Yes, well done”
• “Yes, but is this useful?”
• “Can you explain what you mean”
• “This does not follow”
• A = positive reactions
• A = positive reactions
• B = attempted answers, and not a positive reaction
• C = questions
• D = negative reactions
DMW UOC Open Seminar May 2013
Explaining OpenMentor’s Rules
• Four categories
• A – Positive Reactions
• B – Attempted Answers
• C – Questions
• D – Negative Reactions
DMW UOC Open Seminar May 2013
‘A’ - Positive Reactions
Category Examples of Rules Examples of commentsA - Positive Reactions1. Shows solidarity A1 ...excellent... Excellent Conclusions.
A1 ...(good|comprehensive)... Good, you are drawing on hard facts here.A1 ...nicely... Very nicely stated. Your analysis is thorough and your
conclusions consistent regarding the attractiveness of the budget airline sector. This is a good example of critical thinking.
A1 ...well presented... Very well presented diagram with interesting information.A1 ...effective use... Effective use of the case material here.A1 …well (structured|stated)… Report very well structured.A1 ...(well|clear)(ly)*
(structured|structure|summary| summarised|presented|presentation)...
The corporate vs. business unit strategy is well presented and nicely tied to strategies.
A1 ...reasonable.... A reasonable structure as listed in your table of contents.A1 ...useful point(s)... Generally useful points in this section.
2. Shows tension release A2 ... a helpful...
A2 …(thanks|thank you)…3. Shows agreement A3 ...yes... Yes, the intellectual reactions are both real.
A3 ...indeed... Indeed – if it has one basic strategy it is surely differentiation, though it still has to control costs.
4. Praise then direction A4 good...but... Good model, good quote, but be careful about what industry you analyse ??
DMW UOC Open Seminar May 2013
‘B’ Attempted Answers
Category Rules Examples of commentsB - Attempted Answers
B4 …perhaps… Perhaps even better here to explain the link in your mind between "analysis of stratgeies" and "strategic issues".
B4 …requires…B4 …take care… Take care with your STEP analysis not to make it too industry
focussed.B4 …useful to… Innovation is closely linked to structure and culture- it would be
useful to see some book 6-8 concepts here too.
B4 …you (might|ought)… You ought to have explicitly stated these.
B4 Don’t|never … Don’t introduce new frameworks just for the sake of it in the conclusion. The conclusion should be pulling together what went before.
B4 Please (see|refer to|look at)... Please make sure to read and understand the question correctly
5. Gives opinion B5 I (am|think)... I think I can see where you are going, though a numbered report format might have demonstrated the approach better
B5 This is.... this is an introduction rather than a “summary”B5 ...sounds...like... This sounds as if it could be very popular!!B5 ...not sure... I am not sure about the balance between the environmental
analysis and the review of the resources, capabilities (power, culture, structures and systems) as raised in the question.
B5 I (thought|agree|suggest)… I thought it was because they did not need any external input and saw a significant market sector they could address themselves.
B5 I (do|don’t) think… I don’t think this exercise has helped to develop your analysis. I also think that the development of the perspectives is superficial
6. Gives information B6 …(demonstrates|shows) this….B6 Also… Also, cross link to Leadership issues, Pettigrew on Strategic
Thinking tooB6 ...Q1... etc Q1 = 59/100
DMW UOC Open Seminar May 2013
Is the rule set generic?
Comments Classified from Test Data
0102030405060708090
100
B820 TestSet 1
B820 TestSet 2
B820 TestSet 3
A850 M878 S809
Test Data Set
% o
f co
mm
ents
cla
ssif
ied
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
OpenMentor Transfer: JISC funded
• JISC funded project
• Transfer OpenMentor technology to King’s and Southampton
• What changes are needed for cross institutional use?
• Identify strengths and limitations of OM for training tutors
DMW UOC Open Seminar May 2013
Transferring OM to other HEIs
DMW UOC Open Seminar May 2013
• Transferred to Southampton and Kings London
• Participating Tutors given face to face training
• King's College:
• 3 tutors.
• 25. learning experts at TEL forum gave feedback after a demonstration
• Southampton
• 3 tutors.
• Interviews and questionnaire
• Open University
• 3 distance education tutors
• Questionnaire and epistolary interviews
• 113 students in a Masters course focussing on Innovation in eLearning and 5 tutors.
Lessons learned after completion of first cycle of trials
• Open Mentor’s theoretical framework was robust enough to facilitate and encourage dialogue and reflective activities
• Tutors positive about the system’s functions to support provision of feedback
• Suggestions for change
• a module for user authentication and management
• the development of OM reports to help tutors to progress towards the ideal ‘state’ of feedback provided.
• used for training purposes as an academic development tool.
• Our contact details, blog and references -http://omtetra.ecs.soton.ac.uk/wordpress/
DMW UOC Open Seminar May 2013
What can we learn from modelling tutors marking to construct a formative
e-assessment tool?
• Open Comment project builds on the work of
OpenMentor
• Free text entry for History and Philosophy students
• Immediate feedback (in context) to students
• Influenced by ELIZA (Weizenbaum, 1963)
DMW UOC Open Seminar May 2013
Open Comment addresses the problem of free text entry
• Automated formative assessment tool
• Free text entry for students
• Automated feedback and guidance
• Open questions, divergent assessment
• No marks awarded
• For use by Arts Faculty
DMW UOC Open Seminar May 2013
Causal models of explanation
• First step:
• Identification of question types where students exhibit causal reasoning
• Looked for questions with:
• Salient variables
• Description of these variables
• Identification of trends
• Identification of relationship between the variables i.e. causality
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
Praise for effort and not just ability
• Praise for ability per se can hinder learning (Mueller & Dweck, 1998)
• Praise = being clever
• Negative feedback now without ability
• Disempowering and demoralising
DMW UOC Open Seminar May 2013
Mueller & Dweck (1998)
• Raven’s Matrices (IQ)
• First test pupils praise either for effort or ability
• Second test most difficult
• Third test medium difficulty. Score up 1 points for pupils praised for effort. Down 1 point ability
DMW UOC Open Seminar May 2013
Stages of analysis by computer of students’ free text entry for Open Comment: advice with respect to content (socio-emotional support stylised example)• STAGE 1a: DETECT ERRORS E.g. Incorrect dates,
facts. (Incorrect inferences and causality is dealt with below)
• Instead of concentrating on X, think about Y in order to answer this question Recognise effort (Dweck) and encourage to have another go
• You have done well to start answering this question but perhaps you misunderstood it. Instead of thinking about X which did not…….. Consider Y
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
Computer analysis continued
• STAGE 2a: REVEAL FIRST OMISSION
• Consider the role of Z in your answer Praise what is correct and point out what is missing Good but now consider the role X plays in your answer
• STAGE 2b: REVEAL SECOND OMISSION
• Consider the role of P in your answer Praise what is correct and point out what is missing Yes but also consider P. Would it have produced the same result if P is neglected?
DMW UOC Open Seminar May 2013
Final stages of analysis• STAGE 3:REQUEST
CLARIFICATION OF KEY POINT 1
• STAGE 4:REQUEST FURTHER ANALYSIS OF KEY POINT 1(Stages 3 and 4 repeated with all the key points)
• STAGE 5:REQUEST THE INFERENCE FROM THE ANALYSIS OF KEY POINT 1 IF IT IS MISSING
• STAGE 6:REQUEST THE INFERENCE FROM THE ANALYSIS OF KEY POINT 1 IF IT IS NOT COMPLETE
• STAGE 7:CHECK THE CAUSALITY
• STAGE 8:REQUEST ALL THE CAUSAL FACTORS ARE WEIGHTED
Where are we now?
• Opening up with Open Source
• Moving towards vision and not losing sight of it through tool adaptation
• More work to do for Arts
• Open Comment - pedagogical model open to test
• Feedback
• Changing pedagogy
• Another handle on misconceptions
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
Open Comment drivers for reflection
• Students are able to find facts similar to X
• Know how X might be disputed
• Are able to make predictions about X
• Know how to use X in an argument
• Know how far X can be pushed
• Supported with tools and strategies for effort
SAFeSEAProfessor Denise WhitelockProfessor John Richardson
Professor Stephen Pulman
An automated tool supporting
online writing and
assessmentof essays providing
accurate targeted feedback
SAFeSEA: Supportive Automated Feedback for Short Essay Answers
DMW UOC Open Seminar May 2013
About SAFeSEA
• Effect of summarisation
• What are the beneficial factors?
• Correlate measures of learner activity and essay improvement
• Effect of hints
• http://www.open.ac.uk/iet/main/research-scholarship/research-projects/supportive-automated-feedback-short-essay-answers
DMW UOC Open Seminar May 2013
OpenEssayist: SAFeSEA Web application for summarisation-based formative feedback
DMW UOC May 2013
localhost:8065
phaeros.open.ac.uk:80
openEssayist
PHP, Epiphany[Symfony2]
User
openEssayist RESTful API
PHP, Epiphany
User
User
pyEARESTful API
Python, Flask
localhost:8064
AfterTheDeadlineSpell/Grammar
checker
Java
User
localhost:9998
Apache TikaText Extractor
Java
Orchestrator
(Open)Learner Model
pyEssayAnalyser
Python, NLTK
localhost:8065
phaeros.open.ac.uk:80
openEssayist
PHP, Epiphany[Symfony2]
User
openEssayist RESTful API
PHP, Epiphany
User
User
pyEARESTful API
Python, Flask
localhost:8064
AfterTheDeadlineSpell/Grammar
checker
Java
User
localhost:9998
Apache TikaText Extractor
Java
Orchestrator
(Open)Learner Model
pyEssayAnalyser
Python, NLTK
Key words and phrases visualized in the essay context. Sentences in light-grey (green) background are key sentences as extracted by the EssayAnalyser (the number at the start of the sentence indicates its key-ness ranking);
bigrams are indicated in bold (red) and boxed.
DMW UOC May 2013
The structural elements of the essay can be used jointly with the key word extraction to highlight relevant information within specific parts of the essay, here the introduction (and the assignment question)
DMW UOC May 2013
Key words and phrases as separate lists
DMW UOC May 2013
Dispersion of key words across the essay http://www.open.ac.uk/iet/main/research-scholarship/research-projects/supportive-automated-feedback-short-essay-answers
DMW UOC May 2013
Can we find ways of using graph visualization techniques on the key words and key sentences, to make them helpful and meaningful to students?
DMW UOC May 2013
SAFeSEA
• Support for essay writing
• Shape landscape of eLearning and Learning Analytics
• Improves the student experience
• Support advances in NLP
DMW UOC Open Seminar May 2013
Feedback
• Students must decode feedback and then act on it Boud (2000)
• Students must have the opportunity to act on feedback Sadler (1989)
• Gauging efficacy through action
DMW UOC Open Seminar May 2013
Badge System: Mozilla
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
Characteristics Descriptor
Authentic Involving real-world knowledge and skills
Personalised Tailored to the knowledge, skills and interests of each student
Negotiated Agreed between the learner and the teacher
Engaging Involving the personal interests of the students
Recognise existing skills Willing to accredit the student’s existing work
Deep Assessing deep knowledge – not memorization
Problem oriented Original tasks requiring genuine problem solving skills
Collaboratively produced Produced in partnership with fellow students
Peer and self assessed Involving self reflection and peer review
Tool supported Encouraging the use of ICT
Elliott’s characteristics of Assessment 2.0 activities
A d v i c e f o r A c t i o n
The 4Ts Pyramid
DMW UOC Open Seminar May 2013
Tool Development
Training of staff
TransformationTasks
TransferLearning
National Union of Students’ Principles of Effective Assessment Times Higher Education, 29th January 2009
• Should be for learning, not simply of learning
• Should be reliable, valid, fair and consistent
• Should consist of effective and constructive feedback
• Should be innovative and have the capacity to inspire and motivate.
• Should be conducted throughout the course, rather than being positioned as a final event
• Should develop key skills such as peer and reflective assessment
DMW UOC Open Seminar May 2013
Final thoughts
• There is a growing consensus in the field of assessment that times are changing and that assessment needs to become more embedded/central in the teaching learning cycle (Hatzipanagos & Rochon 2011).
• Our project provides another phase in this type of research where the balance of socio emotive content contained in feedback cannot be ignored (Draper, 2009).
• Feedback that encourages the student to actively change their ideas and ways of organising their answers and discourse within a given subject domain is what is required and advocated by Whitelock (2011) as “advice for action”.
DMW UOC Open Seminar May 2013
“Advice for Action”, Whitelock (2011)
• Helping students find out what they do not know and how to remedy the situation can avoid the trauma of assessment
• Are we on the way with new e-tools?
DMW UOC Open Seminar May 2013
DMW UOC Open Seminar May 2013
References• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J.
(2013) ‘OpenEssayist: Extractive Summarisation & Formative Assessment of Free-Text Essays’. Workshop on Discourse-Centric Learning Analytics, 3rd Conference on Learning Analytics and Knowledge (LAK 2013), Leuven, Belgium
• Whitelock, D., Gilbert, L., Hatzipanagos, S., Watt, S., Zhang, P., Gillary, P. & Saucedo, A. (2012) Supporting tutors with their feedback using OpenMentor in three different UK Universities. 10th International Conference on Computer Based Learning in Science, CBLIS 2012, Barcelona, Spain. 26-29 June 2012.
• Whitelock, D., Gilbert, L. & Gale, V. (2011) ‘Technology-Enhanced Assessment and Feedback: How is evidence-based literature informing practice?’ International Computer Assisted Assessment Conference, DeVere Grand Harbour Hotel, Southampton, 5/6 July 2011. http://caaconference.co.uk/wp-content/uploads/WhitelockB-CAA2011.pdf
• Whitelock, D. (2010) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global. pp. 319–342.
DMW UOC Open Seminar May 2013
References (2)
• Whitelock, D. & Watt, S. (2008) ‘Putting Pedagogy in the driving seat with Open Comment: an open source formative assessment feedback and guidance tool for History Students.’ CAA Conference 2008, Loughborough University, 8/9 July 2008, edited by Farzana Khandia pp. 347-356 ISBN 0-9539572-7-6 http://kn.open.ac.uk/public/document.cfm?docid=11638
• Whitelock, D. & Watt, S. (2007) e-Assessment: How an we support tutors with their marking of electronically submitted assignments? Ad-Lib Journal for Continuing Liberal Adult Education, Issue 32, March 2007 pp 7-9, ISSN 1361-6323.
DMW UOC Open Seminar May 2013
References (3)
• Whitelock, D. (2006) Electronic Assessment: Marking, Monitoring and Mediating Learning. In McAndrew, P. and Jones, A. (eds) Interactions, Objects and Outcomes in learning. Special Issue of International Journal of Learning Technology. Vol. 2, Nos 2/3 pp 264-276.
• Whitelock, D. & Watt, S. (2006) OpenMentor: opening tutors eyes to the written support given to students in their assignments. JISC Conference 2006, Information & Communication Technology in Education and Research. International Conference Centre, Birmingham, 14 March 2006.
DMW UOC Open Seminar May 2013
Top Related