Learning, assessment and technology: in that order Keynote address to AMEE conference September...
-
Upload
lane-haddaway -
Category
Documents
-
view
214 -
download
0
Transcript of Learning, assessment and technology: in that order Keynote address to AMEE conference September...
Learning, assessment and technology: in that order
Keynote address to AMEE conferenceSeptember 2009; Malaga, Spain
Dylan Wiliam
Institute of Education, University of London
www.dylanwiliam.net
Overview of presentationSome theoretical precepts About learning About teaching
• Pedagogies of engagement• Pedagogies of contingency• Pedagogies of formation
The role of technology Supporting, rather than replacing, teachers Classroom aggregation technologies
Some thoughts about supporting teachers in changing practice
How do we improve learning?
What gets learnt?
(Denvir & Brown, 1986)
Key insights from C20th psychologyWhat gets learned as a result of a particular sequence of
instructional activities is impossible to predict, but
Student errors are not random
Conclusions Assessment is the bridge between teaching and learning Teaching is interesting because learners are so different, but only
possible because they are so similar
Learning power environmentsKey concept: Teachers do not create learning Learners create learning
Teaching as the engineering of learning environments
Key features: Create student engagement (pedagogies of engagement) Well-regulated (pedagogies of contingency) Develop habits of mind (pedagogies of formation)
Why pedagogies of engagement?Intelligence is partly inherited So what?
Intelligence is partly environmental Environment creates intelligence Intelligence creates environment
Learning environments High cognitive demand Inclusive Obligatory
264 low and high ability grade 6 students in 12 classes in 4 schools; analysis of 132 students at top and bottom of each class
Same teaching, same aims, same teachers, same class work Three kinds of feedback: scores, comments, scores+comments
[Butler(1988) Br. J. Educ. Psychol., 58 1-14]
Engagement and feedback
Achievement Attitude
Scores no gain High scorers: positive
Low scorers: negative
Comments 30% gain High scorers : positive
Low scorers : positive
[Butler(1988) Br. J. Educ. Psychol., 58 1-14]
Responses
What do you think happened for the students given both scores and comments?
A. Gain: 30%; Attitude: all positiveB. Gain: 30%; Attitude: high scorers positive, low scorers negativeC. Gain: 0%; Attitude: all positiveD. Gain: 0%; Attitude: high scorers positive, low scorers negativeE. Something else
Achievement Attitude
Scores no gain High scorers : positive
Low scorers: negative
Comments 30% gain High scorers : positive
Low scorers : positive
Kinds of feedback (Nyquist, 2003)Weaker feedback only Knowledge of results (KoR)
Feedback only KoR + clear goals or knowledge of correct results (KCR)
Weak formative assessment KCR+ explanation (KCR+e)
Moderate formative assessment (KCR+e) + specific actions for gap reduction
Strong formative assessment (KCR+e) + activity
Effect of formative assessment (HE)
N Effect size
Weaker feedback only 31 0.14
Feedback only 48 0.36
Weaker formative assessment 49 0.26
Moderate formative assessment 41 0.39
Strong formative assessment 16 0.56
(Nyquist, 2003; revised values)
Effects of feedbackKluger & DeNisi (1996) review of 3000 research reports Excluding poorly designed studies left 131 reports, 607 effect
sizes, involving 12652 individuals
On average, feedback increases achievement, but Effect sizes highly variable 38% (50 out of 131) of effect sizes were negative
Engagement in learningAttribution (Dweck, 2000) Personalization (internal vs. external) Permanence (stable vs. unstable) Good learners attribute failure and success to internal, unstable
causes. (It’s down to you, and you can do something about it.)
Views of ‘ability’ Fixed (IQ) Incremental (untapped potential) Essential that teachers inculcate in students a view that ‘ability’ is
incremental rather than fixed (by working, you’re getting smarter).
Dual-pathway theory (Boekaerts, 2006)Long-term learning goals are translated into short-term
learning intentions
Dynamic comparisons of task and situational demands with personal resources
Resulting activation of energy along one of two pathways: Well-being Growth
Motivation: cause or effect?
competence
challenge
Flow
apathyboredom
relaxation
arousalanxiety
worry control
high
low
low high
(Csikszentmihalyi, 1990)
Why pedagogies of contingency?
Fuchs & Fuchs (1986)
Natriello (1987)
Crooks (1988)
Banger-Drowns, et al. (1991)
Kluger & DeNisi (1996)
Black & Wiliam (1998)
Nyquist (2003)
Dempster (1991, 1992)
Elshout-Mohr (1994)
Brookhart (2004)
Allal & Lopez (2005)
Köller (2005)
Brookhart (2007)
Wiliam (2007)
Hattie & Timperley (2007)
Shute (2008)
Hinge-point questionAn experimental study of problem-based learning in undergraduate medical education reports that a result was significant (p<0.05). This means that:A. The experimental group out-performed the control group by at least
5%
B. There is only a 5% chance that the experimental group did not out-perform the control group
C. There is a 5% chance that there is no difference between the experimental group and the control group
D. There is only a 5% chance that the observed result would have happened if the experimental and control groups had the same achievement
Other supports for contingencyAll-student response systems ABCD cards “Exit-pass” questions
Exit-pass questionSummarize the key principles of the following schools of psychology on the appropriate coloured card Associationism (blue) Information processing (orange) Constructivism (red) Situated approaches (green)
Pedagogies of formationInstilling disciplinary habits of mind History Philosophy Statistics
Instilling critical perspectives Values
Three generations of pedagogyFirst generation Traditional pedagogy Negligible contingency
Second generation All student response systems Contingency dependent entirely on teacher skill
Third generation Automated aggregation technologies Contingency supported by technology
Evidence-centred designQuality in assessment is essentially a matter of validity Validity is a property of inferences, not of instruments Assessments should be designed “backwards” from the intended
inferences
Four-process architecture for assessment design Task selection Task presentation Evidence identification Evidence accumulation
Almond, Steinberg and Mislevy (2002)
Task selection/Task presentation
Hinge-point questionWhich of the following is the most important difference between the theories of Piaget and Vygotsky?
A. Piaget places greater importance on the role of conservation in cognitive development
B. Vygotsky places greater importance on the role of cultural artifacts in cognitive development.
C. Vygotsky did not believe in distinct stages of cognitive development.
D. Piaget was a social constructivist while Vygotsky placed greater emphasis on cultural-historical activity theory
Evidence identification
Evidence identificationSingle student response systems
All-student response systems Flash-cards/dry erase boards Classroom ‘clickers’ Traditional keyboards (wired/wireless) Anoto pens
Anoto pen Wireless pen
Special coated paper
Pen ‘knows where it is’
Palm with wireless keyboard
Text-based input
Limited task-presentation capability
Portable
Classroom ‘clickers’ (and their progeny)
Discourse®
www.ets.org/discourse
Evidence identificationAutomated essay scoring (e-rater)
Paraphrase analysers (c-rater)
Graphical and equation analysers (m-rater)
Evidence identification technologies
unstructuredstructured
evidence structure
teacher-mediated
automatedclickers
aggr
egat
ion
ABCDcards
dry-eraseboards
c-rater
Discourse®
latent semantic analysis
m-rater
e-rater
Evidence accumulation
Evidence accumulationUnidimensional student models Useful for summative purposes Almost useless for formative purposesMultidimensional student modelsEvidence-centred design Bayesian inference networks
• Proficiency model• Task model• Evidence model• Student model
Four-process architecture for ECD
Mislevy, Almond and Lukas (2003)
Task selection
Task presentation
Evidence accumulation
Evidence identification
Evidence utilizationWhole-class
Sub-groups Homogenous Heterogenous
Individualization
37
Actually, the technology is the easy partWhat’s hard is changing practice
Telling teachers what to do doesn’t work
Context always intrudes…
38
The ancient yogis used logs of wood, stones, and ropes to help them practise asanas effectively. Extending this principle, Yogacharya Iyengar invented props which allow asanas to be held easily, and for a longer duration without strain.
Yogacharya Iyengar in setubandha sarvangasana
This version of the posture requires considerable strength in the neck, shoulders, and back requiring years of practice to achieve it. It should not be attempted without supervision.
39
A model for teacher learningContent, then process
Content (what we want teachers to change) Evidence Ideas (strategies and techniques)Process (how to go about change) Choice Flexibility Small steps Accountability Support
SummaryLearning has to be done by the learner, not for the learner
Teaching as engineering effective learning environments
Features of effective learning environments Pedagogies of engagement Pedagogies of contingency
• All-student response systems• Classroom aggregation technologies
Pedagogies of formation
Learning milieu focused on growth, rather than well-being
Teachers supported to improve practice continually