Understanding Assessments: What they Mean and What they Do
Transcript of Understanding Assessments: What they Mean and What they Do
![Page 1: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/1.jpg)
DylanWiliam(@dylanwiliam)
UnderstandingAssessments:WhattheyMeanandWhattheyDo
www.dylanwiliamcenter.com www.dylanwiliam.net
![Page 2: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/2.jpg)
Ini$alassump$ons
• Anyassessmentsystemshouldbedesignedtoassesstheschool’scurriculumratherthanhavingtodesignthecurriculumtofittheschool’sassessmentsystem.
• Sinceeachschool’scurriculumshouldbedesignedtomeetlocalneeds,therecannotbeaone-size-fits-allassessmentsystem—eachschool’sassessmentsystemwillbedifferent.
• Thereare,however,anumberofprinciplesthatshouldgovernthedesignofassessmentsystems,and
• Thereissomesciencehere—knowledgethatpeopleneedinordertoavoiddoingthingsthatarejustwrong.
2
![Page 3: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/3.jpg)
Assessment:Acau$onarytale
A B C D E F G H Total
Adams 100 30 47 72 40 75 30 47 441
Brown 90 38 43 60 20 65 48 70 434
Collins 61 36 40 45 41 55 62 80 420
Dorkin 63 32 51 90 30 70 47 35 418
Evans 56 55 41 82 45 40 49 41 409
Fuller 80 45 49 64 65 45 38 20 406
Grant 23 47 45 55 60 80 32 60 402
Howell 40 35 52 70 56 20 60 65 398
Iman 85 40 60 40 28 51 55 30 389
Jones 72 54 50 10 25 35 66 75 387
Keller 48 57 55 34 70 60 36 10 370
Lant 10 60 59 20 35 30 70 58 342Mean 61 44 49 54 43 52 49 49
![Page 4: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/4.jpg)
Equalizingtherangeforeachsubject
A B C D E F G H Total
Adams 100 0 35 77 40 92 0 53 397
Brown 89 27 15 63 0 75 45 86 400
Collins 57 20 0 44 42 58 80 100 401
Dorkin 59 7 55 100 20 83 43 36 403
Evans 51 83 5 90 50 33 48 44 404
Fuller 78 50 45 68 90 42 20 14 407
Grant 14 57 25 56 80 100 5 71 408
Howell 33 17 60 75 72 0 75 79 411
Iman 83 34 100 38 16 52 62 29 414
Jones 69 80 50 0 10 25 90 93 417
Keller 42 90 75 30 100 67 15 0 419
Lant 0 100 95 12 30 17 100 69 423Mean 56 47 47 54 46 54 49 56
![Page 5: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/5.jpg)
Andusingclassranksineachsubject…
A B C D E F G H Total
Adams 1 12 8 3 7 2 12 7 52
Brown 2 8 10 6 12 4 7 3 52
Collins 7 9 12 8 6 6 3 1 52
Dorkin 6 11 5 1 9 3 8 9 52
Evans 8 3 11 2 5 9 6 8 52
Fuller 4 6 7 5 2 8 9 11 52
Grant 11 5 9 7 3 1 11 5 52
Howell 10 10 4 4 4 12 4 4 52
Iman 3 7 1 9 10 7 5 10 52
Jones 5 4 6 12 11 10 2 2 52
Keller 9 2 3 10 1 5 10 12 52
Lant 12 1 2 11 8 11 1 6 52
![Page 6: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/6.jpg)
Beforewecanassess…
• The‘backwarddesign’ofaneducaaonsystem– Wheredowewantourstudentstogetto?
• ‘Bigideas’– Whatarethewaystheycangetthere?
• Learningprogressions– Whenshouldwecheckon/reportprogress?
• Inherentandusefulcheckpoints
6
![Page 7: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/7.jpg)
Bigideas
7
![Page 8: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/8.jpg)
Bigideas8
• A“bigidea”– helpsmakesenseofapparentlyunrelatedphenomena– isgenera)veinthatiscanbeappliedinnewareas
![Page 9: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/9.jpg)
Bigideasinreading
• Wriangisanafempttocommunicatemeaning• Makingsenseoftextohenrequiresmakingconnecaonsbetweensentences
• Writersohenchoosewordsfortheeffecttheyhaveonthelistener/reader
• Thehero’sjourney(Campbell,1949)• …
9
![Page 10: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/10.jpg)
Learningprogressions
Whatisitthatgetsbeferwhensomeonegetsbeferatreading?
10
![Page 11: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/11.jpg)
The“seduc$veallure”ofneuroscience
11
![Page 12: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/12.jpg)
Cor$callanguagelocaliza$on12
• 117individuals(aged4to80)undergoingfrontalorfrontotemporoparietalcraniotomiesasatreatmentforepilepsy
• Subjectswereshownlinedrawingsoffamiliarobjectsandaskedtonamewhattheyhadseenwhileexposedregionsofthecerebralcortexweresamulatedwithelectriccurrent
• Namingerrorsweretakenasindicaangthattheregioninquesaonwasessenaaltolanguage
Ojemann,Ojemann,Leatch,andBerger(1989)
![Page 13: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/13.jpg)
58
6
1417
14
3334
11
1264
8211 45
68 58 51
667261625876
6046 60
5861 44
8
6 10 23 3
Ojemann,Ojemann,Leatch,andBerger(1989)
Numberofpaaentswithasiteineachzone(outof117)
![Page 14: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/14.jpg)
2037
50
5029
43
4523
18
4236
799 27 19
14 26
293629261914
52 7
1921 32
8
0 0 0 0 0
Ojemann,Ojemann,Leatch,andBerger(1989)
Percentageofpaaentswithasiteineachzonewithsignificantnamingerrorsinthatzone
![Page 15: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/15.jpg)
“Allmodelsarewrong;someareuseful”
“Sinceallmodelsarewrongthescienastcannotobtaina‘correct’onebyexcessiveelaboraaon.OnthecontraryfollowingWilliamofOccamheshouldseekaneconomicaldescripaonofnaturalphenomena.Justastheabilitytodevisesimplebutevocaavemodelsisthesignatureofthegreatscienastsooverelaboraaonandoverparameterizaaonisohenthemarkofmediocrity.”(Box,1976p.792)
![Page 16: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/16.jpg)
Learningprogressions
• Whatgetsbeferwhenstudentsgetbeferatreading?– Phonemicawareness– Phonics– Fluency– Vocabulary– Textcomprehension
16
NaaonalReadingPanel(2001)
![Page 17: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/17.jpg)
The“simple”viewofreading17
Scarborough(2001)
Backgroundknowledge
Vocabulary
Languagestructures
Verbalreasoning
Literacyknowledge
Sightrecogniaon
DecodingPhonologicalawareness
![Page 18: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/18.jpg)
Lefers
Translaaonrules
Wordsounds
Syntacacrules
Ideaweb
Spellings
Situaaonmodel
Wordmeanings
Sentencerepresentaaon
Expandedmodelofreading(Willingham,2017)
![Page 19: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/19.jpg)
Copythis19
ЖӘШІК
![Page 20: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/20.jpg)
Readingskills:whataretheyreally?
“Amanifold,containedinanintuiaonwhichIcallmine,isrepresented,bymeansofthesynthesisoftheunderstanding,asbelongingtothenecessaryunityofself-consciousness;andthisiseffectedbymeansofthecategory.”Whatisthemainideaofthispassage?
A. Withoutamanifold,onecannotcallanintuiaon‘mine.’B. Intuiaonmustprecedeunderstanding.C. Intuiaonmustoccurthroughacategory.D. Self-consciousnessisnecessarytounderstanding
Hirsch(2006)
![Page 21: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/21.jpg)
Lostintransla$on?
• “Comprehensiondependsonconstrucangamentalmodelthatmakestheelementsfallintoplaceand,equallyimportant,enablesthelistenerorreadertosupplyessenaalinformaaonthatisnotexplicitlystated.Inlanguageuse,thereisalwaysagreatdealthatislehunsaidandmustbeinferred.Thismeansthatcommunicaaondependsonbothsides,writerandreader,sharingabasisofunspokenknowledge.Thislargedimensionoftacitknowledgeispreciselywhatisnotbeingtaughtadequatelyinourschools.”
Hirsch(2009loc.176)
![Page 22: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/22.jpg)
Domainknowledgeandmemory
• 3rd(N=64),5th(N=67)and7th(N=54)gradestudentsfromHeidelberg,Germany,testedonreadingexperaseandsoccerknowledge– 13-itemquesaonnaireonsoccerknowledge– standardizedreadingcomprehensiontest
• Studentsheard(twice)andreadawell-structuredreadablestoryonayoungplayer’sexperiencesinasoccergame
• Tested15minuteslaterwithaclozeversionofthetestwith20blanks
![Page 23: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/23.jpg)
Schneider,Körkel,andWiener(1989)
High
Low
HighLow
Knowledge of socce
rReading ability
16.417.0
11.111.0
![Page 24: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/24.jpg)
Assessment
24
![Page 25: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/25.jpg)
WriOenexamina$ons
“Theyhavepervertedthebesteffortsofteachers,andnarrowedandgroovedtheirinstrucaon;theyhaveoccasionedandmadewellnighimperaavetheuseofmechanicalandrote
methodsofteaching;theyhaveoccasionedcrammingandthemostvicioushabitsofstudy;theyhavecausedmuchofthe
overpressurechargeduponschools,someofwhichisreal;theyhavetemptedbothteachersandpupilstodishonesty;
andlastbutnotleast,theyhavepermifedamechanicalmethodofschoolsupervision.”
25
(White,1888pp.517-518)
![Page 26: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/26.jpg)
Campbell’slaw
“Themoreanyquanataavesocialindicatorisusedforsocialdecision-making,themoresubjectitwillbetocorrupaonpressuresandthemoreaptitwillbetodistortandcorruptthesocialprocessesitisintendedtomonitor.”(Campbell,1976p.49)– Allperformanceindicatorslosetheirmeaningwhenadoptedaspolicytargets
– Thecleareryouareaboutwhatyouwant,themorelikelyyouaretogetit,butthelesslikelyitistomeananything
26
![Page 27: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/27.jpg)
The“LakeWobegon”effect
3.4
3.5
3.6
3.7
3.8
3.9
4.0
4.1
4.2
4.3
4.4
1986 1987 1988 1989 1990
Grade
equ
ivalen
ts
TestC TestB TestC
Koretz,Linn,DunbarandShepard(1991)
![Page 28: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/28.jpg)
Effectsofnarrowassessment
• Incenavestoteachtothetest– Focusonsomesubjectsattheexpenseofothers– Focusonsomeaspectsofasubjectattheexpenseofothers
– Focusonsomestudentsattheexpenseofothers(“bubble”students)
• Consequences– Learningthatis
• Narrow• Shallow• Transient
28
![Page 29: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/29.jpg)
GeSngassessmentright
29
![Page 30: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/30.jpg)
Whatisanassessment?
• Anassessmentisaprocedureformakinginferences– Wegivestudentsthingstodo– Wecollecttheevidence– Wedrawconclusions
• Keyquesaon:“Onceyouknowtheassessmentoutcome,whatdoyouknow?”
• Foranytest:– someinferencesarewarranted(valid)– somearenot
30
![Page 31: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/31.jpg)
Validity
• Evoluaonoftheideaofvalidity– Apropertyofatest– Apropertyofstudents’scoresonatest– Apropertyofinferencesdrawnonthebasisoftestresults
• “Onevalidatesnotatestbutaninterpretaaonofdataarisingfromaspecifiedprocedure”(Cronbach,1971)
• Consequences– Nosuchthingasavalid(orindeedinvalid)assessment– Nosuchthingasabiasedassessment– Formaaveandsummaavearedescripaonsofinferences
31
![Page 32: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/32.jpg)
Meaningsandconsequencesofassessment
• Evidenaalbasis– Whatdoestheassessmentresultmean?
• Consequenaalbasis– Whatdoestheassessmentresultdo?
• Assessmentliteracy(Saggins,1991)– Doyouknowwhatthisassessmentresultmeans?– Doesithaveualityforitsintendeduse?– Whatmessagedoesthisassessmentsendtostudents(andotherstakeholders)abouttheachievementoutcomeswevalue?
– Whatislikelytobetheeffectofthisassessmentonstudents?
![Page 33: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/33.jpg)
Validityrevisited
“Validityisanintegraaveevaluaavejudgmentofthedegreetowhichempiricalevidenceandtheoreacalraaonalessupporttheadequacyandappropriatenessofinferencesandacaonsbasedontestscoresorothermodesofassessment.”(Messick,1989p.13)• Socialconsequences:– “Rightconcern,wrongconcept”(Popham,1997)
![Page 34: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/34.jpg)
Qualityinassessment
• Threatstovalidity– Construct-irrelevantvariance
• Systemaac:goodperformanceontheassessmentrequiresabiliaesnotrelatedtotheconstructofinterest
• Random:goodperformanceisrelatedtochancefactors,suchasluck(effecavelypoorreliability)
– Constructunder-representaaon• Goodperformanceontheassessmentcanbeachievedwithoutdemonstraangallaspectsoftheconstructofinterest
![Page 35: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/35.jpg)
35
Discussio
n • Workingasagroup,trytoframeonevalidityissueasanissueofconstruct-irrelevantvarianceorofconstructunder-representaaon.
![Page 36: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/36.jpg)
Understandingreliability
36
![Page 37: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/37.jpg)
Understandingtestscores
• Consideratestofstudents’abilitytospellwordsdrawnfromabankof1000words.
• Whatwecanconcludedependson:– Thesizeofthesample– Thewaythesamplewasdrawn– Students’knowledgeofthesample– Theamountofnoacegiven
![Page 38: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/38.jpg)
Samplesandreliability
• Supposeweaskastudenttospell20ofthewordsdrawnatrandom,atfivedifferentamesoftheday,withthefollowingresults– 15 17 14 15 14– Onaverage,thestudentscores15outof20– Ourbestguessisthestudentcanspell750ofthe1000words
• Iftheresultswere:– 20 12 17 10 16– Ourbestguessissallthatthestudentknows750ofthe1000spellings
– Butnowwearemuchlesscertainaboutthis
![Page 39: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/39.jpg)
Someexamples
Example1Actualscore 15 17 14 15 14Differencefromaverage 0 +2 -1 0 -1Averageerror 0(bydefiniaon!)Standarddeviaaonoferrors 1.2
Example2Actualscore 20 12 17 10 16Differencefromaverage 5 -3 +2 -5 +1Averageerror 0(bydefiniaon!)Standarddeviaaonoferrors 4.0
![Page 40: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/40.jpg)
Quan$fyingreliability
• The“standarderrorofmeasurement”or“SEM”isjustthestandarddeviaaonoftheerrorsaveragedoveralltesttakers
• Thereliabilityofthetestis:
![Page 41: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/41.jpg)
Rela$onshipofreliabilityanderror
• Foratestwithanaveragescoreof50,andastandarddeviaaonof15(sothatmostscoresrangefrom20to80),errorsofmeasurementareasfollows:
Reliability Standarderrorofmeasurement
0.70 8.20.75 7.50.80 6.70.85 5.80.90 4.70.95 3.4
![Page 42: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/42.jpg)
Whatdoesthismean?
• Consideraclassof25studentstakingareadingtest– withareliabilityof0.85– anaveragescoreof50– astandarddeviaaonof15(mostscoresrangefrom20to80)
• Then– 17studentsgetascorewithin6pointsoftheirtruescore– 7studentsgetascorethatismorethan6points,butlessthan12pointsfromtheirtruescore
– andonestudentgetsascorethatdiffersfromtheirtruescorebymorethan12points
• Unfortunately…– youwon’tknowwhichstudent– andyouwon’tknowiftheirscorewashigherorlowerthanitshouldhavebeen
![Page 43: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/43.jpg)
Reliability:0.75
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Observedscore
Truescore
![Page 44: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/44.jpg)
Reliability:0.80
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Observedscore
Truescore
![Page 45: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/45.jpg)
Reliability:0.85
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Observedscore
Truescore
![Page 46: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/46.jpg)
Reliability:0.90
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Observedscore
Truescore
![Page 47: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/47.jpg)
Reliability:0.95
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Observedscore
Truescore
![Page 48: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/48.jpg)
Understandingwhatthismeansinprac$ce
48
![Page 49: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/49.jpg)
Groupingstudentsbyability
49
![Page 50: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/50.jpg)
Usingtestsforgroupingstudentsbyability
shouldbein
group1 group2 group3 group4
studentsplacedin
group1 23 9 3
group2 9 12 6 3
group3 3 6 7 4
group4 3 4 8
Usingatestwithareliabilityof0.9,andwithapredicavevalidityof0.7,togroup100studentsintofourabilitygroups:
Only50%ofthestudentsareinthe“right”group
![Page 51: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/51.jpg)
Diagnos$ctes$ng
51
![Page 52: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/52.jpg)
Thelimitsofdiagnos$ctes$ng
• 120-itemmulaplechoicetestforteacherlicensure– Fourmajorsubjectareas
• languagearts/reading• mathemaacs• socialstudies• science
– 30itemspersubjectarea– Sub-scorereliabiliaesrangefrom0.71to0.83
![Page 53: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/53.jpg)
Howreliableare10-itemsubtestscores?
• Itemsforeachsubjectarearankedinorderofdifficulty(i.e.,1to30)
• Threeparallel10-itemformscreatedineachsubjectarea:– FormA:items1,4,7,…28– FormB:items2,5,8,…29– FormC:items3,6,9,…30
• Sub-scorereliabiliaesintherange0.40to0.60• OnformA,271examineesscored7inmathemaacsand3inscience
![Page 54: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/54.jpg)
Scoresof271studentsonformB
Sciencesubscore1 2 3 4 5 6 7 8 9 10
Mathsubscore
1 0 0 0 1 1 1 0 0 0 0
2 0 0 0 1 3 1 2 0 0 0
3 1 0 0 1 2 4 3 1 1 1
4 0 0 2 7 7 6 4 0 1 0
5 0 1 1 1 10 14 8 5 1 1
6 2 0 1 5 10 11 15 8 1 1
7 0 1 4 4 4 11 10 7 4 0
8 0 1 1 5 12 13 7 5 4 0
9 0 0 1 1 6 3 7 4 3 0
10 0 0 0 1 1 2 1 1 0 0
Sinharay,GautamandHalberman(2010)
110outof271(41%)examineesgotabeferformBscoreinsciencethanmathemaacs
![Page 55: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/55.jpg)
Whatdoesthismean?
• Astudentscoring7onmathemaacsand3onsciencewouldprobablywanttoimprovethelafer
• But110ofthe271examineesgotabeferscoreinsciencethanmathemaacsonFormB
• CorrelaaonofsciencesubscoresonFormsAandBis0.48
• CorrelaaonofsciencesubscoreonFormAwithtotalscoreonFormBis0.63
• Inotherwords,thetotalscoreonthetotaltestisabeferguidetothescoreonasub-testthananotherscoreonthesamesub-test
![Page 56: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/56.jpg)
Measuringprogress
56
![Page 57: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/57.jpg)
Reliability,standarderrors,andprogress
Grade Reliability SEMasapercentageofannualprogress
1 0.89 26%2 0.85 56%3 0.82 76%4 0.83 39%5 0.83 55%6 0.89 46%
Average 0.85 49%Inotherwords,thestandarderrorofmeasurementofthisreadingtestisequaltosixmonths’progressbyatypicalstudent
![Page 58: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/58.jpg)
Inotherwords…
• Inaclassof25students,iftheyhaveallmadeexactlytheexpectedprogress,andtheyaretestedwithatypicalreadingtesteverysixmonths:– Fourwillappeartohavemadenoprogressorgonebackwards
– Fourwillappeartohavemadeatleasttwiceasmuchprogressasexpected
– Andagain,youwon’tknowwhichstudentsarewhich…
![Page 59: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/59.jpg)
Trueandobservedgrowthscores
Pre-testaverage: 50Post-testaverage:60Pre-testSD: 15ChangeSD: 2Testreliability: 0.85Progressreliability:0.04
Trueprogress
Observedprogress
![Page 60: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/60.jpg)
Fortunately…
• Whileprogressmeasuresforindividualsareratherunreliable,progressmeasuresforgroupsaremuchmorereliable.
• Asrulesofthumb:– Forindividualstudents,progressmeasuresaremeaningfulonlyiftheprogressismorethantwicethestandarderrorofmeasurementofthetestbeingusedtomeasureprogress
– Foraclassof25students,progressmeasuresaremeaningfuliftheprogressismorethanhalfthestandarderrorofmeasurementofthetestbeingusedtomeasureprogress
![Page 61: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/61.jpg)
Curriculum-basedmeasurement
![Page 62: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/62.jpg)
Curriculum-basedmeasurement62
• Usedforavarietyofpurposesincludingscreening,benchmarking,andprogressmonitoring
• Avoidstheproblemsofmeasuringchangescoresbecauseitfocusesonmulapleassessmentsofstatus.
• Dependsonaclearviewofwhatwillbelearnedbytheendoftheinstrucaonalsequence.
• However,itisnotapanacea
![Page 63: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/63.jpg)
ReviewofstudiesofCBM-R63
“Suchstudiessuggestthatevenunderthebestcondiaons(i.e.,high-qualityprobesetsandaghtlycontrolledcondiaons),(a)aminimumof5or6weeksofdatawithmulapledatapointscollectedperweekareneededtoinformrouaneinstrucaonaldecisionsand(b)aminimumof12weeksofdatawithmulapledatapointscollectedperweekareneededtomakespecialeducaaoneligibilitydecisions”(p.12)
Ardoin,Christ,Morena,Cormier,andKlingbeil(2013)
![Page 64: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/64.jpg)
64
“…atthispoint,therearenostudiestosuggestthatanindividualstudent'sprogresscanbeaccuratelydeterminedusingCBM-Rprogressmonitoringdata”(p.14)“Furthermore,trainersandpublishersofCBM-RmaterialsshouldneithersuggesttoschoolteachersandothereducatorsthatCBM-RprogressmonitoringdatacanbeusedasaprimaryoutcomemeasuretoevaluateindividualstudentgrowthovershortperiodsofamenortrainthemincurrentCBM-Rdecisionrules.”(pp.14-15)
Ardoin,Christ,Morena,Cormier,andKlingbeil(2013)
![Page 65: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/65.jpg)
Discussionquesaon65
Discussio
n • Fromwhatyouhaveheardsofar,whatarethekeychallengesregardingthedesignofreadingassessmentforyourschool/district?
![Page 66: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/66.jpg)
Evidence-centereddesign
66
![Page 67: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/67.jpg)
Evidence-centereddesign
• Conceptualassessmentframework– Studentmodel:whatareweassessing?
• “Degreeofdifficulty”model• “Marksforstyle”model• “Support”model
– Evidencemodel:whatevidencedowewant?– Taskmodel:wherewilltheevidencecomefrom?– Four-processarchitecture
• Taskselecaon• Taskpresentaaon• Evidenceidenaficaaon• Evidenceaccumulaaon
Mislevy,AlmondandLukas(2003);Almond,SteinbergandMislevy(2003)
![Page 68: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/68.jpg)
Taskselec$on
68
![Page 69: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/69.jpg)
KintherLay$cks
SkondohasohenbeendescribedasoneofthefantemgrowingplaidosintheUKduringthelast10years,butthelureofchemicksaboutintabselhasconanuedtoafracttheafenaonofmooricknumbersofBritons.
Thepercentageriseintranspitansinthelastdecadedoesnotmatchtheskondoboombutincreasingtranspitancyhasbeentakingplacesincetheearlynineaesandthedemandonourtuwoaitchanddadinisrevealsthespectacularmoory.
Unfortunately,unlikeskondo,theplaidooflayackshasafendantsnuffsemfortheenthusiasacbutrudioamateur.Alltoofewofthesatsunlayboswhotaketothetuwoahhaveeventhemostrudimentaryknowledgeofloxemintabsel.
1. Nametwopopularplaidos.
2. Havetherebeenmany
deathsfromSkondo?
3. Whichcountryhasalotof
kintherlayacks?
4. Writedowntwo
precauaonstotakefor
layacks
5. Whatissnuffsemabout
skondo?
6. Whatwouldyoufindin
dadinis?
69
![Page 70: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/70.jpg)
Taskpresenta$on
70
![Page 71: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/71.jpg)
Itemformats
• “Noassessmenttechniquehasbeenrubbishedquitelikemulaplechoice,unlessitbegraphology”Wood,1991,p.32)
• Mythsaboutmulaple-choiceitems– Theyarebiasedagainstfemales– Theyassessonlycandidates’abilitytospotorguess– Theytestonlylower-orderskills
![Page 72: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/72.jpg)
Diagnos$cques$onsinEnglish
Inapieceofpersuasivewriang,whichofthesewouldbethebestthesisstatement?
A. ThetypicalTVshowhas9violentincidentsB. ThereisalotofviolenceonTVC. TheamountofviolenceonTVshouldbereducedD. SomeprogramsaremoreviolentthanothersE. ViolenceisincludedinprogramstoboostraangsF. ViolenceonTVisinteresangG. Idon’tliketheviolenceonTVH. TheessayIamgoingtowriteisaboutviolenceonTV
![Page 73: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/73.jpg)
Evidenceiden$fica$on
73
![Page 74: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/74.jpg)
Referentsinassessment
• Norm-referenced– agroupwhowereassessedpreviously
• Cohort-referenced– thegroupassessedatthesameame
• Criterion-referenced– explicitandpreciseperformancecriteria
• Ipsaave– definedonlywithinanindividual
• Construct-referenced– asharedconstructinacommunityofpracace
![Page 75: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/75.jpg)
Quality
“Maximscannotbeunderstood,salllessappliedbyanyonenotalreadypossessingagoodpracacalknowledgeoftheart.Theyderivetheirinterestfromourappreciaaonoftheartandcannotthemselveseitherreplaceorestablishthatappreciaaon”.(Polanyi,1958p.50).“Qualitydoesn’thavetobedefined.Youunderstanditwithoutdefiniaon.Qualityisadirectexperienceindependentofandpriortointellectualabstracaons”.(Pirsig,1991p.64).
![Page 76: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/76.jpg)
Evidenceaccumula$on
76
![Page 77: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/77.jpg)
Memoryonlandandunderwater
• 18(5f,13m)studentmembersofauniversitydivingclubweretestedontheirrecalloftwo-andthree-syllablewordsfromfour36-wordliststakenfromtheTorontoWordBankspokentothemtwice.
• Studentslearned,andweretestedon,thewordswhileunderwater,andwhileontheshore,resulanginfourcondiaons:– DD(learndry,recalldry)– DW(learndry,recallwet)– WD(learnwet,recalldry)– WW(learnwet,recallwet)
77
![Page 78: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/78.jpg)
Memoryiscontext-dependent78
Recallenvironment
Dry Wet
Learningenvironment
Dry 13.5 8.6
Wet 8.4
GoddenandBaddeley(1975)
Nosignificantmaineffects;interacaoneffect:F=22.0;df=1,12;p=<0.001
11.4
![Page 79: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/79.jpg)
Alcoholandmemory
NumberofitemscorrectDay1 Day2
Day1:sober;day2:sober 17 17Day1:sober;day2:intoxicated 17 11Day1:intoxicated;day2:sober 18 13Day1:intoxicated;day2:intoxicated 16
• 32adults(aged22to43)askedtomemorizeamapanda19-itemsetofinstrucaonsforajourney
• Halfdidsosoberandhalfatthelegallimitforintoxicaaon• Thefollowingday,halfofthemweretestedsoberandhalfat
thelegallimitforintoxicaaon.
Lowe(1981)
16
79
![Page 80: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/80.jpg)
Discussionquesaon80
Discussio
n • Howwillyoudecidehowmuchevidenceisneededtodecidewhetherastudenthasreachedlearnedsomething?
![Page 81: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/81.jpg)
Recording
81
![Page 82: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/82.jpg)
SylvieandBrunoconcluded(Carroll,1893)
“That’sanotherthingwe’velearnedfromyourNaaon,”saidMeinHerr,“map-making.Butwe’vecarrieditmuchfurtherthanyou.Whatdoyouconsiderthelargestmapthatwouldbereallyuseful?”“Aboutsixinchestothemile.”“Onlysixinches!”exclaimedMeinHerr.“Weverysoongottosixyardstothemile.Thenwetriedahundredyardstothemile.Andthencamethegrandestideaofall!Weactuallymadeamapofthecountry,onthescaleofamiletothemile!”“Haveyouuseditmuch?”Ienquired.“Ithasneverbeenspreadout,yet,”saidMeinHerr:“thefarmersobjected:theysaiditwouldcoverthewholecountry,andshutoutthesunlight!Sowenowusethecountryitself,asitsownmap,andIassureyouitdoesnearlyaswell.
82
![Page 83: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/83.jpg)
Whatisagrade?
“…aninadequatereportofaninaccuratejudgmentbyabiasedandvariablejudgeoftheextenttowhichastudenthasafainedanundefinedlevelofmasteryofanunknownproporaonofanindefinitematerial.”(Dressel,quotedinChickering,1983p.12)
![Page 84: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/84.jpg)
Repor$ng
84
![Page 85: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/85.jpg)
Effectsoffeedback
• Kluger&DeNisi(1996)• Reviewof3000researchreports• Excludingthose:
– withoutadequatecontrols– withpoordesign– withfewerthan10paracipants– whereperformancewasnotmeasured– withoutdetailsofeffectsizes
• leh131reports,607effectsizes,involving12652individuals
• Onaveragefeedbackdoesimproveperformance,but– Effectsizesverydifferentindifferentstudies– In38%(50outof131)ofstudies,effectsizeswerenegaave
![Page 86: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/86.jpg)
GeSngfeedbackrightishard
Responsetype Feedbackindicatesperformance…
exceedsgoal fallsshortofgoal
Changebehavior Exertlesseffort Increaseeffort
Changegoal Increaseaspira$on Reduceaspiraaon
Abandongoal Decidegoalistooeasy Decidegoalistoohard
Rejectfeedback Feedbackisignored Feedbackisignored
![Page 87: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/87.jpg)
Meaningsandconsequencesofschoolgrades
• Tworaaonalesforgrading– Meanings
• Assessmentasevidenaaryreasoning• Assessmentoutcomesassupportsformakinginferences
– (e.g.,aboutstudentachievement)
– Consequences• Assessmentoutcomesasrewardsandpunishments• Assessmentscreateincenavesforstudentstodowhatwewantthemtodo
– Thesetworaaonalesinteract,andconflict• achievementgradesforcompleaonofhomework• achievementgradesforeffort• penalaesforlatesubmission• zeroesformissingwork
87
![Page 88: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/88.jpg)
Dual-pathwaytheory(Boekaerts,2006)
• Long-termlearninggoalsaretranslatedintoshort-termlearningintenaons
• Dynamiccomparisonsoftaskandsituaaonaldemandswithpersonalresources,takingintoaccount:– Currentpercepaonsofthetask– Beliefsaboutthesubjectortask– Beliefsabout“ability”andtheroleofeffortinthesubject– Interestinthesubject(personalvs.situaaonal)– Previousexperiencesonsimilartasks– Costsandbenefits
88
![Page 89: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/89.jpg)
Andthenitcomesdownto…
• Resulangacavaaonofenergyalongoneoftwopathways– Wellbeing– Growth
• Weneedassessmentsystemsthatpushourstudentstowardsafocusongrowth,ratherthanwellbeing
![Page 90: Understanding Assessments: What they Mean and What they Do](https://reader030.fdocuments.net/reader030/viewer/2022012101/61db4a09dc78fa0fe9230ba5/html5/thumbnails/90.jpg)
Summary
• Beforewecanassess,weneedclearmodelsofprogression
• Validityisnotapropertyoftestsorassessments,butofinferences,whichareweakenedby– constructunderrepresentaaon– construct-irrelevantvariance
• Reliabilityisakeyrequirementforvalidity• Limitedtestreliabilityhasparacularlysevereconsequencesforchangesscoresanddiagnosis
• Assessmentsareimportantforwhattheydoaswellaswhattheymean