Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or...
-
date post
22-Dec-2015 -
Category
Documents
-
view
224 -
download
6
Transcript of Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or...
![Page 1: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/1.jpg)
Measurement
Joseph Stevens, Ph.D.
© 2005
![Page 2: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/2.jpg)
Measurement Process of assigning quantitative or qualitative
descriptions to some attribute Operational Definitions
Assessment Collection of measurement information Interpretation Synthesis Use
Evaluation Value added to assessment information (e.g.
good, poor, “ought”, “needs improvement”)
![Page 3: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/3.jpg)
Assessment Decisions/Purposes Instructional Curricular Treatment/Intervention Placement/Classification Selection/Admission Administration/Policy-making Personal/Individual Personnel Evaluation
![Page 4: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/4.jpg)
Scaling
Process of systematically translating empirical observations into a measurement scale
Origin Units Information Types of scales
![Page 5: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/5.jpg)
Score Interpretation
Direct interpretation Need for analysis, relative
interpretation Normative interpretation Anchoring/Standards
![Page 6: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/6.jpg)
Frames of Reference for Interpretation
Current versus future performance Typical versus maximum or potential Standard of comparison
To self To others To standard
Formative versus summative
![Page 7: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/7.jpg)
Domains Cognitive
Ability/Aptitude Achievement Memory, perception, etc.
Affective Beliefs Attitudes Feelings, interests, preferences,
emotions Behavior
![Page 8: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/8.jpg)
Cognitive Level
Knowledge Comprehension Application Analysis/Synthesis Evaluation
![Page 9: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/9.jpg)
Assessment Tasks Selected Response – MC, T-F, matching Restricted Response – cloze, fill-in,
completion Constructed Response - essay Free Response/Performance Assessments
Products Performances
Rating Ranking Magnitude Estimation
![Page 10: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/10.jpg)
CRT versus NRT
Criterion Referenced Tests (CRT) Comparison to a criterion/standard Items that represent the domain
Relevance Representativeness
Norm Referenced Tests Comparison to a group Items that discriminate one person from
another
![Page 11: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/11.jpg)
Kinds of Scores
Raw Standard scores Developmental Standard Scores Percentile Ranks (PR) Normal Curve Equivalent (NCE) Grade Equivalent (GE)
![Page 12: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/12.jpg)
Scoring Methods
Objective Subjective
Holistic Analytic
![Page 13: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/13.jpg)
![Page 14: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/14.jpg)
Standard
MetDid Not Meet
Pe
rce
nt
100
80
60
40
20
0
![Page 15: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/15.jpg)
Aggregating Scores
Total scores Summated scores Composite scores
Issues Intercorrelation of components Variance Reliability
![Page 16: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/16.jpg)
Theories of Measurement
Classical Test Theory (CTT)X = T + E
Item Response Theory (IRT)http://work.psych.uiuc.edu/irt/tutorial.asp
x
x
1(
eePg
![Page 17: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/17.jpg)
Logistic Reponse Model Item: 2The parameter a is the item discriminating power, the reciprocal (1/a) is the itemdispersion, and the parameter b is an item location parameter.
0
0.2
0.4
0.6
0.8
1.0
-3 -2 -1 0 1 2 3
b
Ability
Pro
bab
ilit
y
Item Characteristic Curve: 2 a = 0.725 b = -1.367
![Page 18: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/18.jpg)
Logistic Reponse Model Item: 3The parameter a is the item discriminating power, the reciprocal (1/a) is the itemdispersion, and the parameter b is an item location parameter.
0
0.2
0.4
0.6
0.8
1.0
-3 -2 -1 0 1 2 3
b
Ability
Pro
bab
ilit
y
Item Characteristic Curve: 3 a = 0.885 b = -0.281
![Page 19: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/19.jpg)
Reliability
Consistency Consistency of Decisions Prerequisite to validity Errors in measurement
![Page 20: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/20.jpg)
Reliability Sources of errors
Variations in physical and mental condition of person measured
Changes in physical or environmental conditions Tasks/Items Administration conditions Time Skill to skill Raters/judges Test forms
![Page 21: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/21.jpg)
Estimating Reliability
Reliability versus standard error of measurement (SEM)
Internal Consistency Cronbach’s alpha Split-half Example
Test-Retest Inter-rater
![Page 22: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/22.jpg)
Estimating Reliability
Correlations, rank order versus exact agreement
Percent Agreement Exact versus close (number of agreements/number of
scores x 100) Problem of chance agreements
![Page 23: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/23.jpg)
Estimating Reliability Kappa Coefficient
Takes chance agreements into account Calculate expected frequencies and subtract Kappa ≥ .70 acceptable Examine pattern of disagreements
Example Percent agreement = 63.8% r = .509 Kappa = .451
![Page 24: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/24.jpg)
Below Meets Exceeds Total
Below 9 3 1 13
Meets 4 8 2 14
Exceeds 2 1 6 9
Total 15 12 9 36
![Page 25: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/25.jpg)
Estimating Reliability
Spearman-Brown prophecy formula More is better
![Page 26: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/26.jpg)
Reliability as error
Systematic error Random error SEM _______
SEM = SDx √ 1 - rxx
![Page 27: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/27.jpg)
Factors affecting reliability
Time limits Test length Item characteristics
Difficulty Discrimination
Heterogeneity of sample Number of raters, quality of
subjective scoring
![Page 28: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/28.jpg)
Validity
Accuracy Unified View (Messick)
Use and Interpretation Evidential basis
Content Criterion Concurrent-Discriminant Construct
Consequential basis
![Page 29: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/29.jpg)
Validity
Internal, structural Multitrait-Multimethod (Campbell &
Fiske) Predictive
![Page 30: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/30.jpg)
Test Development
Construct Representation Content analysis Review of research Direct observation Expert judgment (panels, ratings, Delphi) Instructional objectives
![Page 31: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/31.jpg)
Test Development Blueprint
Content X Process Domain sampling Item frames Matching item type and response format
to purpose Item writing Item Review (grammar, readability,
cueing, sensitivity)
![Page 32: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/32.jpg)
Test Development
Writing instructions Form design (NAEP brown ink) Field and pilot testing Item analysis Review and revision
![Page 33: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/33.jpg)
Equating
Need to link across forms, people, or occasions
Horizontal equating Vertical equating Designs
Common item Common persons
![Page 34: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/34.jpg)
Equating
Equipercentile Linear IRT
![Page 35: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/35.jpg)
Bias and Sensitivity
Sensitivity in item and test development
Differential results versus bias Differential Item Functioning (DIF) Importance of matching, legal versus
psychometric Understanding diversity and individual
differences
![Page 36: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/36.jpg)
Item Analysis
Difficulty, p Means and standard deviations Discrimination, r-point biserial Omits Removing or revising “bad” items Example
![Page 37: Measurement Joseph Stevens, Ph.D. © 2005. Measurement Process of assigning quantitative or qualitative descriptions to some attribute Operational Definitions.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d815503460f94a65cde/html5/thumbnails/37.jpg)
Factor Analysis
Method of evaluating structural validity and reliability
Exploratory (EFA) example Confirmatory (CFA) example