RELIABILITY consistency or reproducibility of a test score (or measurement)
-
date post
20-Dec-2015 -
Category
Documents
-
view
223 -
download
0
Transcript of RELIABILITY consistency or reproducibility of a test score (or measurement)
RELIABILITY
consistency or reproducibility of a test score (or measurement)
Common approaches to estimating reliability
Classical True Score Theory– test-retest, alternate forms, internal consistency
useful for estimating relative decisions
– intraclass correlation useful for estimating absolute decisions
Generalizability Theory– can estimate both relative & absolute
Reliability is a concept central to all behavioral sciences. To some extent all measures are unreliable. This is especially true with psychological measures and measurements based on human observation
Sources of Error
Random– fluctuations in the measurement based purely
on chance. Systematic
– Measurement error that affect a score because of some particular characteristic of the person or the test that has nothing to due with the construct being measured.
CTST
X = T + E– Recognizes only two sources of variance
test -retest (stability) alternate forms (equivalence in item sampling) test-retest with alternate forms (stability &
equivalence but these are confounded)
– Cannot adequately estimate individual sources of error influencing a measurement
ICC
Uses ANOVA to partition variance due to between subjects and within subjects– Has some ability to accommodate multiple
sources of variance– Does not provide an integrated approach to
estimating reliability under multiple conditions
Generalizability Theory
The Dependability of Behavioral Measures, (1972) Cronbach, Glaser, Nanda, & Rajaratnam
Dependability
The accuracy of generalizing from a person’s observed score on a measure to the average score that person would have received under all possible testing conditions the tester would be willing to accept.
The Decision Maker
The score on which the decision is to be based is only one of many scores that might serve the same purpose. The decision maker is almost never interested in the response given to the particular moment of testing.
Ideally the decision should be based on that person’s mean score over all possible measurement occasions.
Universe of Generalization
Definition & establishment of the universe admissible observations:– observations that the decision maker is willing
to treat as interchangeable.– all sources of influence acting on the measurement of the trait under study.
What are the sources of ERROR influencing your measurement?
Generalizability Issues
Facet of Generalization– raters, trials, days, clinics, therapists
Facet of Determination– usually people, but can vary (e.g. raters)
Types of Studies
Generalizability Study (G-Study)
Decision Study (D-Study)
G-Study
Purpose is to anticipate the multiple uses of a measurement.
To provide as much information as possible about the sources of variation in the measurement.
The G-Study should attempt to identify and incorporate into its design as many potential sources of variation as possible.
D-Study
Makes use of the information provided by the G-Study to design the best possible application of the measurement for a particular purpose.
Planning a D-Study:– defines the Universe of Generalization– specifies the proposed interpretation of the
measurement.– uses G-Study information to evaluate the effectiveness
of alternative designs for minimizing error and maximizing reliability.
Design Considerations
Fixed Facets
Random Facets
Fixed Facet
When the levels of the facet exhaust all possible conditions in the universe to which the investigator wants to generalize.
When the level of the facet represent a convenient sub-sample of all possible conditions in the universe.
Random Facets
When it is assumed that the levels of the facet represent a random sample of all possible levels described by the facet.
If you are willing to EXCHANGE the conditions (levels) under study for any other set of conditions of the same size from the universe.
Types of Decisions
Relative– establish a rank order of individuals (or
groups).– the comparison of a subject’s performance
against others in the group. Absolute
– to index an individual’s (or group’s) absolute level of measurement.
– measurement results are to be made independent from the performance of others in the group.
Statistical Modeling
ANOVA – just as ANOVA partitions a dependent variable
into effects for the independent variable (main effects & interactions), G-theory uses ANOVA to partition an individual’s measurement score into an effect for the universe-score and an effect for each source of error and their interactions in the design.
In ANOVA we were driven to test specific hypotheses about our independent variables and thus sought out the F statistic and p-value.
In G-theory we will use ANOVA to partition the different sources of variance and then to estimate their amount (Variance Component).
Statistical Modeling
One Facet Design
4 Sources of Variability– systematic differences among subjects
(object of measurement)
– systematic differences among raters (occasions, items)
– subjects*raters interaction– random error
confounded
Two Facet DesignComponents of Variance
Example of a fully crossed two facet design (Kroll, et. al.)
Seven sources of variance are estimated:– subjects– raters– observations– sxr– sxo– rxo– sxrxo,e
Variance Components
Subjects (s) Observations (o)
Raters (r)
(sxr) (oxr)
(sxo)
(sxrxo)+
Error
TABLE 1 - Variance Components and Percentage of Variation for Measures of Pelvic Tilt (raters=2,observations=5)
Resting Pelvic Tilt Anterior Pelvic Tilt Posterior Pelvic Tilt
Source ofVariation
VC Percent VC Percent VC Percent
Persons 19.956 75.2 47.683 84.8 20.607 72.3
Raters 1.726 6.5 0.000 0.0 2.508 8.8
Observations 0.148 0.6 0.000 0.0 0.011 0.0
P x R 1.671 6.3 1.935 3.4 1.910 6.7
P x O 0.042 0.2 0.972 1.7 1.077 3.8
R x O 0.000 0.0 0.000 0.0 0.000 0.0
P x R x O, E 3.050 11.5 5.646 10.0 2.394 8.4
Abbreviations: P x R = persons by raters; P x O = persons by observations; R x O = raters byobservations; P x R x O, E = persons by raters by observations combined with residual error
TABLE 2 - Variance Components and Percentage of Variation for Modified Schober, Attraction Method,and Lower Abdominal Strength Measures (raters=2, observations=3)
Modified Schober Attraction Method Lower AbdominalStrength
Source VC Percent VC Percent VC Percent
Persons 1.006 67.8 0.360 81.3 105.055 52.9
Raters 0.000 0.0 0.000 0.0 0.000 0.0
Observations 0.008 0.5 0.000 0.0 0.000 0.0
P x R 0.181 12.2 0.000 0.0 71.349 36.0
P x O 0.029 2.0 0.083 18.7 3.695 1.9
R x O 0.016 1.1 0.000 0.0 0.757 0.4
P x R x O, E 0.243 16.4 0.000 0.0 17.577 8.9
Abbreviations: P x R = persons by raters; P x O = persons by observations; R x O = raters byobservations; P x R x O, E = persons by raters by observations combined with residual error
Relative ErrorFacet of Determination: Subjects
Subjects (s) Observations (o)
Raters (r)
(sxr) (oxr)
(sxo)
(sxrxo)+
Error
2rel = 2
sr /nr + 2so /no+ 2
sro,e/nrno
Absolute ErrorFacet of Determination: Subjects
Subjects (s) Observations (o)
Raters (r)
(sxr) (oxr)
(sxo)
(sxrxo)+
Error
2abs = 2
r/nr + 2o /no + 2
sr /nr + 2so /no + 2
or /nonr + 2sro,e /nonr
Generalizability CoefficientsAKA: Reliability Coefficients
Absolute Generalizability Coefficient for Subjects:
2s
= -------------2
s + 2abs
Relative Generalizability Coefficient for Subjects:
2s
2 = -------------2
s + 2rel
TABLE 3 - Variance Components and Percentage of Variation for Right and Left Hamstring FlexibilityMeasures (raters = 2, observations = 3)
Right HamstringFlexibility
Left HamstringFlexibility
Source VC Percent VC Percent
Persons 398.526 93.1 382.639 91.9
Raters 0.000 0.0 0.000 0.0
Observations 1.767 0.4 2.123 0.5
P x R 20.656 4.8 24.030 5.8
P x O 0.708 0.2 1.235 0.3
R x O 0.001 0.0 0.707 0.2
P x R x O, E 6.407 1.5 5.727 1.4
Abbreviations: P x R = persons by raters; P x O = persons by observations; R x O = raters byobservations; P x R x O, E = persons by raters by observations combined with residual error
TABLE 4 - Variance Component and Percentage of Variation of Abdominal and Trunk MuscleEndurance Methods (raters=2, observation=2)
Abdominal MuscleEndurance
Trunk MuscleEndurance
Source VC Percent VC Percent
Persons 646.177 68.9 1160.656 83.6
Raters 43.936 4.7 0.000 0.0
Observations 0.000 0.0 0.000 0.0
P x R 0.000 0.0 21.732 1.6
P x O 15.736 1.7 24.559 1.8
R x O 0.000 0.0 0.000 0.0
P x R x O, E 232.117 24.7 181.944 13.1
Abbreviations: P x R = persons by raters; P x O = persons by observations; R x O = raters byobservations; P x R x O, E = persons by raters by observations combined with residual error
TABLE 5 - Generalizability of Pelvic Tilt Measures
Resting Pelvic Tilt Anterior Pelvic Tilt Posterior Pelvic Tilt
G-study D-study G-study D-study G-study D-study
nr =no =
25
11
25
11
25
11
ρ2 0.946 0.809 0.967 0.848 0.936 0.793
φ 0.907 0.750 0.967 0.848 0.886 0.723
Abbreviations: nr = number of raters; no = number of observations; ρ2 = generalizability (G) coefficient forrelative decisions; φ = G-coefficient for absolute decisions
TABLE 6 - Generalizability of Trunk Flexibility and Strength Measures
ModifiedSchober
AttractionMethod
LowerAbdominalStrength
RightHamstringFlexibility
LeftHamstringFlexibility
study G D G D G D G D G D
n r =no =
23
11
23
11
23
11
23
11
23
11
ρ2 0.877 0.690 0.928 0.813 0.752 0.531 0.972 0.935 0.966 0.925
φ 0.873 0.678 0.928 0.813 0.724 0.530 0.970 0.931 0.964 0.919
Abbreviations: nr = number of raters; no = number of observations; ρ2 = generalizability (G) coefficient forrelative decisions; φ = G-coefficient for absolute decisions
TABLE 7 - Generalizability of Trunk Endurance Measures
Flexion Extension
G-study D-study G-study D-study
nr =no =
22
11
22
11
ρ2 0.908 0.723 0.944 0.836
φ 0.880 0.689 0.944 0.836
Abbreviations: nr = number of raters; no = number of observations; ρ2 = generalizability (G) coefficient forrelative decisions; φ = G-coefficient for absolute decisions