Panelist: Paula LorgellyDeputy Director, OHEISPOR European Congress, Vienna, 2016
Anchoring Vignettes: identifying response bias and DIF in self assessed health
Anchoring Vignettes: identifying response bias and DIF in self assessed health
Background•Individual and household surveys often rely on self-assessed measures of health
• In general, would you say your health is: excellent, very good, good, fair or poor?
•Analyses using measures of self-assessed health (SAH) rely on the measure being an accurate reflection of the true health of the groups or individuals concerned•But responses to questions on subjective scales will be inaccurate if groups of individuals systematically differ in their use and/or interpretation of the response categories•Systematic variation in the use of response categories is known as reporting heterogeneity or response scale heterogeneity or differential item functioning (DIF)
2
Anchoring Vignettes: identifying response bias and DIF in self assessed health
DIF and the EQ-5D• The EQ-5D is the most commonly used instrument for measuring
preference-based health-related quality of life (HRQoL) • Commonly used in economic evaluations, but increasingly
collected via routine data collection in health care systems (PROMs programmes in England, Sweden and Canada) and included in health surveys as a measure of population health status
• When used to measure and compare health profiles or utilities across sub-groups of the population, the results will be misleading if groups systematically differ in use of response categories
• Could the EQ-5D suffer from DIF like other SRH measures?
Anchoring Vignettes: identifying response bias and DIF in self assessed health4
Anchoring Vignettes: identifying response bias and DIF in self assessed health
Differential Item Functioning
τ4
τ3
τ2
τ1
No problems
Slight problems
Moderate problems
Severe problems
Unable to walk
Group 2
Und
erly
ing
late
nt h
ealth
sca
le fo
r mob
ility τ4
τ3
τ2
τ1
No problems
Slight problems
Moderate problems
Severe problems
Unable to walk
Group 1High mobility
Low mobility
Group 2’s mean health
Group 1’s mean health
Anchoring Vignettes: identifying response bias and DIF in self assessed health
Anchoring vignettes• In order to obtain any meaningful comparison between the health
of groups 1 and 2 it is essential to adjust for DIF • Anchoring vignettes (King et al. 2004) can be used to adjust for
DIF • Previously been used to address DIF in political efficacy,
job/income/life satisfaction, general/specific health measures• Vignette - a brief health description of a hypothetical individual• Respondents are asked to rate the health state described by the
vignette using the same ordered categories they use to rate their own health
• Since the actual level of health of the people in the vignettes is the same for all respondents, the variation in ratings can be used to identify and correct for DIF
Anchoring Vignettes: identifying response bias and DIF in self assessed health
Anchoring vignettes• Example of a vignette for the mobility domain:
Belinda walks for one or two kilometres and climbs three flights of stairs every day without tiring.
Select the one option that best describes Belinda’s mobility:
She has no problems with walking around She has slight problems with walking around She has moderate problems with walking around She has severe problems with walking around She is unable to walk around
Anchoring Vignettes: identifying response bias and DIF in self assessed health
Anchoring vignettes• Typically, a series of vignettes are presented for each health
construct of interest, at varying levels of severity• Suppose we give groups 1 and 2 two vignettes to rate, of differing
severity:• Vignette 1 – limited problems in walking around• Vignette 2 – more problems in walking around
8
Anchoring Vignettes: identifying response bias and DIF in self assessed health
Anchoring vignettes
τ4
τ3
τ2
τ1
No problems
Slight problems
Moderate problems
Severe problems
Unable to walk
Group 2
Und
erly
ing
late
nt h
ealth
sca
le fo
r mob
ility
Vignette 2
Vignette 1
High mobility
Low mobility
τ4
τ3
τ2
τ1
No problems
Slight problems
Moderate problems
Severe problems
Unable to walk
Group 1
Anchoring Vignettes: identifying response bias and DIF in self assessed health
Necessary assumptions• Vignette equivalence (VE) holds if all respondents interpret the
health states described by the vignettes in the same way and on the same uni-dimensional scale, aside from random error.• VE is demonstrated in the example above by the horizontal
dotted lines • Response consistency (RC) is where respondents rate the health
of the hypothetical people described in the vignettes in the same way or using the same underlying scale that they would rate their own health. • RC would be violated if, for example, respondents rated the
health described by the vignettes either more or less harshly than they did their own health
Anchoring Vignettes: identifying response bias and DIF in self assessed health
What do we know so far?• Au and Lorgelly (2014) Quality of Life Research
• Evidence that vignettes for the EQ-5D-5L are feasible• Suggested improvements required in the wording in order to
improve response consistency• Knott et al (2016) Health Economics
• Considers some of the issues of using vignettes• Reviews benefits of operationalising the approach
• Knott et al (2016) HEDG York Working paper (16/14)• Vignettes can be used identify DIF in the EQ-5D-5L (at least in
certain age groups)• Failure to adjust for DIF can lead to conclusions that are
misleading• Further work is needed to achieve vignette equivalence
Anchoring Vignettes: identifying response bias and DIF in self assessed health
DIF adjusted indices
Female Male
Low ed
ucati
on
Med ed
ucatio
n
High ed
ucatio
n
Born A
ustra
lia
Other E
ngl. S
p.Asia
Other
Married
/de fa
cto
Divorce
d/wido
wed
Never
married
Employe
d
Unemplo
yed
NILF/re
tired
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
Index based on self-reports DIF-adjusted index
EQ-5
D In
dex
Female Male
Low ed
ucati
on
Med ed
ucatio
n
High ed
ucatio
n
Born A
ustra
lia
Other E
ngl. S
p.Asia
Other
Married
/de fa
cto
Divorce
d/wido
wed
Never
married
Employe
d
Unemplo
yed
NILF/re
tired
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
Index based on self-reports DIF-adjusted index
EQ-5
D In
dex
Diffe
renc
e =
0.04
9
Diffe
renc
e =
0.09
5
Anchoring Vignettes: identifying response bias and DIF in self assessed health
What do we know so far?• Knott & Lorgelly (2016) HESG Paper - summer
• It is possible to correct for DIF using responses to anchoring vignettes that are collected externally to the main dataset of interest
• Resulting QALY measures can be considered comparable across different population groups
Anchoring Vignettes: identifying response bias and DIF in self assessed health
DIF adjustment – group differences
Male - Female High educ - Low educ Migrant - Born Aus Employed - Not employed
Married - Alone Aged 65 plus - Under 65
-0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
-0.004
0.0539999999999999
0.0379999999999999
0.093
0.0650000000000001
0.08
Unadjusted scores DIF-adjusted scores
Diff
eren
ce in
EQ
-5D
-5L
indi
ces
Male - Female High educ - Low educ Migrant - Born Aus Employed - Not employed
Married - Alone Aged 65 plus - Under 65
-0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.016
0.079
0.037
0.141
0.0960000000000001 0.097
Unadjusted scores DIF-adjusted scores
Diff
eren
ce in
EQ
-5D
-5L
indi
ces
MID=0.074
Anchoring Vignettes: identifying response bias and DIF in self assessed health
Where to next?• More research to better understand the vignette equivalence
failure issue• Will there always be a trade-off with response consistency?
• Is there value in exploring DIF cross-culturally? • Multi-national clinical trials, often apply one country’s tariff as if
all respondents are within that country• Is the external adjustment as good as (or a close substitute for)
collecting them within a study?• What does this mean for economic evaluations and the decisions
they inform? • Could response behaviour change over time?
Anchoring Vignettes: identifying response bias and DIF in self assessed health
For enquiries relating to this presentation, please contact Paula Lorgelly at [email protected]
To keep up with the latest news and research, subscribe to our blog, OHE NewsFollow us on Twitter @OHENews, LinkedIn and SlideShare
Office of Health Economics (OHE)Southside, 7th Floor105 Victoria StreetLondon SW1E 6QT United Kingdom+44 20 7747 8850 www.ohe.orgOHE’s publications may be downloaded free of charge from our website.
Top Related