Panelist: Paula LorgellyDeputy Director, OHEISPOR European Congress, Vienna, 2016

Anchoring Vignettes: identifying response bias and DIF in self assessed health

Anchoring Vignettes: identifying response bias and DIF in self assessed health

Background•Individual and household surveys often rely on self-assessed measures of health

• In general, would you say your health is: excellent, very good, good, fair or poor?

•Analyses using measures of self-assessed health (SAH) rely on the measure being an accurate reflection of the true health of the groups or individuals concerned•But responses to questions on subjective scales will be inaccurate if groups of individuals systematically differ in their use and/or interpretation of the response categories•Systematic variation in the use of response categories is known as reporting heterogeneity or response scale heterogeneity or differential item functioning (DIF)

2

Anchoring Vignettes: identifying response bias and DIF in self assessed health

DIF and the EQ-5D• The EQ-5D is the most commonly used instrument for measuring

preference-based health-related quality of life (HRQoL) • Commonly used in economic evaluations, but increasingly

collected via routine data collection in health care systems (PROMs programmes in England, Sweden and Canada) and included in health surveys as a measure of population health status

• When used to measure and compare health profiles or utilities across sub-groups of the population, the results will be misleading if groups systematically differ in use of response categories

• Could the EQ-5D suffer from DIF like other SRH measures?

Anchoring Vignettes: identifying response bias and DIF in self assessed health4

Anchoring Vignettes: identifying response bias and DIF in self assessed health

Differential Item Functioning

τ4

τ3

τ2

τ1

No problems

Slight problems

Moderate problems

Severe problems

Unable to walk

Group 2

Und

erly

ing

late

nt h

ealth

sca

le fo

r mob

ility τ4

τ3

τ2

τ1

No problems

Slight problems

Moderate problems

Severe problems

Unable to walk

Group 1High mobility

Low mobility

Group 2’s mean health

Group 1’s mean health

Anchoring Vignettes: identifying response bias and DIF in self assessed health

Anchoring vignettes• In order to obtain any meaningful comparison between the health

of groups 1 and 2 it is essential to adjust for DIF • Anchoring vignettes (King et al. 2004) can be used to adjust for

DIF • Previously been used to address DIF in political efficacy,

job/income/life satisfaction, general/specific health measures• Vignette - a brief health description of a hypothetical individual• Respondents are asked to rate the health state described by the

vignette using the same ordered categories they use to rate their own health

• Since the actual level of health of the people in the vignettes is the same for all respondents, the variation in ratings can be used to identify and correct for DIF

Anchoring Vignettes: identifying response bias and DIF in self assessed health

Anchoring vignettes• Example of a vignette for the mobility domain:

Belinda walks for one or two kilometres and climbs three flights of stairs every day without tiring.

Select the one option that best describes Belinda’s mobility:

She has no problems with walking around She has slight problems with walking around She has moderate problems with walking around She has severe problems with walking around She is unable to walk around

Anchoring Vignettes: identifying response bias and DIF in self assessed health

Anchoring vignettes• Typically, a series of vignettes are presented for each health

construct of interest, at varying levels of severity• Suppose we give groups 1 and 2 two vignettes to rate, of differing

severity:• Vignette 1 – limited problems in walking around• Vignette 2 – more problems in walking around

8

Anchoring Vignettes: identifying response bias and DIF in self assessed health

Anchoring vignettes

τ4

τ3

τ2

τ1

No problems

Slight problems

Moderate problems

Severe problems

Unable to walk

Group 2

Und

erly

ing

late

nt h

ealth

sca

le fo

r mob

ility

Vignette 2

Vignette 1

High mobility

Low mobility

τ4

τ3

τ2

τ1

No problems

Slight problems

Moderate problems

Severe problems

Unable to walk

Group 1

Anchoring Vignettes: identifying response bias and DIF in self assessed health

Necessary assumptions• Vignette equivalence (VE) holds if all respondents interpret the

health states described by the vignettes in the same way and on the same uni-dimensional scale, aside from random error.• VE is demonstrated in the example above by the horizontal

dotted lines • Response consistency (RC) is where respondents rate the health

of the hypothetical people described in the vignettes in the same way or using the same underlying scale that they would rate their own health. • RC would be violated if, for example, respondents rated the

health described by the vignettes either more or less harshly than they did their own health

Anchoring Vignettes: identifying response bias and DIF in self assessed health

What do we know so far?• Au and Lorgelly (2014) Quality of Life Research

• Evidence that vignettes for the EQ-5D-5L are feasible• Suggested improvements required in the wording in order to

improve response consistency• Knott et al (2016) Health Economics

• Considers some of the issues of using vignettes• Reviews benefits of operationalising the approach

• Knott et al (2016) HEDG York Working paper (16/14)• Vignettes can be used identify DIF in the EQ-5D-5L (at least in

certain age groups)• Failure to adjust for DIF can lead to conclusions that are

misleading• Further work is needed to achieve vignette equivalence

Anchoring Vignettes: identifying response bias and DIF in self assessed health

DIF adjusted indices

Female Male

Low ed

ucati

on

Med ed

ucatio

n

High ed

ucatio

n

Born A

ustra

lia

Other E

ngl. S

p.Asia

Other

Married

/de fa

cto

Divorce

d/wido

wed

Never

married

Employe

d

Unemplo

yed

NILF/re

tired

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Index based on self-reports DIF-adjusted index

EQ-5

D In

dex

Female Male

Low ed

ucati

on

Med ed

ucatio

n

High ed

ucatio

n

Born A

ustra

lia

Other E

ngl. S

p.Asia

Other

Married

/de fa

cto

Divorce

d/wido

wed

Never

married

Employe

d

Unemplo

yed

NILF/re

tired

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Index based on self-reports DIF-adjusted index

EQ-5

D In

dex

Diffe

renc

e =

0.04

9

Diffe

renc

e =

0.09

5

Anchoring Vignettes: identifying response bias and DIF in self assessed health

What do we know so far?• Knott & Lorgelly (2016) HESG Paper - summer

• It is possible to correct for DIF using responses to anchoring vignettes that are collected externally to the main dataset of interest

• Resulting QALY measures can be considered comparable across different population groups

Anchoring Vignettes: identifying response bias and DIF in self assessed health

DIF adjustment – group differences

Male - Female High educ - Low educ Migrant - Born Aus Employed - Not employed

Married - Alone Aged 65 plus - Under 65

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

-0.004

0.0539999999999999

0.0379999999999999

0.093

0.0650000000000001

0.08

Unadjusted scores DIF-adjusted scores

Diff

eren

ce in

EQ

-5D

-5L

indi

ces

Male - Female High educ - Low educ Migrant - Born Aus Employed - Not employed

Married - Alone Aged 65 plus - Under 65

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.016

0.079

0.037

0.141

0.0960000000000001 0.097

Unadjusted scores DIF-adjusted scores

Diff

eren

ce in

EQ

-5D

-5L

indi

ces

MID=0.074

Anchoring Vignettes: identifying response bias and DIF in self assessed health

Where to next?• More research to better understand the vignette equivalence

failure issue• Will there always be a trade-off with response consistency?

• Is there value in exploring DIF cross-culturally? • Multi-national clinical trials, often apply one country’s tariff as if

all respondents are within that country• Is the external adjustment as good as (or a close substitute for)

collecting them within a study?• What does this mean for economic evaluations and the decisions

they inform? • Could response behaviour change over time?

Anchoring Vignettes: identifying response bias and DIF in self assessed health

For enquiries relating to this presentation, please contact Paula Lorgelly at plorgelly@ohe.org

To keep up with the latest news and research, subscribe to our blog, OHE NewsFollow us on Twitter @OHENews, LinkedIn and SlideShare

Office of Health Economics (OHE)Southside, 7th Floor105 Victoria StreetLondon SW1E 6QT United Kingdom+44 20 7747 8850 www.ohe.orgOHE’s publications may be downloaded free of charge from our website.