© Nuffield Trust 17 July 2014

Use of retrospective matching methods to

study health services and other sectors

Martin Bardsley Nuffield Trust

© Nuffield Trust

Tomb raiders?

Adapted from slide by Iain Buchan Manchester University

Audit and Quality Improvement

Patient safety (e.g. monitoring drug

side effects or surgical mortality

rates)

Public Health programmes

(immunisation; monitoring cancer

rates)

Evaluate Services (are they

effective and cost effective?)

Planning services (e.g. ICU bed

availability; pandemic flu plans;

manage changing patterns of

demand)

Manage Performance (e.g.

readmission targets; health

outcomes indicators)

Resource allocation

Research

© Nuffield Trust

Health and social care timeline – an individual’s

history

© Nuffield Trust

Evidence for evaluation

Evidence exists in different forms

Not everything can be a prospective

randomised trial

Observational data can look at how

things work in real life

Detailed linked data sets allow quite

sophisticated ways to standardise for

prior risk

Aslam S et al Matching research design to clinical research

questions. Indian J Sex Transm Dis 2012;33:49-53

© Nuffield Trust

Methods using matched controls

Idea is to track changes over time in a study cohort before and after they receive a service. (eg emergency admissions)

Identify people who didn’t receive the service but look almost the same.

Match on an individual basis (1:1 or 1 to many) using propensity score matching (similarity at baseline) or prognostic matching (similar on expected outcomes).

Balance across a wide range of variables.

Com[are outcome measure between the two groups

© Nuffield Trust

IC collates and adds

HES IDs Sites collate patient lists

Patient identifiers

(e.g. NHS number)

Trial information (e.g.

start and end date)

Non-patient identifiable keys

(e.g. HES ID)

Participating sites Information Centre

Nuffield Trust

Lining local site data to national data sets

© Nuffield Trust

Evaluation: The Marie Curie Nursing Service

Intervention:

• Nursing care support to people at end of life, in their homes

Nuffield commissioned to evaluate impact:

• Are recipients more likely to die at home?

• Reduction in emergency hospital admissions at end of life?

Methods:

• Retrospective matched control study – use of already existing administrative data

© Nuffield Trust

0%

10%

20%

30%

40%

50%

Comorbidities

0%

5%

10%

15%

20%

25%

30%

35%

Cancer diagnoses

Control group – how well matched? Diagnostic history

0%

10%

20%

30%

40%

50%

Comorbidities

0%

5%

10%

15%

20%

25%

30%

35%

Cancer diagnoses

Marie Curie Controls

© Nuffield Trust

© Nuffield Trust

And for 3 virtual wards…

© Nuffield Trust

Impact of eight different interventions on hospital use

© Nuffield Trust

Results (case management interventions only)

• Effects were more

pronounced for the case

management interventions.

• We found evidence of

imperfect matching of cases

and controls for the case

management pilots.

• Sensitivity analysis showed

that we cannot be sure that

the pilots increased

emergency admissions, but it

is unlikely that they reduced

them.

Difference in difference analysis

(individual patient level)

Absolute

difference

(per head)

Relative

difference

p-value

Emergency

admissions 0.046 9% 0.02

A&E

attendance -0.016 -3% 0.40

Elective

admissions -0.107 -21% <.01

Outpatient

attendance -0.545 -22% <.01

© Nuffield Trust

Potential of retrospective matching methods

• Powerful but not perfect. Two key weaknesses in terms of ‘hidden confounders’ being limited to outcome recorded on routine data

• Quick and cheap compared to prospective data collection – but can still require quite intensive analysis?

• Can we make these types of approaches more accessible to providers and commissioners?

© Nuffield Trust 17 July 2014

www.nuffieldtrust.org.uk

Sign-up for our newsletter www.nuffieldtrust.org.uk/newsletter

Follow us on Twitter: Twitter.com/NuffieldTrust

© Nuffield Trust

Ian.blunt@nuffieldtrust.org.uk

Adam.steventon@nuffieldtrust.org.uk