© Nuffield Trust 18 February 2015

Using big datasets in evaluating health

service interventions in community

settings.

Digital Health Assembly: Open Innovation International Conference, 10 – 12 February 2015, SWALEC Stadium, Cardiff

Dr Martin Bardsley

Director of Research

Nuffield Trust

© Nuffield Trust

• Promote independent analysis

and informed debate on

healthcare policy across the UK

• Charitable organization founded

in 1940

• Formerly a grant-giving

organization

• Since 2008 we have been

conducting in-house research

and policy analysis

• Significant interest in uses of

predictive risk techniques

The Nuffield Trust

William Morris

1st Viscount Nuffield

(1877 -1963)

© Nuffield Trust

Background

Pace of change in health service delivery continually increasing

Demands for evidence based policy but information about ‘what works’ and its generalisability often limited

But collecting bespoke information slow and costly.

a. Making the most of existing data

b. In prospective studies eg RCTs

c. In retrospective matching studies

© Nuffield Trust

Ten-year trend in emergency admissions (46 million admits)

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

200

1/0

2 Q

1

200

1/0

2 Q

3

200

2/0

3 Q

1

200

2/0

3 Q

3

200

3/0

4 Q

1

200

3/0

4 Q

3

200

4/0

5 Q

1

200

4/0

5 Q

3

200

5/0

6 Q

1

200

5/0

6 Q

3

200

6/0

7 Q

1

200

6/0

7 Q

3

200

7/0

8 Q

1

200

7/0

8 Q

3

200

8/0

9 Q

1

200

8/0

9 Q

3

200

9/1

0 Q

1

200

9/1

0 Q

3

201

0/1

1 Q

1

201

0/1

1 Q

3Nu

mb

er

of

em

erg

en

cy a

dm

issio

ns

(millio

ns)

No ACS diagnosis ACS primary diagnosis ACS secondary diagnosis

+35% (40%)

+34%

© Nuffield Trust

Interventions to reduce avoidable admissions

Primary Care ED Depts Hospital Transition

Practice features Assess/obs wards Structured

Discharge

Transition care

management

Medication review GPs in A&E Medication

Review

Rehabilitation

Case

management

Senior Clinician

Review

Specialist Clinics Self management

and education

Telemedicine Coordination EOL

care

Hospital at home

Virtual Wards

see Purdy et al (2012) Interventions to Reduce Unplanned Hospital Admission: A series of systematic reviews. Bristol University Final Report)

© Nuffield Trust

Many interventions, few proven as effective…

© Nuffield Trust

What information do we have on whether these are working?……

© Nuffield Trust

© Nuffield Trust

There's a lot of data about…

GP

Local Authority

Commissioner

A&E

OP

IP

Pharmacy Community

Health

Services

Up there

Housing Council

Tax

Council

Social

Services

Social care

provider

Ambulance

Control NHS111

Commissioning data ...

© Nuffield Trust

Tomb raiders?

© Nuffield Trust

Exploiting person level data

Linking data

a. over time to look at what happens to people – not just events

b. across care providers to build broader picture

Person level

Capture services provided ->costs; quality

Descriptions of health -> outcomes

© Nuffield Trust

Data linkage of hospital records not new...

“In Britain many important data are

recorded about health and disease, and

we have certain types of health statistics

available on a national scale.... Their

limitation is that they tell little or nothing

concerning the combinations of events

implicit in the terms 'readmission',

'recurrence', 'prognosis‘ and 'differential

survival'; nor about the association

between diseases, and nothing concerning

the patterns of disease in families.”

Acheson ED, Evans JG. The Oxford Record Linkage

Study: A Review of the Method with some Preliminary

Results. Proc R Soc Med. 1964 April; 57(4): 269–274.

© Nuffield Trust

Health and social care timeline – an individual’s

history

© Nuffield Trust

Data linkage

Social & secondary care interface

© Nuffield Trust

Cost of care in last year of life by age

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

<55 55-64 65-74 75-84 85-94 >=95Est

imate

d a

vera

ge c

ost

s p

er

deced

en

t,

£

Age band

Female

All costs

Hospital costs

Social care costs

© Nuffield Trust

What was the average cost of hospital care?

-120 -90 -60 -30 0

Day relative to diagnosis

Emergency inpatient

A&E visit

Elective inpatient

Outpatient attendance

GP visit

Person A

-120 -90 -60 -30 0

Day relative to diagnosis

Emergency inpatient

A&E visit

Elective inpatient

Outpatient attendance

GP visit

Person A

-120 -90 -60 -30 0

Day relative to diagnosis

Emergency inpatient

A&E visit

Elective inpatient

Outpatient attendance

GP visit

Person B

Colorectal Cancer – route to diagnosis

25% colorectal cancers diagnosed following emergency – poor

prognosis for these individuals

We used cancer registry data, linked to GP, hospital data (N = 943)

Did emergency group tend to have prior primary care?

Opportunities for intervention

(or audit)?

Using using existing data for research?

Advantage Disadvantage

• Descriptors of events and health

status

• Constrained by the data that are

collected – and quality/consistency

of coding

• Volume of cases versus costs of

data collection

• Handling sensitive personal

information (+/- consent)

• Comprehensive coverage • Coverage of the data – unknown

unknowns

• Enables retrospective studies/ not

time sensitive

• Volume of data – complex

processing

Using existing data sets within RCTs……

© Nuffield Trust

Impact of telehealth on hospital

utilisation and mortality:

provisional findings from WSD

Authors:

Adam Steventon [1], Martin Bardsley [1], John Billings [2], Jennifer Dixon [1] ,

Helen Doll [3], Shashi Hirani [4], Martin Cartwright [4], Lorna Rixon [4],

Martin Knapp [5], Catherine Henderson [5], Anne Rogers [6], Ray Fitzpatrick [3], Jane Hendy

[7], Stanton Newman [4] (Principal Investigator)

for the Whole System Demonstrator Evaluation Team

[1] The Nuffield Trust

[2] New York University

[3] University of Oxford

[4] City University London

[5] London School of Economics and Political Science

[6] University of Manchester

[7] University of Surrey

What is telehealth?

“the remote exchange of data between a patient

and health care professionals as part of the

diagnosis and management of health care

conditions”

Telehealth devices enable items such as blood

glucose level and weight to be measured by the

patient and transmitted to healthcare professionals

working remotely.

Image is the c

opyrig

ht

of

Tu

nsta

ll G

roup L

td

Long-term conditions of trial participants (n=3,154)

COPD

Diabetes

Heart failure

Multi-dimensional evaluation

March 2011

Theme 1

(Nuffield Trust)

Impact of service

use and associated

costs for the NHS

and social services

All 3,000 people

Theme 2

(City & Oxford)

Participant

reported outcomes

Subset of people

plus their informal

carers

Theme 3

(LSE)

Costs and cost

effectiveness

Subset of people

Theme 4

(Manchester

& Oxford)

Experiences of

service users,

informal carers and

professionals

Qualitative

interviews

Theme 5

(Imperial)

Organisational

factors and

sustainable

adoption and

integration

Qualitative

interviews

Accident and

emergency 350,000 records

Outpatients 1,680,000 records

Inpatients 360,000 records

Social care 240,000 records

Community

matrons 20,000 records

GPs 60 practices

48.5 million records

Relative size of data sets collected for one WSD area

Linking pseudonymised data

March 2011

Patterns of service use (telehealth arm, very high risk)

March 2011

Start of trial

Able to track key trial metrics eg Unadjusted trends in

emergency hospital admissions

Able to look 4 years before start Continuous monitoring in trial period

Compare a range of secondary endpoints

Control Intervention Absolute

difference

Percentage

difference

Mortality (%) 8.3 4.6 -3.7 -44.5%

Emergency admissions per head 0.68 0.54 -0.14 -20.6%

Elective admissions per head 0.49 0.42 -0.07 -14.3%

Outpatient attendances per head 4.68 4.76 0.08 1.7%

Accident and Emergency visits per head 0.75 0.64 -0.11 -14.7%

Bed days per head 5.68 4.87 -0.81 -14.3%

Tariff costs (£) 2,448 2,260 188 -7.7%

Challenges of prospective evaluation

Difficult to randomise a distinctive treatment and control group within the same organisations or service.

Service delivery patterns may change incrementally over time.

The client/patient group may change over time.

Randomised trials can be costly and sometimes out of proportion to the investment in the change).

Can be slow – changes need to be made embedded and cases followed up for a long time.

Results may only reflect experiences of a subset of users.

© Nuffield Trust

Alternative retropective approaches Example (1)

Impact of Marie Curie Nursing Service on place of death &

hospital use at the end of life

http://www.nuffieldtrust.org.uk/publications/marie-curie-

nursing

Chitnis, X. , Georghiou, T., Steventon, A. & Bardsley, M. J. (2013). Effect of a home-based end-of-life nursing service

on hospital use at the end of life and place of death: a study using administrative data and matched controls. BMJ

Supportive & Palliative Care, 1–9. doi:10.1136/bmjspcare-2012-000424

© Nuffield Trust

© Nuffield Trust

Evaluation: The Marie Curie Nursing Service

Intervention:

• Nursing care support to people at end of life, in their homes

Nuffield commissioned to evaluate impact:

• Are recipients more likely to die at home?

• Reduction in emergency hospital admissions at end of life?

Methods:

• Retrospective matched control study – use of already existing administrative data

© Nuffield Trust

Methods

• 29,538 people who received MCNS care from January 2009

to November 2011

• Sophisticated matching techniques used to select 29,538

individually matched controls from those who died in

England from January 2009 – November 2011

• Matched on demographic, clinical and prior hospital use

variables

• People started receiving MCNS care on average 8 days

before death

© Nuffield Trust

Matched control studies – broad aim

>1M individuals - died Jan 2009 to Nov 2011, did

not receive service

(everyone else)

Aim to find 30,000 individuals who match

almost exactly on a broad range of

characteristics

Use this group as study control group

30,000 individuals - died Jan 2009 to Nov 2011 &

received Marie Curie nursing service before death

© Nuffield Trust

1

2

3

4

5

6

7

8

9

. .

1

2

3

4

5

6

7

8

9

. .

1

2

3

4

5

6

7

8

9

. .

Marie Curie Data Linkage Service Nuffield Trust

New Identifier New Identifier New Identifier

(NHS no) (NHS no)

Names Names

Address Address

DOB DOB

HESID HESID

Marie Curie person identifiers (File A)

© Nuffield Trust

0%

10%

20%

30%

40%

50%

Comorbidities

0%

5%

10%

15%

20%

25%

30%

35%

Cancer diagnoses

Control group – how well matched? Diagnostic history

0%

10%

20%

30%

40%

50%

Comorbidities

0%

5%

10%

15%

20%

25%

30%

35%

Cancer diagnoses

Marie Curie Controls

© Nuffield Trust

Emergency admissions for cases where nursing started 8-14 days

before death

© Nuffield Trust

Impact of MCNS care on hospital costs

Table 1 – Post index date hospital costs for Marie Curie cases and matched controls

Mean (sd) hospital costs per person

Activity Type Marie Curie cases Matched controls Difference

Emergency admissions £463 (£1,758) £1,293 (£2,531) £830

Elective admissions £106 (£961) £350 (£1,736) £244

Outpatient attendances £33 (£212) £76 (£340) £43

A&E attendances £9 (£34) £31 (£60) £22

All hospital activity £610 (£2,172) £1,750 (£3,377) £1,140

• Significantly greater reduction in costs among those with no

recent history of cancer

• Also cost reduction much greater for those who started

receiving MCNS care earlier (£2,200 for those >2 weeks

before death)

Example (2)

Evaluation of Community Based Interventions impact on

hospital admissions

Retrospective evaluation using matched controls

Adam Steventon, Martin Bardsley, John Billings, Theo Georghiou and Geraint Lewis An evaluation of the impact of

community-based interventions on hospital use. A case study of eight Partnership for Older People Projects (POPP) .

Nuffield Trust March 2011

© Nuffield Trust

© Nuffield Trust

The Partnership for Older People Projects (POPPs)

Support workers for community

matrons

Intermediate care service with

generic workers

Integrated health and social care

teams

Out-of-hours and daytime response

service

+ 4 different short term assessment

and signposting services

• £60m investment by DH

with aim to: “shift resources

and culture away from

institutional and hospital-

based crisis care”

• 146 interventions piloted in

29 sites.

• National evaluation of

whole programme found

£1.20 saving in bed days

per £1 spent.

From the 146 interventions offered under

POPP, we selected 8 for an in-depth study of

hospital use

© Nuffield Trust

Prevalence of health diagnoses categories in intervention

and control groups

0%

10%

20%

30%

40%

50%

60%

Control Intervention

© Nuffield Trust

Overcoming regression to the mean using a control

group

March 2011

0.0

0.1

0.2

0.3

-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 12Nu

mb

er

of

em

erg

en

cy h

osp

ital

ad

mis

sio

ns

pe

r h

ead

pe

r m

on

th

Month

Intervention

Start of intervention

© Nuffield Trust

Overcoming regression to the mean using a control

group

March 2011

0.0

0.1

0.2

0.3

-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 12Nu

mb

er

of

em

erg

en

cy h

osp

ital

ad

mis

sio

ns

pe

r h

ead

pe

r m

on

th

Month

Intervention

Start of intervention

© Nuffield Trust

Overcoming regression to the mean using a control

group

March 2011

0.0

0.1

0.2

0.3

-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 12Nu

mb

er

of

em

erg

en

cy h

osp

ital

ad

mis

sio

ns

pe

r h

ead

pe

r m

on

th

Month

Intervention

Start of intervention

© Nuffield Trust

Overcoming regression to the mean using a control

group

March 2011

0.0

0.1

0.2

0.3

-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 12Nu

mb

er

of

em

erg

en

cy h

osp

ital

ad

mis

sio

ns

pe

r h

ead

pe

r m

on

th

Month

Control Intervention

Start of intervention

© Nuffield Trust

Impact of eight different interventions on hospital use

© Nuffield Trust

Observations on POPPs work

• Able to undertake a retrospective evaluation of changes in hospital use

for 8 projects, over 5000 subjects

• Study took less than 3 months once permissions obtained

• Findings suggest that none of these projects were delivering the

anticipated reduction in hospital use

• Similar studies failed to fined expected reductions in emegerncy

admissions

• The approach has limitations eg there is always the risk of unmeasured

confounders; end points limited by the data available.

© Nuffield Trust

Summary

• Huge scope to use existing data sets in evaluative studies of

health service innovation

• Linkage of records – across services and over time is the key to

building a better understanding of the inputs and outcomes of

care

• Retrospective matching studies offer a much more robust

approach to assessing the impacts of existing services.

• Potential to develop much better evidence about what works in

health care delivery – and through studies of the actual events of

people who used the service.

© Nuffield Trust 18 February 2015

www.nuffieldtrust.org.uk

Sign-up for our newsletter www.nuffieldtrust.org.uk/newsletter

Follow us on Twitter: Twitter.com/NuffieldTrust

© Nuffield Trust

Ian.blunt@nuffieldtrust.org.uk

Adam.steventon@nuffieldtrust.org.uk