www.england.nhs.uk

Dan Wellings,

Head of Insight and

Feedback

NHS approaches to patient experience and patient reported outcome measures (PROMs)

28th April 2015

www.england.nhs.uk

Text Text

World class data and intelligence

Patient voiceand insight

Customer choiceand control

Transparency

Participation

Working on cornerstones of aworld class patient service

Knowledge

for all

2

www.england.nhs.uk

What does insight mean?

3

• Using qualitative and

quantitative data from

users to inform what we do

• Using whatever data

sources we have - not just

surveys but a whole range

of feedback techniques

• Always asking the

question, what do we do

with the data?

• Making a difference with

data

“The most important single

change in the NHS in

response to this report would

be for it to become, more

than ever before, a system

devoted to continual learning

and improvement of patient

care, top to bottom and end

to end”

A promise to learn – a

commitment to act

The Berwick Report

www.england.nhs.uk

Patient experience

data

www.england.nhs.uk

National Patient Experience Surveys

5

• Current National Surveys – feedback from c1.5 million people a year

• GP Patient survey (NHS England, twice a year)

• Community & Mental Health survey (CQC, annual)

• Inpatient survey (CQC, annual)

• A&E survey (CQC, every 3 years)

• Outpatient survey (CQC, every 3 years)

• Maternity survey (CQC, every 3 years)

• National Cancer Patient Experience survey (NHS England, annual)

• VOICES survey of Bereaved People (NHS England, annual)

• Staff survey (NHS England, annual)

• Adult Social Care survey (Department of Health, annual)

www.england.nhs.uk

Biggest challenge is measuring

experience of integrated care

6

www.england.nhs.uk

The Friends and Family Test

www.england.nhs.uk

Friends and Family Test Overview

‘How likely are you to recommend our <ward / A&E

department> to friends and family if they needed similar care

or treatment?’

1 Extremely Likely

2 Likely

3 Neither Likely nor Unlikely

4 Unlikely

5 Extremely Unlikely

6 Don’t know

Why?

www.england.nhs.uk

The Friends and Family Test Review

• Positive findings: FFT is making an impact

• 78% of Trusts reported that FFT had increased the emphasis placed

on patient experience at their Trust

• FFT is being used mainly as a service improvement tool

• Provides feedback to frontline staff, often boosting morale

• Feedback is real-time and granular: it tells staff about what

is happening in their ward

• Identifies problems and encourages action to be taken

• Not used widely for performance management

• Concerns over robustness of data, including comparability

between trusts

www.england.nhs.uk

FFT is not a survey

• FFT is a real-time local feedback tool

• It has different characteristics from national surveys

FFT

• Real-time

• Ward-level

• Effective for service improvement

• Can be used as early-warning system

• Not representative, not comparable

National surveys

• Robust, comparable data at Trust level

• Data on a range of issues

• Suitable for performance management

• Not meaningful to front-line staff

• Has not historically changed behaviours

• FFT and National surveys are complementary sources of Insight

• FFT is a formative measure: it provides data to improve services

• National surveys are summative measures: they provide an accurate picture

of relative performance

www.england.nhs.uk

FFT has been seen through the wrong lens

www.england.nhs.uk

A measure and an intervention –

6.5 million responses so far

www.england.nhs.uk

The PROMs programme

www.england.nhs.uk

(Berwick 1997)

“The ultimate measure by which to judge the quality of medical effort is whether it helps patients (and their families) as they see it. Anything done in health care that does not

help a patient or family is, by definition, waste, whether or not the professions and their associations traditionally hallow it”

www.england.nhs.uk

(Appleby and Devlin 2010)

“The first health system

internationally to measure what

it produces in terms of health.”

www.england.nhs.uk

• 4 national PROMs for

surgical conditions

introduced in 2009

• Hip replacement

• Knee replacement

• Groin hernia

• Varicose vein

• 180,000 pre-op

questionnaires per year

• Cost to trusts annually ~

based on approx

£3- £5 / patient

• Cost to develop the

process, tools, pilot and

establish collections, to

date ~ approx £7-8 million

The established PROMs programme - scope

www.england.nhs.uk

• Original objectives

• Outcome comparison

between NHS and

independent sector provider;

• Inform patient choice;

• Performance management

of organisations and

clinicians;

• Facilitate service

improvement by clinicians

and managers

• Patient centred care

• Evaluation of practices and

policies

• Established uses

• Outcomes Framework

indicator 3.1

• CQC Intelligence Monitoring

• Hip/ Knee best practice tariff

from April 2014

• CCG Data Packs

• Using PROMs in “Beyond

Compliance” approvals of

new prostheses to market

The established PROMs programme - aims

www.england.nhs.uk

Q1

• Pre-operative questionnaire

• Condition-specific + generic instruments

Q2

• Post-operative questionnaire (3 or 6 months)

• Repeats the same instruments

Q2-Q1

• Difference between scores in Q2 and Q1 = health improvement/gain

• Case-mix adjustment

• Outlier identification

The established PROMs programme - process

www.england.nhs.uk

Patients report big health improvements• Patients fill in a pre-op questionnaire at a consultation prior to

surgery and a post-op questionnaire about six months after surgery.

• The difference in Oxford Hip (Knee) scores between the pre-operative and post-operative scores can be taken as an indication of the health impact of the surgery - generally very positive.

www.england.nhs.uk

There is variation in outcomes, with some

trusts significantly above the England

average, even after we have taken into

account the case-mix

0

5

10

15

20

25

30

0 100 200 300 400 500 600 700 800 900

Ad

jus

ted

Healt

h G

ain

Number of Modelled Records

95% control limits

99.8% controllimits

England

All

Positive Outliers are identified above the

upper 99.8% Confidence Limit

21

www.england.nhs.uk

There is variation in outcomes, with some

trusts significantly below the England

average, even after we have taken into

account the case-mix

0

5

10

15

20

25

30

0 100 200 300 400 500 600 700 800 900

Ad

jus

ted

Healt

h G

ain

Number of Modelled Records

95% control limits

99.8% controllimits

England

All

Negative Outliers are identified below the

lower 99.8% Confidence Limit

For Hips and Knees since April 2014 this

may mean not receiving Best Practice Tariff

22

www.england.nhs.uk

2010 Health White Paper

“Providers will be paid according to their performance. Payment should reflect outcomes, not just activity, and provide an incentive for better quality.”

www.england.nhs.uk

Best Practice Tariff

• Introduced in 2014/15 for primary hip and knee replacement

• Level of reimbursement conditional on patient outcomes

• Only providers meeting a number of conditions receive full

payments.

• Providers who have significantly poorer outcomes receive an

about 10% smaller payment per procedure.

www.england.nhs.uk

Best Practice Tariff (knee replacement

with complications and comorbidities)

5,300

5,400

5,500

5,600

5,700

5,800

5,900

6,000

6,100

6,200

6,300

6,400

(£)

£633 Incentive

Standard

tariffBPT

www.england.nhs.uk

“Little is known about the impact of PROMs.”

N. Black, BMJ 2013

www.england.nhs.uk

Outcomes have improved since 2009/10

Knee outcomes*

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Excellent results(hip)

Poor results(hip)

Hip outcomes*

* Oxford Hip/Knee Scores: Poor result: < 27

Excellent result > 41

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Poor results(knee)

Excellentresults (knee)

27

www.england.nhs.uk

• Anecdotal evidence of use

of PROMs for service

improvement: • “Northumbria Hospitals have

changed the type of

prosthesis used after

analysis of the data from

PROMs and the National Joint

Registry showed differential

performance, on the basis of

patient scores on

complications, pain levels,

Oxford hip/knee scores, and

EQ-5D scores”

• Some anecdotal evidence

on usage with patients

• For choice

• For shared decision

making

• Some anecdotal evidence

on how CCGs are using

data for commissioning

Evidence of Impact

www.england.nhs.uk

Challenges of using PROMs

• The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research.

• Boyce, Browne, Greenhalgh. BMJ Quality and Safety (2014)

• •Practical problems (technical support, workload)

• •Negative attitudes (suspicion of managerial objectives, concerns about impact on patient-clinician relationship)

• •Methodological concerns (validity, interpretability)

• •Doubts about impact (cannot turn the data into practical solutions).

www.england.nhs.uk

www.england.nhs.uk

• Develop PROMs tools for shared use by patients and

clinicians

• Support patients in managing their condition and

allow clinicians to monitor their progress remotely

• Personalised goals can ensure that what matters to

the patient is being measured, but also that the care

designed by the clinician reflects these outcomes

Making patients partners in their

care – PCOMs?

www.england.nhs.uk

NOW

• Make better use of existing

PROMs data to drive

change, raising awareness

and improving

understanding of what the

data tells us

• Consult on existing

PROMs

• Explore new methods of

collecting PROMs data

including electronic

solutions

FUTURE

• Pilot PROMs in new

clinical areas, where the

evidence is poor

• Develop methods where

patients become partners

in their own outcome data

• Exploring how PROMs

data can be included in

national clinical audits

• Exploring how outcomes

can affect tariff

PROMs and PCOMs – a policy area still in its

relative infancy

www.england.nhs.uk

The Insight Strategy

Insight &

Feedback

Less descriptive Le

ss g

ener

alis

able

Mo

re g

ener

alis

able

More descriptive

In depth interviewsExperience Based Co-Design

Focus groups& patient forumsNHS TrustsCCGsHealthwatchPatient organisations

SurveysGP Patient SurveyBereaved VoicesCancer Patient Experience SurveyNHS Staff SurveyPROMs

Online reviews & ratingsNHS ChoicesPatient OpinionIWantGreatCareCare Connect

Patient storiesHealthTalkOnlinePatient Voices

Feedback tools (kiosks, SMS, apps)Examples include:HospediaSynaptaFeedback apps

Social medialisteningSalesforcePolecatHealthberry

Public MeetingsNHS TrustsCCGs

Matrix of insight methods from Measuring patient experience: evidence scan Health Foundation, June 2013

Friends & Family Test

Complaints

www.england.nhs.uk

Objective of Insight Strategy

• To establish a clear strategic vision for how

the different elements of insight & feedback

can complement each other most effectively,

to be greater than the sum of their individual

parts in giving us a clear picture about what

patients and the public think of their NHS

services

www.england.nhs.uk

Overall themes

www.england.nhs.uk

Designing the appropriate methodology

for the intended purpose

37

• Performance management

• Service improvement

• Patient involvement

• Service re-design

• Commissioning services

• Patient Choice

• Pay for performance

• Regulation

www.england.nhs.uk

Methodologies – pros and cons

RobustnessUsability

www.england.nhs.uk

Balance between local and national

collections - ownership

39

www.england.nhs.uk

Is data being presented back

appropriately to each audience?

40

Patients and the

public Frontline staff

Managers and

commissioners Academics

www.england.nhs.uk

Conclusions and a few thoughts on

what we might all look to do

41

• Be very clear on what data is collected for and do not overload

a collection

• Think early on about how data will be translated into change

• Think clearly about audience(s)

• Involve patients as much as possible in the design

• Train people to use the data and show how change can be

made

• Can we reward appropriate usage of insight?

• Numbers without meaning can be dangerous – signal not noise

• Reflect on ownership - engagement

www.england.nhs.uk

Means nothing if nothing is done