PROMs in secondary mental health services in

the UK: a feasibility study and lessons learnt.

Anju Keetharuth, John Brazier, Elizabeth Gibbons, Janice Connell, Thomas Ricketts, Mahmood Khan, Michael Barkham

09/ 2014 © The University of Sheffield

Acknowledgements

• All services users, clinicians and healthcare staff at both sites

• Staff at both sites that were involved in the study

• We acknowledge the support of the NIHR Clinical Research Network.

FundingDepartment of Health through the Policy Research Unit in Economic Evaluation of Health and Care Interventions.

Views expressed are those of authors.

Aim• To assess the feasibility of PROMs with this

group of service users (all-age adults)

Objectives • To assess acceptability, usefulness and

practicality (MoA)of routine collection of PROMs (for service users, clinical staff, managers and commissioners)

• Identify barriers and how to minimise them

1. Response 2. Completion 3. Quality of data

Methods• Roll-out stage across two sites

• Each participant completes a condition specific measure (PHQ-9 or CORE-10) and EQ-5D

• Mode of administration: Paper and pen ; tablets

• Data analysis – statistical

• Qualitative interviews before and after roll-out

• Focus groups and individual interviews with clinicians, service users and other healthcare professionals

• Interviews at beginning to help with development of the study

• Interviews at end to understand the barriers and enabling factors

• Data analysis: Framework analysis

28/11/2017 © The University of Sheffield

RESULTS

Qualitative interviews at the beginning – Development stage

Service users n = 16 (2 focus groups ; 2 individual interviews) Clinicians n = 34 (4 focus groups)

• In general, service users more positive than clinicians

1. Timing of questionnaires: Engagement issues at baseline; follow-up questionnaire

2. Tablets: anxiety, dexterity and confidence

3. What will the data be used for?

4. Practical considerations – presentation, the measures

28/11/2017 © The University of Sheffield

Demographics

Both sites N (%)

NationalAverage a %

Gender 258

Male 118 (46) 49

Female 140 (54) 51

Ethnicity

White 205 (79) 85

Asian/Asian Brit 21 (8) 8

Mean SD Min Max

Age 45 17 18 93

IMD 33.75 19.12 2.27 75.23

a Source: Census 2011 Across sites: • no gender difference; no IMD difference • Ethnicity – different (p <0.01); Age different

EQ-5D

28/11/2017 © The University of Sheffield

02

46

810

Per

cent

-.5 0 .5 1EQ-5D score at T1 using values on EuroQol website

N Mean SD Range

EQ-5D score at T1 241 0.425 0.318 -0.37 to 1

EQ-5D score at T2 38 0.468 0.246 0.036 to 0.812

Gen pop norm for a 45 year old (HSE)

0.877 - -

CMH condition (APMS)(Roberts et al. 2014)

2,126 0.658 0.273 -0.38 to 1

No statistically significant differences across sites for EQ-5D, PHQ-9 and CORE-10

PHQ-9

28/11/2017 © The University of Sheffield

0 – 4 None5 – 9 Mild10 – 14 Moderate15 – 19 Moderately Severe20 – 27 Severe

The recommended clinical cut-off is 10

N PHQ-9 mean SD Range

PHQ-9 scores at T1 198 18.6 8 0 to 27

PHQ-9 scores at T2 20 13 8 0 to 27

NAPT 2011 32,332 14.1 4.8 -

IAPT (Parry et al ) 500 15.7 7.1 -

• 5

05

10

15

20

25

Pe

rcen

t

0 5 10 15 20 25Score for PHQ-9 at T1

Ceiling effect

CORE-10

28/11/2017 © The University of Sheffield

N Mean SD Range

CORE-10 score at T1 59 23 6.8 3 to 35

CORE-10 score at T2 18 20 8 8 to 34

General population* 553 5 4.8 -

Primary care counselling* 1,835 20 7.7 -

010

2030

Per

cent

0 10 20 30 40Score for CORE-10 at T1

0 –

5 Healthy5 – 10 Low Level10 – 15 Mild15 – 20 Moderate20 – 25 Moderate

Severe25 + Severe

The recommended clinical cut-off is 10

* CORE-10 Manual 2007

Feasible YES/NO?

1. Response rate – difficulty of identifying a denominator (clinician’s case load, cannot determine how many were asked and declined)

2. Completion rate

Usable rate EQ-5D: 93%, PHQ-9: 99%,

CORE-10: 100%

Feasible YES/NO?

3. Quality of the data

• Internal consistency (the degree to which the items are measuring the same things)

Cronbach alpha PHQ-9: 0.91 (excellent)

CORE-10: 0.76 (good)

• Known group difference

Difference is statistically significant – those more severe condition, lower QoL (EQ-5D and PHQ-9)

Conclusions so far

• Results need to be interpreted with caution due to low n

• Is it feasible to collect PROMs in secondary care? MAYBE

• The scores make sense

• Are service users getting better ?

• Getting T2 data has proved challenging

• The challenges not to be underestimated

28/11/2017 © The University of Sheffield

Qualitative interviews at the end

• Telephone interviews, in person individual interviews and focus groups across both sites

• Healthcare staff n = 23

• Psychiatrists, CPN, quality managers, administrators, research co-ordinators

• Service users n = 14

28/11/2017 © The University of Sheffield

Barriers

• Consenting

• Perception that service-users lack capacity to consent

• Lack of engagement in research

• Uncertainty of diagnosis

• Workload

• Electronic data capture (IPADS/EPADS)

• Lack of understanding about outcome measures

• No dedicated outcomes systems

• Measuring outcomes – not part of the culture

Relevance of the PROMs- content• Too simplistic - service-users complexity

CORE and PHQ-9• Service-users concerns related to ‘Self-harm’

items

• Healthcare professionals views of suicide items

• EQ-5D• Generic- not relevant versus identify other issues

• Impact of responses/fear/honesty/gaming

ReQoLRecovering Quality of Life

New measure – currently being developed www.reqol.org.uk

Uses of PROMs

• Individual patient monitoring

• Healthcare professionals-baseline useful/impact on consultation time

• Service-users-replace personalised care/self monitoring

• Treatment decisions- impact of specific interventions

• Evaluating services- Team monitoring

• National approaches to PROMs data collection

• Used to determine funding allocation?

Lessons learnt for future implementation

• MH – different kettle of fish compared to elective surgery

• Timings- dependant on how frequently services are accessed

• Administration- where, when, who with

• Presentation of questionnaires- lack of engagement when unwell

• Presentation of results

• Systems/resources/logistics (T2 data)

• Mandatory

28/11/2017 © The University of Sheffield

References• CORE-10 User Manual (version 1.0) 2007.

• Roberts J., Lenton P. Keetharuth A.D, Brazier J. (2014). Quality of Life impact of mental health conditions in England: results from the adult psychiatric morbidity surveys. Health Quality of Life Outcomes. 12(6)

• Keetharuth A.D, Whyte S. (2011) Norms to calculate QALY loss: data from Health Survey for England. Unpublished.

• Parry, G., Barkham, M., Brazier, J., Dent-Brown, K., Hardy, G., Kendrick, T., ... & Lovell, K. (2011). An evaluation of a new service model: Improving Access to Psychological Therapies demonstration sites 2006-2009. Final report. NIHR Service Delivery and Organisation programme.

28/11/2017 © The University of Sheffield