NO CONFLICT OF INTEREST TO DECLARE

Can peer review improve quality?

David Smith

Peer review origin?

Ethics of the Physician

Ishaq bin Ali al-Rahwi (854–931) of Syria

All Quality Assurance programmes seem to rely ultimately on peer review, often in the form of an expert panel, to set the standards and the limits of acceptability.

Problems with peer reviewProblems with peer review

• Provision of Provision of – objective standardsobjective standards– structured processesstructured processes– training reviewerstraining reviewers

• Perverse incentives/perverse activitiesPerverse incentives/perverse activities• Reviewer bias Reviewer bias • Authority of the reviewersAuthority of the reviewers• Indemnity/protectionIndemnity/protection

GMC procedures to assess the performance GMC procedures to assess the performance of doctors who may be seriously deficientof doctors who may be seriously deficient

• peer review of the doctor’s practice at the workplace • tests of competence and skills.

Peer reviews are conducted by three trained assessors, two from the same speciality as the doctor being assessed, with one lay assessor and include:– the doctor’s portfolio of training, experience, practice and self

assessment – a review of the doctor’s medical records– discussion of cases selected from these records – observation of consultations – a tour of the doctor’s workplace – interviews with at least 12 third parties (five nominated by the

doctor)– structured interviews with the doctor.

Failure of peer review

Given the magnitude of the resources devoted to quality assurance and the centrality of peer assessment to these efforts, the results of this literature analysis indicate the need for a global reexamination of the peer review process.

Several of the proposals discussed herein appear to have considerable potential for improving the reliability of peer judgments.

Research directed at evaluating this potential should be part of the overall reassessment of peer review.

Goldman RL JAMA. 1992;267:958-960

Reviewer agreementReviewer agreement

physician agreement regarding quality of care is only slightly better than the level expected by chance.

Goldman RL JAMA. 1992;267:958-960

Goldman suggestionsGoldman suggestions

• Use of multiple reviewers

• More objective assessments

• Higher standards for peer reviewers

• Use of practice guidelines

• Use of outcome assessments

Watching the doctor watchersWatching the doctor watchers

The screening process also must be more accurate in order to be cost-effective, as it was only slightly better than random sampling at correctly identifying below standard care. More reproducible physician review is also needed and might be accomplished through – improved reviewer selection and training– a structured review method– more physician reviewers per record

Rubin et al JAMA 1992;267:2349-2354

Review of 339 hospitals PR systemReview of 339 hospitals PR system

• Only 24% of respondents said it was very likely to improve quality while 33% said it was unlikely

• Needs standardisation, • more governance, • bigger case load for reviews (>1% of admissions), • more personnel (already 1.1wte per 100 beds)

The Process of Peer Review in U.S. HospitalsMarc T. Edwards & Evan M. Benjamin

JCOM 2009 Vol. 16;10 461-467

Reviewer agreementReviewer agreement

• 13 board-certified physicians completed a total of 741 structured implicit record reviews of 95 records for patients who experienced severe in hospital adverse events while in the hospital (hypokalemia, hyperkalemia, renal failure, hyponatremia, and digoxin toxicity).

• They independently assessed the degree to which each adverse event was caused by medical care and the quality of the care leading up to the adverse event.

• Working in pairs, they then discussed differences of opinion, clarified factual discrepancies, and re-rated the record. The authors compared the reliability of each measure before and after discussion, and between and within pairs of reviewers.

• The reliability of the assessment of whether the complication was iatrogenic was 0.46 before discussion and 0.71 after discussion between paired reviewers, indicating considerably improved agreement between the members of a pair.

• However, across reviewer pairs, the reviewer reliability was 0.36 before discussion and 0.40 after discussion.

• Similarly, for the rating of overall quality of care, reliability of physician review went from 0.35 before discussion to 0.58 after discussion as assessed by pair. However, across pairs the reliability increased only from 0.14 to 0.17

Conclusions. When 2 physicians discuss a record that they are reviewing, it substantially improves the

agreement between those 2 physicians. However, this improvement is illusory, as discussion does not improve the overall reliability as assessed by examining the reliability between physicians who

were part of different discussions.

Hoper TP et al Medical Care 2000, 38:2 152-161

GuidelinesGuidelines

Surrogate markers of quality

May be ill founded

May be biased

Open to interpretation

Unintended consequences

Limit development

Are guidelines following guidelines?Are guidelines following guidelines?A review of 279 guidelines A review of 279 guidelines

1985-19971985-1997

Guidelines published in the peer-reviewed medical literature during the past decade do not adhere well to established methodological standards. While all areas of guideline development need improvement, greatest improvement is needed in the identification, evaluation, and synthesis of the scientific evidence.

JAMA. 1999;281(20):1900-1905

Practice guidelines developed by specialty Practice guidelines developed by specialty societies: the need for a critical appraisalsocieties: the need for a critical appraisal

431 guidelines were eligible for the study. Most did not meet the criteria

Despite improvement over time, the quality of practice guidelines developed by specialty societies is unsatisfactory.

Explicit methodological criteria for the production of guidelines shared among public agencies, scientific societies, and patients' associations need to be set up.

Lancet 2000:355;103-106

Scientific Evidence Underlying the ACC/AHA Scientific Evidence Underlying the ACC/AHA

Clinical Practice GuidelinesClinical Practice Guidelines

Recommendations issued in current ACC/AHA clinical practice guidelines are largely developed from lower levels of evidence or expert opinion.

The proportion of recommendations for which there is no conclusive evidence is also growing.

These findings highlight the need to improve the process of writing guidelines and to expand the evidence base from which clinical practice guidelines are derived

JAMA. 2009;301(8):831-841

Risk adjusted outcomesRisk adjusted outcomesfunnel plotsfunnel plots

0.00%1.00%2.00%3.00%4.00%5.00%6.00%7.00%8.00%9.00%

10.00%

0 500 1000 1500 2000 2500

Number of cases

Per

cen

tag

e ac

tual

an

d

pre

dic

ted

MA

CC

E o

utc

om

e

Actual

Predicted

LCL

UWL

UCL

LWL

BCIS ‘peer review’ and BCIS ‘peer review’ and angioplasty qualityangioplasty quality

Historical ‘no holds barred’ case review is an exemplar of peer review but to be rigorous it needs a systematic format and a written record

A prospective study of colonoscopy practice in the UK today: are we adequately prepared for national colorectal cancer screening tomorrow?

C J A Bowles

GUT 2004 53 277-283

Can peer review improve Can peer review improve quality?quality?

Increasingly robust risk adjusted outcome data

Local systematic egalitarian case review arrangements

BCIS Peer Review • What about BCIS per review• BCIS were asked to set up a PR system in 1999 and felt individual

PR was not feasible and designed a unit PR system.The first draft had many of the features that Goldman suggests but fell at the first few hurdles of reviewer acceptability, resources, lack of indemnity

• At that point it was highjacked by the department of Health in order to limit the potential proliferation of of low grade angioplasty units. It has continued in this format of a box ticking exercise for new units measured against current BCIS guidelines.

• limited as it is does it improve quality? One of the few but always contentious, surrogate measures of quality that are recorded is the number of procedures done per year.

• Yet there are some units consistently doing less than 400 for up to 7 years

Numbers of small volume units over last ten years

0

2

4

6

8

10

12

14

16

18

20

2001 2 3 4 5 6 7 8 9

<200

200-400

Report Headings

1.Introduction and outline of visit

1.Background

1.Business Case

1.Cardiac Network support

1.Relationship with other PCI Centres

1.Surgical service support

1.Ambulance service support

1.Facilities and Equipment

1.Staffing and on-call arrangements

1.Protocols for Patient Care

1.Audit and National Returns of Activity and Outcome

1.Commentary and Conclusions

Appendix – specific notes from staff interviews

Number of PCIs performed in 2006

0 20 40 60 80 100 120 140 160 180 200

East Surrey Hospital (2006)

Sunderland Royal Hospital (2006)

Wexham Park Hospital (2006)

Great Western Hospital, Wiltshire (2006)

King Edward VII (closed 2006)

Worthing Hospital (2006)

Eastbourne Hospital (restart 2006)

Craigavon Hospital (2006)

Lister Hospital (2006)

Southend Hospital (2004)

Whipps Cross (2005)

St Peter's Hospital, Chertsey (2003)

2006 data: Ludman

Centres performing < 200 procedures

PCI program start date

• The purpose of peer review is to provide an expert view of an individual’s ability to meet professionally acceptable standards and thus help ensure an acceptable level of continued professional competence. Such review might necessarily identify individuals who do not conform to acceptable standards and who may therefore be required to undergo retraining or be suspended. If the outcome of a peer review might put at risk the ability of an individual to continue to practice, then such a review has to be rigorous in the extreme. It would need to be supported by a large volume of accurate statistics and involve comparison to a detailed set of performance standards that is accepted by the speciality as a whole

• Audit

• Case review

• Direct observation of clinical activity/procedures

• 360 Appraisal