The State of the Evidence The State of the Evidence in Patient Safety in Patient Safety

Kaveh G. Shojania, MDCanada Research Chair in Patient Safety

and Quality ImprovementDepartment of Medicine

Sunnybrook HospitalUniversity of Toronto

• Concise, evidence-based reviews of over 75 specific patient safety practices

• Over 140,000 copies obtained since publication in 2001

Peri-operative beta-blockers to reduce cardiac complications of non-cardiac surgery

• 5 randomized trials at time of AHRQ report (2001)– Total patients ~ 600 across all 5 trials

– Substantial benefit: 1 major event averted for every 4-8 patients treated

Received 2nd highest evidence rating in AHRQ report

• Meta-analysis of 21 trials subsequently showed questionable benefit and increases in harm (2005)

• Recent mega-trial showed increased total mortality (Devereaux et al. Lancet 2008)

Shojania et al. Making Healthcare Safer. 2001.

Other Examples of Major Changes or Starkly Conflicting Evidence

• Universal MRSA screening

• Hip protectors to prevent fall-related injuries

• Supplemental oxygen to decrease postoperative infections

• Acetylcysteine to prevent contrast-nephropathy

• Rapid response teams

• ‘Survival analysis’ of 100 meta-analyses reviewed in ACP J Club

• Major qualitative or quantitative changes in evidence occurred for 23% within 2 years and for 15% within 1 year

–7% were already out of date at time of publication

• Multifaceted interventions (13 studies) showed a borderline significant reduction in falls but not fractures

• No other strategy (hip protectors, removal of physical restraints, fall alarm devices, changes in the physical environment, medication review in hospital) showed consistent, significant effects on falls, fallers, or fractures

“Safety of Patients Isolated for Infection Control” (Stelfox et al. Jama 2003)

Isolated patients twice as likely to experience adverse events (31 vs 15 adverse events per 1000 days; P<.001). Included significant difference in preventable events (20 vs 3 adverse events per 1000 days; P<.001)

Isolated patients also more likely to• have no vital signs recorded as ordered (51% vs 31%; P<.001)• have days with no physician progress note (26% vs 13%; P<.001)

Catheter-related bloodstream infections showed a significant decrease from 0.62 (95% CI: 0.47 to 0.81) at baseline to 0.34 (95% CI, 0.23 to 0.50) at 18 months.• missing data for approx 40% of ICU months• CRBSIs definition open to bias • no corroborating blood culture results

Main Arguments Against Needing Evidence in Patient Safety

• Too challenging: interventions are too complex to study using conventional EBM paradigm

• Some interventions are self-evident

Villages…were pair-matched and randomly allocated to receive the intervention at study onset… or 3 years later… Loans were provided to poor women who enrolled in the intervention group. A participatory learning and action curriculum was integrated into loan meetings, which took place every 2 weeks…

Cluster RCTs in Patient Safety/QI

• Rapid response teams – Hillman et al. Lancet 2005

• Teamwork training– Nielsen et al. Obstet Gynecol. 2007

• Computerized decision support– Eccles et al. BMJ. 2002

• Feedback of acute MI performance data to hospitals– Beck et al. Jama 2005

• PDSA and Chronic Care Model– O’Connor et al. Diabetes Care. 2005

Without these trials, we would waste huge amounts of money on

ineffective (versions of these) interventions

“Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials.”

Lethal errors involving injections of concentrated KCl

Remove concentrated potassium from clinical areas“forcing function” that prevents errors from happening

Concentrated KCl resembles other iv

solutions

A simple, obviously beneficial patient safety intervention

Sobering example of hospital where delays in receiving KCl from Pharmacy resulted in surreptitious hording of KCl on wards Increase, rather than decrease in hazard

State of the Evidence in Patient Safety

• Superficial knowledge about epidemiology of many important safety problems and their causes

• Few established, highly effective interventions – most things either don’t work or we don’t know if they

work

• Those interventions that do work are often costly, complex, and may even create new problems

Evidence vs. Action

• Robust evidence directly informs a minority of clinical situations.

often treat patients on basis of anecdotal experience

– But we don’t turn these practices into major recommendations

• In patient safety, robust evidence lacking for almost everything

individual hospitals proceed with promising strategies

– But widespread dissemination will require rigorous evaluation to confirm benefit in wide range of settings

Auerbach et al. NEJM. 2007

Suggested Framework for EvaluationBenefit self-evident?

Y N

RCT feasible?Monitor introduction

Y N

(cluster) RCT

Controlled Before-After Study or

Interrupted Time Series

Multivariate modelling

Y N

KCl removal

RRTs, Teamwork

P-4-PWork hour

reductions

High volume providers,

Staffing ratios

Prospective evaluation

Modified from R.J. Lilford

Conclusions• Evidence in patient safety still fairly sparse

– but about what one would expect for a field this young

• Scant attention to implementation issues, unintended consequences, and costs

• Need to distinguish levels of evidence to proceed locally with a given intervention and that required for dissemination

Precisely because of complexity and contextual factors, we need large-scale evaluations to determine what intervention components required to achieve benefit in which settings