Designing Risk Managed QC Strategies...• If we double our QC Frequency, which is not a bad idea...

Designing Risk Managed QC Strategies John Yundt-Pacheco, MSCS Senior, Principal Scientist Quality Systems Division, Bio-Rad Laboratories

Learning Objectives

1. Understand how test method performance, reliability and clinical utility affect our tolerance for erroneous results.

2. Learn how to determine an acceptable probability of patient harm.

3. Know how to compute a predicted probability of patient harm for a test method.

4. Understand how to compute a risk management index, comparing predicted probability of patient harm with the acceptable probability of patient harm.

Agenda

Introduction to Risk Managed QC Design

Determining an Acceptable Probability of Patient Harm

Computing the Predicted Probability of Patient Harm

Computing a Risk Management Index

Risk Managed QC Design Examples

The Implications of Risk Managed QC Design

Conclusions

1

2

3

4

5

6

7

Introduction to Risk Managed QC Design

1

Some level of patient harm is inevitable…

The goal of risk management in the clinical laboratory is to manage the risk of patient harm due to erroneous

results, keeping it at an acceptable level.

Introduction to Risk Managed QC Design 1

Risk Managed QC Strategy Design

1. Determine an acceptable level of patient harm.

2. Compute the predicted probability of producing patient harm.

3. Find a QC strategy that has predicted probability of patient harm less than the acceptable level of patient harm.


The ISO model for acceptable risk

The ISO 14971:2012 Medical Devices – application of risk management to medical devices presents a model for risk management and determining acceptable levels of risk. Steps to use the ISO risk model:

1. Assign a Severity of Harm category to an analyte 2. Use the risk acceptability matrix to determine the

acceptable probability of patient harm.



2

2

Severity of Harm Categories

Severity of harm is described in terms of the consequence to the patient Both ISO 14971 and CLSI EP23 give the same example severity of harm categories

Negligible • Inconvenience or temporary discomfort

Minor • Temporary injury or impairment not requiring

professional medical intervention

Serious • Injury or impairment requiring professional

medical intervention

Critical • Permanent impairment or life-threatening injury

• Patient death Catastrophic


ISO 14971 - Risk Acceptability Matrix

Probability of Harm Negligible Minor Serious Critical Catastrophic

Frequent Unacceptable Unacceptable Unacceptable Unacceptable Unacceptable

Probable Unacceptable Unacceptable Unacceptable Unacceptable Unacceptable

Occasional Acceptable Acceptable Unacceptable Unacceptable Unacceptable

Remote Acceptable Acceptable Acceptable Unacceptable Unacceptable

Improbable Acceptable Acceptable Acceptable Acceptable Acceptable

Severity of Harm

2 Determining an Acceptable Probability of Patient Harm

Probability of Harm Categories

Category Level

CLSI EP23Example

ISO 14971Example

Frequent Once/week ≥1/1,000

Probable Once/month <1/1,000 and ≥1/10,000

Occasional Once/year <1/10,000 and ≥1/100,000

Remote Once/few years <1/100,000 and ≥1/1,000,000

Improbable Once/life of measuring system <1/1,000,000


EP23: A Risk Acceptability Matrix


Frequent Unacceptable Unacceptable Unacceptable Unacceptable Unacceptable

Probable Acceptable Unacceptable Unacceptable Unacceptable Unacceptable

Occasional Acceptable Acceptable Acceptable Unacceptable Unacceptable

Remote Acceptable Acceptable Acceptable Acceptable Unacceptable


Severity of Harm

The Occasional Probability of Harm has an upper limit of 1/10,000


A Practical Risk Acceptability Matrix

Negligible Minor Serious Critical Catastrophic

Acceptable Unacceptable Unacceptable Unacceptable Unacceptable

Acceptable Acceptable Unacceptable Unacceptable Unacceptable

Acceptable Acceptable Acceptable Unacceptable Unacceptable

Acceptable Acceptable Acceptable Acceptable Unacceptable

Acceptable Acceptable Acceptable Acceptable Acceptable

Severity of Harm


Practical Probability of Harm Categories

Category Level Practical ISO 14971

Frequent <1/100 and ≥1/1,000 ≥1/1,000

Probable <1/1,000 and ≥1/10,000 <1/1,000 and ≥1/10,000

Occasional <1/10,000 and ≥1/100,000 <1/10,000 and ≥1/100,000

Remote <1/100,000 and ≥1/1,000,000 <1/100,000 and ≥1/1,000,000

Improbable <1/1,000,000 <1/1,000,000


1/10,000 is the upper limit for Occasional Probability of Harm

Category Level Practical ISO 14971

Frequent <1/100 and ≥1/1,000 ≥1/1,000

Probable <1/1,000 and ≥1/10,000 <1/1,000 and ≥1/10,000

Occasional <1/10,000 and ≥1/100,000 <1/10,000 and ≥1/100,000

Remote <1/100,000 and ≥1/1,000,000 <1/100,000 and ≥1/1,000,000

Improbable <1/1,000,000 <1/1,000,000



3


We need 2 things: 1. Our rate of producing erroneous results

• estimated as the ratio of erroneous results to the specimens tested

2. The probability that an erroneous result will result

in patient harm • This is a conditional probability – the probability of

harm given an erroneous result: Ph|u

3 Computing the Predicted Probability of Patient Harm

Erroneous results are produced when we are In Control and when we are Out of Control. The probability of producing an erroneous result – or the rate is computed by:

PU = PUic + PUoc Where: • PU = probability of producing an erroneous result • PUic = probability of producing an erroneous result

while In Control. • PUoc = probability of producing an erroneous result while

Out of Control

Computing the Rate of Producing Erroneous Results


In the technical literature, PUic is referred to as:

PUic = PE(0)

• Our test methods have imprecision so there is a probability of reporting incorrect results when the test system is in control.

• PE(0) depends on the performance of the in-control test method relative to the quality specification for the test method.

• PE(0) is computed as a probability.

Producing Erroneous Results In-Control


The probability of producing an erroneous result when Out of Control is a function of the size of

the error condition and is computed by:

PUoc(SE) =NU Out of Control SE

N In Control + N Out of Control SE

Producing Erroneous Results Out of Control


Computing the Predicted Probability of Harm

Going back to our earlier equation and substituting, we get:

PH SE = {PE(0) +E(Nuf(SE)

MPBF + ANPed(SE)} ∗ Ph|u


Computing a Risk Management Index

4

We define the patient risk management index as:

RMI =

Predicted PHAcceptable PH

• RMI ≤ 1 implies acceptable risk. • RMI values permit easy assessment and comparison of

multiple analytes • with different frequencies of test system failure • with different probabilities of harm given an incorrect result • with different severities of patient harm

Computing a Risk Management Index (RMI)

Computing a Risk Management Index 4

Risk Managed QC Design Examples 1

5

Example 1

Cholesterol procedure: • CLIA Quality Specification: 10% • 600 patients per day • 2 QC Events with 2 levels per day

– QC Means: 105.4, 257.0 mg/dL – QC SDs: 2.5, 4.75 – 1:3s QC Rule

• Severity of Harm Category: Minor • Mean Time Between Failures is 2 days

5 Risk Managed QC Design Examples 1

Example 1

• Severity of Harm Category: Minor – Maps to Probable Probability of Harm or 1/1,000

• Mean Time Between Failures is 2 days – 600 patients per day – Mean Number of Patients Between Failures

• 2 * 600 = 1,200



Frequent Acceptable Unacceptable Unacceptable Unacceptable Unacceptable

Probable Acceptable Acceptable Unacceptable Unacceptable Unacceptable




Severity of Harm

Example 1: RMI 0.63


Example 1: False Rejection Rate

• This QC Strategy has a False Rejection Rate (FRR) of 0.010353 per QC Event.

• The Expected Time between False Rejections (ETFR) can be computed as:

1/(FRR * Number of QC Events per day) • ETFR = 1/(0.010353 * 2) ~ 48 days. • So we try a 1:3.5s rule


Example 1: RMI of 1.37 with 1:3.5s QC rule


Example 1: QC Frequency

• Risk Managed QC Design allows us to use more QC to lower our False Rejection Rate.

• If we double our QC Frequency, which is not a bad idea with frequent failures, we will go from 300 patients to 150 patients between QC events.

• This will allow us to use a 1:3.5s rule


Up Frequency Down Rejection Rate

Example 1: RMI of 0.89, 1:3.5s QC Rule


Example 1: Final Design

• This QC Strategy has a False Rejection Rate (FRR) of 0.002246 per QC Event.

• The Expected Time between False Rejections (ETFR) can be computed as:

1/(FRR * Number of QC Events per day)

• ETFR = 1/(0.002246 * 4) ~ 111 days.


Risk Managed QC Design Example 2

5

Example 2

Albumin procedure: • Biological Variation Minimum Quality Specification: 5.79% • 250 patients per day • 2 QC Events with 2 levels per day

– QC Means: 2.70, 4.50 g/dL – QC SD’s: 0.07, 0.10 – Repeat 1:2s QC Rule

• Severity of Harm Category: Negligible • Mean Time Between Failures is 15 days


Example 2

• Severity of Harm Category: Negligible – Maps to Frequent Probability of Harm or 1/100


• 250 * 15 = 3750


Example 2: RMI of 2.12 with Ph|u at 1


Example 2: RMI of 2.12 with Ph|u at 1

The Red Line is on RMI of 1

Baseline test performance above RMI 1

More QC is not going to help.


Example 2

• In this case, our in-control probability of producing an erroneous result (PE(0)) is too high.

• Review how results outside the quality specification are being handled clinically.

• Consider how likely it is that an albumin result with an error of 5.8% will cause patient harm.


Harm

Example 2

• The probability of harm given an erroneous albumin result is set to 0.2.

• Ph|u for albumin = 0.2


Harm

Example 2: RMI of 0.42 after Ph|u set to 0.2


Risk Managed QC Design Example 3

5

Example 3

Sodium procedure: • CLIA Quality Specification: 4 mmol/L • 600 patients per day • 2 QC Events with 2 levels per day

– QC Means: 121.4, 153.8 mmol/L – QC SD’s: 1, 1.1 – Repeat 1:2s QC Rule

• Severity of Harm Category: Serious • Ph|u = 0.2 • Mean Time Between Failures is 3 days


Example 3

• Severity of Harm Category: Serious – Maps to Occasional Probability of Harm or 1/10,000


• 600 * 3 = 1800



Frequent Acceptable Unacceptable Unacceptable Unacceptable Unacceptable

Probable Acceptable Acceptable Unacceptable Unacceptable Unacceptable




Severity of Harm

Example 3: RMI of 1.43 on Initial Design


Example 3: RMI is due to Out of Control

Need more

QC


Example 3: Reduce patient samples between QCs

• We need to add 2 additional QC events to bring the number of patients down from 300 to 150 between QCs.

• Introducing additional electrolyte control events is often necessary as electrolytes are ordered frequently and usually have tight control parameters


QC QC Patient

Samples

Example 3: RMI of 0.97 after increasing QC Frequency


The Implications of Risk Managed QC Design

6

The Impact of Risk Managed QC Design

Risk Managed QC Strategy Design allows you to: – Use the clinical utility of the test method to assess

the acceptable level of patient harm – Estimate the probability of producing erroneous test

results – Estimate the predicted probability of patient harm

from erroneous patient results – Compare the predicted probability of patient harm to

the acceptable level of patient harm in a Risk Management Index

6 The Implications of Risk Managed QC Design

The Impact of Risk Managed QC Design

The estimates for the probability of producing erroneous patient results consider:

– the quality specification – the quality control strategy – test method performance – test method reliability – patient volume

Unlike conventional QC Design, Risk Managed QC Design allows you to reduce the false rejection rate with additional QC.

6 The Implications of Risk Managed QC Design

Conclusion

7

7

Summary

1. The clinical utility of a test method determines our tolerance for producing incorrect results and can be used to determine an acceptable level of patient harm.

2. Our QC strategy, in conjunction with test method performance and reliability, can be used to estimate the probability of producing erroneous results and predict the probability of patient harm from erroneous results for a test method.

3. We can compute a Risk Management Index (RMI) as a ratio of the predicted probability of patient harm to the acceptable probability of patient harm.

4. We can use the RMI to find suitable QC strategies.

Conclusion

For more information

See Dr. Curtis Parvin’s webinar “Linking QC Performance to Patient Risk” at: http://qcnet.com/missioncontrol/in-depth.html

7 Conclusion

Designing Risk Managed QC Strategies...• If we double our QC Frequency, which is not a bad idea...

Documents

Transcript of Designing Risk Managed QC Strategies...• If we double our QC Frequency, which is not a bad idea...