Troubleshooting Failed QC

Dr Douglas ChesherClinical Biochemistry, NSW Health Pathology, Royal

North Shore HospitalNorthern Clinical, School University of Sydney

Outline

• Types of errors• Approach to troubleshooting• Problem fixed, now what?• Lot number changes

Experimental Errors

• Systematic Error• Random Error• Blunders

Systematic Errors

• Due to identified causes and can, in principle, beeliminated. Errors of this type result in measuredvalues that are consistently too high orconsistently too low.

• Types– Instrumental– Observational. For example, parallax in reading a

meter scale.– Environmental– Theoretical. Due to simplification of the model system

or approximations in the equations describing it.

http://www.physics.nmsu.edu/research/lab110g/html/ERRORS.html

Random Errors

• Random errors are positive and negativefluctuations that cause about one-half of themeasurements to be too high and one-half to betoo low.– Sources of random errors cannot always be identified.

• Types of random error– Observational. For example, errors in judgment of an

observer when reading the scale of a measuringdevice to the smallest division.

– Environmental. For example, unpredictablefluctuations in line voltage, temperature, ormechanical vibrations of equipment.

http://www.physics.nmsu.edu/research/lab110g/html/ERRORS.html

Blunders

• An outright mistake.• Should stick out like sore thumbs if we

make multiple measurements or if oneperson checks the work of another.

http://www.physics.nmsu.edu/research/lab110g/html/ERRORS.html

“My QC Failed”

Response to a failed QC

• Stop• Do not just rerun the QC

– Unless part of your rule (eg. 1 2s rerun ‘once’)– Regression towards the mean

Response to failed QC

• Do not just recalibrate as a routine.– Calibrations add noise and increase

imprecision.

120

125

130

135

140

145

150

1 3 5 7 9 11131517192123252729313335373941434547495153555759

126128130132134136138140142144146148

1 3 5 7 9 11131517192123252729313335373941434547495153555759

Points of failure• Instrument Failure:

– Check for error messages (printed or displayed). Refer to thetroubleshooting section of the instrument manual.

• Reagent, Quality Control Materials, Calibrators:– In correct reconstitution, i.e. wrong or contaminated diluent or

use of the wrong volume of diluent.– In correct storage, e.g. left at room temperature for excessive

amount of time.– Prepare fresh (allow enough time to dissolve)– NOTE: Bulk liquid reagents may be contaminated.

• Human:– Reread the method - Check whether the correct sample volumes

or reagents were used. Make sure that a step in the procedurewas not missed.

– Were the correct parameters entered?

Have other tests also failed?

• Blunders– Swapped QC levels

• Short sampled

Are there any instrument flags?

What rule has failed?

• Systematic error– 1 3s, 2 2s, 4 1s, 10x etc

• Random error– 1 3s, R4s

Review your QC Chart

142

143

144

145

146

147

148

Serum X

-3SD

-2SD

-1SD

Mean

1SD

2SD

3SD

Systematicerror

Systematicerror

Has anything changed?

• Just calibrated• New bottle of reagent• New lot of reagent

New reagent lotNew reagent lotQuarterlymaintenance

Quarterlymaintenance

Review points of failure

• Reagents– Correct reagent?, sufficient volume, shelf expiry

date, on-board expiry date• Quality Control Material

– Material, lot no. & assigned value, preparation,shelf expiry date, in-use expiry date, storageconditions.

• Calibrators– Correct material, lot no. & assigned values,

preparation, storage, expiry date. Inspectcalibration trace of last calibration.

Review points of failure

• Instrument– Is maintenance up to date?– Recheck flags, probes, lamps, cuvettes, water

bath.

Correctiveaction

requiredimmediately

?

Correctiveaction

requiredimmediately

?

Instrumentflags?

Instrumentflags?

Maintenance up todate?

Maintenance up todate?

ReagentsOK?

ReagentsOK?

QC OKQC OK

CalibratorsOK?

CalibratorsOK?

QC FailureQC Failure

InstrumentOK?

InstrumentOK?

Monitor QC performanceand correct non-urgentfaults as appropriate

Monitor QC performanceand correct non-urgentfaults as appropriate

Correct instrument faultCorrect instrument fault

Perform all outstandingmaintenance

Perform all outstandingmaintenance

Correct reagent faultCorrect reagent fault

Make up and run correctQC

Make up and run correctQC

Make-up and run correctcalibrator

Make-up and run correctcalibrator

Fix identified instrumentfault

Fix identified instrumentfault

Rerun QC to confirmproblem fixed

Rerun QC to confirmproblem fixedCall for Help!Call for Help!

Y

N

Y

Y

N

N

N

N

Y

Y

Y

N

Problem Solved!

Problem solved, now what?

• Run QC for evidence that problem has beensolved.

• Document what you have done.Troubleshooting logs, QC annotation.

• Address patient results from previous goodQC to when QC failure occurred.– Repeat testing

• Exclude / Inactivate failed QC from dataanalysis if cause of the outlier is clearlyidentified.

Reagent lot changes

• CLSI EP26-A User Evaluation of Between-Reagent Lot Variation; Approved Guideline.

• “The protocol attempts to balance the need toreliably detect clinically significant change inreagent performance that may affect patientresults with the recognition that reagent lotverification is a relatively frequent task thatputs demands on the laboratory’s limitedresource”

QC material is not alwayscommutable• Shift in QC may not reflect a similar shift in

patient results• Just because QC does not show a shift, does

not mean the patients will not show a shift.

• Verify all new lots before they are put intouse.– Need only perform once for group of labs if using

same lot QC and reagent.

Test patient samples withcurrent and candidate

reagent lots

Test patient samples withcurrent and candidate

reagent lots

Estimate averagedifference between lots

Estimate averagedifference between lots

Averagediff < CD?Average

diff < CD?

Was QCacceptable with new

lot?

Was QCacceptable with new

lot?

Investigate lot difference.Do not report patientresults with this lot

Investigate lot difference.Do not report patientresults with this lot

Update QCtargets

Update QCtargets

Candidate lot acceptablefor patient testing

Candidate lot acceptablefor patient testing

YesYes

NoNo

CD: Critical Difference

Defining the critical difference

• Evaluation of comparability based on clinicaloutcomes.

• Evaluation of comparability based on clinical decisions– derived from either biological variation or– from data based on clinicians’ opinions.

• Published professional recommendations.• Performance goals set by regulatory bodies or

organizers of External Quality Assessment Schemes.• Goals based on the current state of the art as

demonstrated by data from EQA or from currentpublications on methodology.

Determine the statisticalpower

Determine the statisticalpower

Determine the number ofsamples to be tested and

the rejection limit

Determine the number ofsamples to be tested and

the rejection limit

Determine the number ofreplicates per sample

Determine the number ofreplicates per sample

Determine the number oftarget concentrations

Determine the number oftarget concentrations

• Alpha (Type I error, false positive)• Probability of detecting an error

when there is no error.• Typically set at 0.05

• Beta (Type II error, false negative)• Probability of not detecting a

difference when there is adifference

• Typically set at 0.20 to 0.05• Statistical Power = 1 – Beta.

Generally measure at clinical decisionpoints, often similar to the QC levels.

Statistical tables may suggest that only onepatient is required but it is recommendedthat at least three samples are used toavoid random errors.If difficult to get sufficient samples,replicate analysis may provide thenecessary additional statistical power.However, under normal circumstancesmeasure in singlicate

Conclusions

• Do not automatically re-run QC or recalibratean assay.

• Type of QC failure and Quality Control Chartsmay give a clue to the nature of the problem.

• Have a structured approach totroubleshooting.

• Have a process for managing your patientresults

• Prevention is better than cure– Manage introduction of new lots to the laboratory