rMethod Validation

Studies

It’s a Jungle out There!

r

Well Duh...

rPurposes of Method Validation Studies:• To quantifiably characterize system performance

• To assess potential for error

• To identify method-to-method differences

• To meet regulatory guidelines

r

When to Do Method Validation Studies:• When considering purchasing a new system

• When placing a new system into service

• At regular intervals to assess on-going system performance

• When troubleshooting questionable system performance

rBasic Components of Method Validation Studies:

Any or All of the Following...

• Precision Studies

• Accuracy Studies

• Range Validation/Verification Studies• Reportable Range• Critical Limits• Analytical Measurement Range• Clinically Reportable Range • Calibration Verification / Linearity• Reference Range Studies

rHow to Do Method Validation Studies:

Don’t wing it!!! Consult appropriate reference guides.

• NCCLS evaluation document series

• Manufacturer’s package inserts, evaluation protocol manuals and supplemental protocols.

r

Precision Studies• Characterize the reproducibility of a test system

• Performed by repetitive testing of the same sample• Intra-run• Inter-run• Inter-instrument

• Statistical parameters• Mean• Standard Deviation (SD), mg/dL• Coefficient of Variation (CV), %

• Acceptability Expectations…the lower the better

r

Accuracy Studies

• Method-to-method studies comparing a POC instrument to a properly calibrated primary Laboratory analyzer.

• For the purpose of assessing accuracy, the same samples should be tested on both the Laboratory analyzer and the Point-of-Care instrument.

rAccuracy Studies, cont’d:

• A typical Accuracy Study includes:• all, or a representative sub-set, of Point-of-Care

instruments• optimally at least 40 different samples

(minimum of 20 for statistical validity)• good distribution of sample values ranging

from low to high• same samples tested on Point-of-Care

instruments and Laboratory analyzer. If capillary samples are compared to venous glucose results, use only samples from fasting patients (Capillary glucose levels can be 20-70 mg/dL higher than venous levels at the same time in the non-fasting state).

rAccuracy Studies, cont’d

• Statistical analysis of accuracy study results typically includes:

• Regression analysis• Bias analysis

rAccuracy Studies, Cont’d

Raw Data Table

Lab Glucose mg/dL

Accu-Chek Glucose,

mg/dL

Lab Glucose mg/dL

Accu-Chek Glucose,

mg/dL19 12 108 11829 25 146 15834 30 189 18435 36 200 18437 38 217 18937 30 232 22238 40 237 21445 42 254 24447 52 266 24349 45 302 28355 55 321 29655 55 325 30959 57 330 29262 57 355 34174 78 365 34480 81 403 34282 85 405 38787 83 458 484105 112 552 551

r

Accu-Chek Vs Lab Regression Analysis

0

50

100

150

200

250

300

350

400

450

500

0 50 100 150 200 250 300 350 400 450 500

(X - Axis) Lab Glucose, mg/dL

(Y -

Axi

s) A

ccu-

Che

k G

luco

se, m

g/dL

Regression Analysis Plotting the Comparative Data...

r

Accu-Chek Vs Lab Regression Analysis

0

50

100

150

200

250

300

350

400

450

500

0 50 100 150 200 250 300 350 400 450 500

(X - Axis) Lab Glucose, mg/dL

(Y -

Axi

s) A

ccu-

Che

k G

luco

se, m

g/dL

Regression Analysis Plotting the Comparative Data...

r

Accu-Chek Vs Lab Regression Analysis

0

50

100

150

200

250

300

350

400

450

500

0 50 100 150 200 250 300 350 400 450 500

(X - Axis) Lab Glucose, mg/dL

(Y -

Axi

s) A

ccu-

Che

k G

luco

se, m

g/dL

Regression Analysis Plotting the Comparative Data...

rAccuracy Studies, cont’d

• Regression Statistics Review:• Correlation Coefficient (r) - characterizes the

dispersion of results around the line of best fit.• Slope - The “lean” of the line of best fit

(proportional bias)• Y-Intercept - the point at which the line of best fit

intersects the Y axis. (constant bias)

• Acceptability Criteria:• Correlation Coefficient (r) - the closer to 1.0 the

better• Slope - The closer to 1.0 the better• Y-Intercept - the closer to zero the better

r

Accuracy Studies, Cont’d

The Equation of the Line of Best Fit:

y = mx + b

m = Slope

b = Y Intercept

rAccuracy Studies, Cont’d

The Equation of the Line of Best Fit

y = 0.991x + 8.4

Slope

Y-intercept

rAccuracy Studies, Cont’d

The Equation of the Line of Best Fit

y = 0.991x + 8.4

Slope

Y-intercept

rAccuracy Studies, cont’d:

• Bias Analysis:• Calculate mg/dL or % difference between

Point-of-Care instrument and Laboratory analyzer for each set of paired data-points.

• Acceptability Criteria (Glucose):• 95% of meter results within +/- 15 mg/dL of

Laboratory analyzer results below 75 mg/dL or within +/-20% of Laboratory analyzer results equal to or greater than 75 mg/dL (NCCLS document C30A-2)

rBias Analysis, Cont’d

Bias ChartAccu-Chek vs Lab Bias Chart

-50

-45

-40

-35

-30

-25

-20

-15

-10

-5

0

5

10

15

20

25

30

35

40

45

50

0 50 100 150 200 250 300 350 400 450 500

Lab Glucose, mg/dL

Accu

-Ch

ek B

ias*

*Bias is calculated as mg/dL difference from lab results <75 mg/dL and as % difference from lab results >75 mg/dL.

rRange Validation Studies

• Definitions• Reportable Range - The range of results for

which a system has been proven to yield numerically accurate results

• Analytical Measurement Range (AMR) - The range of results through which a method yields numerical values

• Clinically reportable range - (CRR) An extension of the AMR supplemented by dilution or concentration protocols

• Calibration Verification - The process of verifying that a system is properly calibrated

• Critical Limits - Low and high result limits which, when exceeded, require follow up action

• Reference Range - Normal range

rRange Validation Studies

• Definitions• Analytical Measurement Range (AMR) - The

range of results through which a method yields numerical values

• Clinically reportable range - (CRR) An extension of the AMR supplemented by dilution or concentration protocols

• Calibration Verification - The process of verifying that a system is properly calibrated

• Reportable Range - The range of results for which a system has been proven to yield numerically accurate results

• Critical Limits - Low and high result limits which, when exceeded, require follow up action

• Reference Range - Normal range

rRange Validation Studies, Cont’d

• Validation of Reportable Range, Analytical Measurement Range, Reference Range, Critical Limits and Calibration Verification (CAP POC July 2003

Checklist, Page 34 Commentary) • Can all be accomplished through the same protocol if

the range of testing spans the broadest range (AMR)• Materials used must be “matrix appropriate” such as:

• Split blood samples• linearity/calibration verification materials supplied

by manufacturer or other vendor• However...

rCAP POC.08700 - If the laboratory has more than one method-system for performing tests for a given analyte, are they checked against each other at least twice a year for correlation of patient results?

NOTE: This includes same or different instrument makes/models. This comparison must include all instruments. If multiple methods are used for a particular analyte, comparing a subset of each method to one another is acceptable only after all like instruments have been compared to one another to ensure correlation and establish that a representative subset is used in the method-to-method

correlation.

rCAP POC.08700, Cont’d: Significant changes to the commentary!!!

• “POCT glucose devices are a special case in which a large number of devices may be in use in an institution. In this case alternate approaches are acceptable for documenting correlation of patient results among the devices and with the main laboratory. One approach is to document for a lot of reagent strips the agreement between patient results for a laboratory method and representative POCT device(s) and simultaneously collect results for QC materials run on the POCT devices. “

• “Other correlation schemes may be suitable. The critical factor is to have a mechanism that documents the agreement between patient results performed by the POCT glucose devices and the main laboratory.”

r

Where to Find the Method Validation Regulations

CLIA

CAP

JCAHO

rRegulatory Guidelines for Method Validation:

• CLIA 88 Final Rule - (January 24, 2003)

• CAP General Checklist (July 2003)

• CAP Point-of-Care Checklist (July 2003)

• JCAHO - WT and QC Section (2004 Pre-Publication with Crosswalks from 2003 Guidelines)

• NCCLS - C30A-2 - (Item 6.3.2).

r

CLIA Guidelines for Method Validation:

• CLIA 88 Final Rule - Subpart K (Quality Systems for Non-waived Tests) 493.1253 • Revised January 24, 2003 • Outlines requirements for unmodified “non-

waived”, FDA cleared tests• By definition, waived tests are not subject to

items in Subpart K

rCAP Guidelines for Method Validation:

• CAP Laboratory General Checklist • Revised July 2003 (GEN.42020 - GEN.42163)• Does not distinguish based on CLIA designated

test complexity • Requires validation of precision (intra- and

inter-run) and accuracy• Requires verification of analytic sensitivity

(Low end)• Requires assessment of analytical interference• Requires verification of reportable range (AMR)• Requires verification of reference range

rCAP Guidelines for Method Validation:

• CAP Laboratory Point-of-Care Checklist • Revised July 2003 (POC.08200 - POC.08700)• Does not distinguish based on CLIA

designated test complexity • Requires calibration verification• Requires validation of Analytical

Measurement Range • Requires method-to-method and instrument-

to-instrument correlation

rJCAHO Guidelines for Method Validation:

• WT Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines)• Item WT.1.40 - Non-specific reference to requiring

“Equipment performance evaluation”

• QC Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines)• Items QC.1.70 - Method Validation requirements

for precision, accuracy and reportable range apply to FDA cleared, unmodified moderate complexity tests.

• Requires reference range verification for all new methods

rJCAHO Guidelines for Method Validation:

• QC Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines)• Item QC.1.70

• Method Validation requirements for precision, accuracy and reportable range apply to FDA cleared, unmodified moderate complexity tests.

• Requires reference range verification for all new methods

• Item QC.1.80• Method-to-method and instrument-to-

instrument requirement specifies that waived tests are exempt.

r

Informative Regulatory Links• http://www.phppo.cdc.gov/clia/regs2/toc.asp

(CLIA Regs with 1/24/03 updates)

• http://www.aacc.org/meetings/qc/ (Oct. 29th Teleconference

• http://www.jcaho.org/accredited+organizations/laboratory+services/standards/new+standards/2004+standards.htm (JCAHO Pre-publication 2004 Standards with Crosswalk)

• http://www.cap.org/html/ftpdirectory/checklistftp.html (CAP Checklist download page)

• http://www.dgrhoads.com/ee5/ee5.sht (EP Evaluator Release 5)

r

Links to Informative Articles

• http://www.cap.org/captoday/archive/2003/0303/language_barrier.html (CAP Today on AMR/CRR and Calibration Verification)

• http://www.westgard.com/cliafinalrule5.htm (CLIA 1/24/03 method validation changes by Sharon Ehrmyeyer, Ph.D)

• http://www.westgard.com/essay50.htm (CLIA Key 1/24/03 Changes by Dr Westgard)

• http://www.westgard.com/essay15.htm (Method Validations: The Inner, Hidden, Deeper, Secret Meaning by Dr Westgard)

r

Method Validation Studies

Questions and Discussion

r