PT FREQUENCY

Background

• CCIL strongly endorses the requirement foracceptable PT performance to maintain accreditation

Background

• ISO / IEC 17025 and the accreditors, SCC,CAEAL andMDDEP require PT participation

• No consensus Internationally on frequency

• Majority of laboratories

– Once every 4 years (minimum)

– 2 X per year

Background

• Under active consideration internationally

• APLAC has a committee addressing the issue

– Leaning toward a 1 X 2 sample approach

– SCC and CAEAL attend all meetings

Background

• PT providers must be accredited to ISO Guide 43(Can-P-1594:2001)

• CAEAL provides the lion’s share of Canadian PTsamples

Background

• Historically CAEAL has distributed 2 sets of four PTsamples / yr for the parameters they supply.

• Require a minimum of two samples / yr for otheraccredited parameters.

Background

• PT costs are the major component of accreditationcosts

$4,400$2.900Small Lab

$30,000$5,500Large Lab

PTAssessment

Background

• PT samples are an essential part of a Lab QualityProgram

• Can we reduce the number of samples and maintainthe integrity of the program?

• CCIL decided to evaluate the effectiveness of thecurrent program

The Study

• Member labs were ask to submit 10 sets of failing ormarginal data (Score 70% to 80%) plus the data forthe same parameter(s) in the subsequent study

• 232 data sets were submitted from 21 labs

The Study

• In response to customer feedback and workshopinput, CAEAL recently developed and implemented anew scoring system

– New vs Old was evaluated

– CAEAL New vs SCC protocol was also evaluated

The Study

• Items evaluated

– Range of PT sample sets

– How often do all four samples trend in the samedirection?

The Study

• Items evaluated

– Effect of reduced sample sets

• Recalculated scores using samples

– 1 & 3, 2 &4

– 1, 3

Disclaimer

• The conclusions discussed below are based on alimited data set

• Analysis of larger data sets may lead to differentconclusions.

Findings

• CAEAL Old– Scores of 5,4,2 or 0 were assigned to each sample

on the following basis:– < 1 sd from the mean 5– >1 < 2 sd from the mean 4– >2 < 3 sd from the mean 2– >3 sd from the mean 0– Scores from 4 samples summed X 5– Scores ≥ 70 were acceptable

Findings

• CAEAL New– Calculate Z score = │(xi – X) │/ sd– Where– xi is the reported result– X is the assigned value, usually the adjusted

mean– sd is the standard deviation of the adjusted mean

Findings

• CAEAL New– Z Score modified to account for low level data– Z score = │(xi – X) │/ √ ( sd2 + (RDL/3)2)– Z > 6.6 assign Z = 6.6– “<“ reported, RDL > assigned value• Z=2 (some exceptions)

– “<“ reported, RDL < assigned value• Use RDL value to calculate Z

Findings

• Reporting Low Level Data– 2 Approaches RDL = MDL, RDL >> MDL– Result < 5X MDL and other techniques more

sensitive• Greatly increased chance of failure

Findings

• CAEAL New– Calculate average Z Score– PT score (%) = 100 + (-15* avgZ).– PT Score ≥ 70% is acceptable

Findings

• SCC– Any Z Score >3 is unacceptable

Findings

• CAEAL New vs Old

– 77% of the results improved

– Average 9.5 points

– 39 of 180 results went from fail to pass

• CAEAL New vs SCC

– CAEAL 102 failures

– SCC 79 failures

Findings

• CAEAL New Protocol

– Addresses previous concerns

– Appears Fair and well thought out

Findings

• Sample Ranges

– The average concentration range covered by thefour samples all parameters was 9

– Organics 6

– Wet Chemistry 11

– Metals 12

– 4 of 149 sets had a range > 50

Findings

• Sample Ranges

– Most current analytical techniques cover 2 ½ to4 ½ orders of magnitude

– In general the CAEAL PT samples do not coverthe calibration range

Findings

• Trends

– 79% of the data sets had all four samples on thesame side of the assigned value

– Confirmed by CAEAL analysis of a full years data

• PT > 80 48%

• PT 70 – 80 79%

• PT < 70 73%

Findings

• Reasons

– Samples normally analyzed together in a singlerun. Samples are homogeneous, relatively closein concentration

– Most sources of lab error produce consistent bias

• Inaccurate standards

• Calibration bias

• Drift

Findings

• Recurrent Problems

– 11% of data sets had problems on the subsequent set

– CAEAL 3 full studies

• Consecutive failures 23%

– i.e. a significant number of labs were unable to correct theirproblems

– Argues for at least a 2X / yr frequency

Findings CAEAL Protocol

8%5%Pass to Fail

7%13%Fail to Pass

3%2%Change > 20%

18%16%Change > 10%

Samples 2 & 4Samples 1 & 3Score ChangeCompared to4 Sample Set

Findings CAEAL Protocol

19%18%Pass to Fail

11%9%Fail to Pass

16%11%Change > 20%

43%33%Change > 10%

Sample 4Sample 2Score ChangeCompared to4 Sample Set

Findings SCC Protocol

79

4 Samples

12%12%

2727Fail to PassPass to Fail

5252Failures

Samples 2 & 4Samples 1 & 3

Findings SCC Protocol

79

4 Samples

21%20%

4845Fail to PassPass to Fail

3134Failures

Sample 4Sample 2

Findings

• 2 Sample sets give much better correlation withcurrent practice than single sample sets

Findings

• Two sample sets correlate well with 4 sample sets

– 17% Pass to Fail or Fail to Pass

• These data represent about 15% of the total

– ≈ 10% failure ≈ 5% marginal

– Other 85% all pass

Findings

• Extrapolating to the full data set

– 2 Samples vs 4

CAEAL Protocol

– 97.5 % of labs, same result

SCC Protocol

– > 98% of labs, same result

Conclusions:

• Frequency should not be decreased

– Substantial %age of repeat problems

• Two sample sets provide adequate problem detection

– Similar pass / fail rates

Conclusions:

• Four sample sets do not provide enough additionalinformation to warrant the additional effort

– All 4 samples trend the same direction 79% of the time

– Average range < one order of magnitude

Current Status:

• Report sent to CAEAL and SCC

• Prompt response from both

• Additional database analysis by CAEAL

– Indicated a risk based approach might be viable

• Willing to entertain changes

– Provided the program isn’t weakened

Current Status:

• Working Group formed

– Robert Lessard

– Barry Loescher

– Ken Middlebrook

– SCC

• Major topic at CAEAL June AGM