BMTA July 2005: 1

Valid Analytical Measurement Studies of Proficiency Testing scheme

performance

S Ellison

LGC Limited, Teddington

The work described in this paper was supported under contract with the Department of Trade and Industry

as part of the Valid Analytical Measurement programme

BMTA July 2005: 2

or63 routes to the wrong result

... and what to do about it

BMTA July 2005: 3

Introduction

• PT in analytical chemistry• Why study mistakes?• How does the UK do?

– PT results compared to international performance

• What goes wrong? (and why)– Web-based study of causes of poor PT scores

BMTA July 2005: 4

PT in analytical chemistry - organisation

• Typical rounds comprise:– test sample preparation, characterisation and distribution– analysis by participants– data collection and processing– preparation and distribution of the report

• Frequency– Typically 6-12 rounds per year

• Analytes (measured quantities)– 1-30 per sample per round

• Participants– Typically 30-100 per round, but strongly scheme-dependent

BMTA July 2005: 5

The aims of proficiency testing

• Primary aim:

“To provide the infrastructure for a laboratory to monitor and improve the quality of its routine analytical measurements”

• Other aims– Provide information on the state-of-the-art in analytical

measurements– Compare performance of analytical methods– Assist a laboratory in the validation of new methods

BMTA July 2005: 6

Principle of performance assessment

• Observed error – difference between

laboratory result (x) and assigned value (X)

• ‘Target range’– usually a standard

deviation ( ) or uncertainty

Compare…..

…..using an acceptability criterion

BMTA July 2005: 7

Performance Scoring: z-scores

x submitted result X assigned valuestandard deviation for proficiency assessment

Z 2 Satisfactory performance

2Z 3 Questionable performance

Z 3 Unsatisfactory performance

Xxz

Interpretation of z is consistent across schemesbut depends on

BMTA July 2005: 8

Typical analytical performance data.

-8 -6 -4 -2 0 2 4 6 8

Z-score

5 values < -8 3 values >8

Collected foodanalysis data:Various analytes

BMTA July 2005: 9

PT data for benchmarking

• Three studies of UK performance– Clinical– Food– Environment

• Clinical: Backed by IMEP-17 study (20 analytes: 35 countries)

• Food: FAPAS PT scheme data (6 representative analytes; 2000 labs; ca. 250 countries and regions)

• Environment: CONTEST and CoEPT project data

BMTA July 2005: 10

UK performance: Clinical

UK performance:Consistent withothers; Rarely poor

BMTA July 2005: 11

UK Performance: Food

Rest of world

UK

-5 0 5

GeMMA z-score by country

z-score

GMO measurement

BMTA July 2005: 12

UK Performance: FoodAflatoxins

BMTA July 2005: 13

UK Performance: FoodPirimphos-Me (pesticide residue)

GB Other

-10

-50

51

0

GB Pirimphos-Methyl Performance for all RoundsZ

-sco

re

UK Other

BMTA July 2005: 14

Problem analytes: ArsenicUK

FAPAS Arsenic data: Rounds 735-753GB results only

Z-score

Fre

qu

en

cy

-10 -5 0 5 10 15 20

01

02

03

0

BMTA July 2005: 15

FAPAS Arsenic data: Rounds 735-753abs(Z)<=20

Z-score

Fre

qu

en

cy

-10 -5 0 5 10 15 20

05

01

00

15

0Problem analytes: Arsenic

All countries

BMTA July 2005: 16

UK Performance: Environment

Total polycyclic aromatics

SO5 SO7 SO6 SO2 SO1 SO3

0

200

400

600

800

TotalPAH

Scheme ID

mg

/kg

UK SchemeOther

BMTA July 2005: 17

UK Performance: Summary

• Broadly comparable to other countries• No problems unique to the UK• Some problems (e.g. Arsenic) shared with other

countries

BMTA July 2005: 18

Part 2: Causes of error

• VAM Project KT2.4/3: Causes of poor PT performance• Aim:

Study "…the principal causes of poor performance in laboratories and ... the effectiveness of the steps taken by Participants in PT to improve the reliability of their results”

• Methodology– Web-based questionnaire– Focussed on documented problems identified via PT scores– Lead questions with follow-up for positive responses

BMTA July 2005: 19

Why study poor scores in PT?

• Why PT?– PT participants are already committed to quality improvement– Participants follow up poor PT scores

• Why only poor scores?– Acceptable scores give poor information about problems– Correlation of scores with general methodology is not very

effective– Every good lab has documented problems and corrective

actions

BMTA July 2005: 20

Top causes of poor scores

Sample preparationEquipment problemHuman error

CalibrationSelection of methodCalculation errorReporting problem

111 respondents230 causes

BMTA July 2005: 21

Top causes of poor scores

Sample preparation

Extraction/recovery

Dilution to volume

BMTA July 2005: 22

Top causes of poor scores

Equipment problem

Equipment failure

BMTA July 2005: 23

Top causes of poor scores

Human error

Training/experience

Transcription error

Reporting error

BMTA July 2005: 24

Top causes of poor scores

Calibration

No reference materialDefective RM

Incorrect procedureCalibration range

Calibration

BMTA July 2005: 25

Top causes of poor scores

Reporting problem

Reporting problems

Value correct but not in customer units

Transcription/typographical error

Incorrect units

BMTA July 2005: 26

Top causes of poor scores

Calculation error

Commercial software problem

Spreadsheet problem

Spreadsheet user errorCalculator error

Arithmetic error

Value mis-enteredSoftware mis-applied

Other

BMTA July 2005: 27

Corrective action

Training

New proceduresRevalidationMethod documentation

New equipment

Additional calibration

Method change

RM change

Other

Detailed information showed problem-specific responses

BMTA July 2005: 28

Corrective action - efficacy

• No significant difference in efficacy across different corrective actions

• Only 50% of actions were marked as ‘fully effective’

• Monitoring of efficacy tended to use local/immediate methods– Monitor QC results– Internal audit

BMTA July 2005: 29

Causes of error: Summary

• Most PT errors were caused by basic lab operations– Incorrect dilution to volume– Transcription and reporting errors– Data and spreadsheet formula entry errors

• Equipment failure is perceived as a problem• Extraction/recovery problems important• Commercial software faults caused no problems

• Corrective actions are problem-specific and ‘multifactor’– More than one action generally required.

BMTA July 2005: 30

Conclusions

• UK analytical labs perform similarly to international partners, and share similar problems

• The most common cause of PT performance failures are not technical, but simple human errors such as incorrect volumetric operations and transcription errors

• Time to look harder at human factors?

• Study web page: via http//www.vam.org.uk - surveys link