Presentation slides of Dr. Martina Hedrich

BAM I Department of Analytical Chemistry, Reference Materials APLAC RMP Accreditation Training Course, Tsukuba/Japan 2007-06-01 1

Martina HedrichFederal Institute for Materials‘ Research and Testing (BAM)

Unter den Eichen 87, D-12205 Berlin, Germany

Adriaan van der VeenNederlands Meetinstituut (NMi)

Thijsseweg 11, NL-2629 JA Delft, The Netherlands

ISO Guide 35 –Design of a Certification Project


Revision of ISO Guide 35Guide published in 2006

ISO Guide 35:2006 Reference materials --General and statistical principles for certification

Development took 7 yearsFinal bits and pieces: – Wrapping up discussions on RM/CRM– Processing comments from voting– Editorial aspects


What is new?

Definitions RM/CRMHomogeneity testing in connection with uncertainty estimationStability testing– testing the assumed model in a stability study– estimation of shelf life and uncertainty

Characterisation– use of uncertainty in characterisation of an RM


Revision of RM/CRM definitions

Purpose– Widen their scope to

» Qualitative measurement» Measurement of curves (and structures in more

dimensions)– Reflect that a CRM is a specific kind of RM– Create consistency with ISO Guides 34 and 35


General considerations

Increased awareness of quality assurance in qualitative measurementMany CRMs are certified for non-numerical propertiesFuture guidance may encompass RMs for qualitative measurementMetrology in qualitative measurement is quickly developing


Reference Material

3.1 reference material RM Material, sufficiently homogeneous and stable with respect to one or more specified properties, which has been established to be fit for its intended use in a measurementprocess.

NOTE 1 RM is a generic term

NOTE 2 Properties can be quantitative or qualitative, e.g., identity of substances or species.

NOTE 3 Uses may include the calibration of a measurement system, assessment of a measurement procedure, assigning values to other materials, and quality control.

NOTE 4 An RM can only be used for a single purpose in a given measurement


Certified Reference Material

3.2 certified reference material CRM reference material, characterized by a metrologically valid procedure for one or more specifiedproperties, accompanied by a certificate that provides the value of the specified property, its associateduncertainty, and a statement of metrological traceability

NOTE 1 The concept of value includes qualitative attributes such as identity or sequence. Uncertainties for suchattributes may be expressed as probabilities.

NOTE 2 Metrologically valid procedures for the production and certification of reference materials are given in, among others, ISO Guides 34 and 35.

NOTE 3 ISO Guide 31 gives guidance on the contents of certificates.


Definition of the uncertainty of a property value of a CRM

Uncertainty of a property value of a CRM equals– uncertainty of the certified value as obtained for

the batch (CHARACTERISATION)– transferred to a single package

(HOMOGENEITY)– as dispatched to the customer (STABILITY)– at the time of sale (STABILITY)


Why these requirements?

Property values sufficiently accurate for intended useCustomer only interested in the package he sees, at the time that it arrives at his desk!


Contents of ISO Guide 35

ScopeIntroductionNormative referencesDefinitionsSymbolsDesign of a certification project


Contents ISO Guide 35 (continued)Evaluation of measurement uncertaintyHomogeneity studyStability studyDetermination of property valuesData and uncertainty evaluationCertificationAnnexes– statistical approaches– examples


Scope of ISO Guide 35 Statistical principles to assist in the understanding and development of valid methods to assign values to properties of a reference material, including– evaluation of uncertainty– metrological traceability

Not comprehensive in every respect;Mainstream approaches assuming normally

distributed data


Project definition Example:“Preparation of a soil CRM containing a series

of trace elements at relevant content levels for environmental analytical chemistry with an uncertainty associated with the certified values of less than or equal to x %”

Crucial– fit for intended use– proper choice of stated references


Samplepreparation

Sampling

Samplechoice

Subdivision

Bottling &packaging

Parameterchoice

Matrixchoice

Drying Homogeni-sation

Stabili-sation

Homogeneitytesting

Characteri-sation

Stabilitytesting Certification

Production and certification process


Sample choice

Matrix, relevant to measurement practiceProperty values of interestOptions for obtaining material that is “sufficiently homogeneous” and “sufficiently stable”Material should be equipped with good mechanical properties


SamplingSampling strategy dependent on specifications of sampleSampling often non-representative, selectiveCollect and prepare sufficient material for all experimentsMajor consumers of samples are– feasibility study– homogeneity and stability tests– characterisation


Sample preparation

Improvement of stability– drying– conservation– proper packaging/temporary storage

Improvement of homogeneity– crushing/grinding to unimodal particle size

distribution– mixing/blending


Subdivision and bottling

Choice of proper packaging materialSubdivision should ensure same composition for all samples of the batchPotential problems– flow behaviour– random fluctuations in bulk composition– segregation


Number of packagesDependent upon– number of samples of the (C)RM needed;– need for a feasibility study;– number of samples needed for the homogeneity

study;– number of samples needed for the stability study;– number of samples needed for the

characterisation of the candidate CRM;– amount of material needed for one measurement.


Project planning formCandidate material (matrix, amount, shape)Intended use (motivation)Parameters to be certified, uncertainties Homogeneity test (who, how much, when)Stability test (who, how much, when)Certification campaign (who, how much, …)Duration of projectResponsible…






ISO Guide 35 –Evaluating Measurement Uncertainty


Fundamental paper is the GUMa) Set up model equationb) List all sources contributing to uncertaintyc) Evaluate standard uncertainty (type A/type B)d) Evaluate covariances between input quantitiese) Calculate property valuef) Combined standard uncertainty (propagation)g) Determine coverage factor k

(expanded uncertainty U)h) Report property value together with U and k


Other approaches …

… should be chosen in cases– without a closed mathematical form between

property value and input quantities– where linear approximation (propagation) is

invalid

Monte Carlo, bootstrap methodsResults of validation/intercomparison studies


Homogeneitytesting

Characteri-sation

Stabilitytesting

Long-termtest

Short-termtest

Between-bottle test

Within-bottle test

Trend?

Trend?

Homogeneity OK?

Minimumsample intake

u(bb)

u(lts)

u(sts)

Shelf life

Property values +uncertainties U(CRM)

Yes!

No!

No!


Evaluation of measurement uncertainty (clause 6)

Main process: see Guide to the expression of uncertainty in measurement (GUM)ISO Guide 35 outlines some critical aspects– Certification model– Compilation of list of all uncertainty

components– Choice of a coverage factor– Recertification


Uncertainty of a CRM

Uncertainty of a CRM is given by:

where uchar is the uncertainty component derived from the characterisationmeasurements

22222charstsltsbbCRM uuuuu +++=


Uncertainty sourcesList all relevant uncertainty components:

PurityBalanceVolumetric flaskCalibrationInstrumental read-outRecovery rate…


Distribution function

Regression analysis and analysis of variance assume normally distributed data setsIn case of multiple maximums– certification impossible– method dependent certification


“S-shaped” curve

0,71

0,72

0,73

8/I 4/P 7/P 6/A 5/A 3/A 5/P 1/I Ges.

Mn [%] in CuNiFe (BAM-368)


“Camel” distribution


Coverage factor

Normal distribution, 95 % level of confidencek = 2

Normal distribution, 99 % level of confidencek = 3

Poisson distributionconfidence interval

Student’s t-distributionat low degrees of freedom


Recertification Property value changes over time:

Withdrawal of CRM in case of– (matrix) deterioration– remaining batch too small

Recertification including– homogeneity test– stability test– characterisation

(gradual change – time dependent function)


Trends

Matrices become more complex– Homogeneity and stability issues– Preparation issues

Properties to be certified are more complex– CRMs for identification– Particle size distributions– Complex measurands


TrendsTransport issues– Customs problems– Text on labels sometimes give rise to trouble– Transport of hazardous materials

User’s perspective– Incomprehensible certificates– Missing traceability and/or uncertainty

statements– Contamination issues with CRMs for multiple

use


New challenges

Improvement of technology for preparing homogeneous and stable batches of samplesAddressing the customer’s needs better– Does the user understand how the CRM can/should

be used?Development of new recipes for assigning values and evaluating MU


Measurement uncertainty

GUM fundamental document for expressing MU of property values ISO Guide 35 provides basic guidance and a mainstream approachStatistical models in ISO Guide 35 are not exhaustive, and for some materials clearly not valid


Model equation

1

111

K

KKK M

Pmn

∗=

n: Amount of substance in molm: Mass in gM: Molar mass in g/molP: Purity, no dimension

Gas mixture, gravimetrically prepared, component K1


Fishbone diagram

Wägung (D)Wägung (K)

WP

WM

W0

DVU + DS

DA DR

DVU + DS

W1

ny * AE 2

x AE

VU3

VU4

mK1

WWWA

WW

WP

WA

WM

DR DA






ISO Guide 35 –Homogeneity Study


Concepts of homogeneity

3.5between-bottle homogeneity bottle-to-bottle variation of a property of a reference material

NOTE It is understood that the term “between-bottle homogeneity” applies to other types of packages (e.g. vials) and other physical shapes and test pieces.

3.6 within-bottle homogeneity variation within one bottle of a property of a reference material


Homogeneity testing

“within-bottle” → minimum sample intake“between-bottles”– batch homogeneity– quality control– determination of remaining heterogeneity

among samples


Nested uncertainties

Homogeneity study, stability study and characterisation contain “nested uncertainties”Uncertainty calculation frequently requires separationAnalysis of variance is theoretical basis


Bottle#1

Transf.#2

Transf.#1

Ms. #2

Ms. #1

Ms. #2

Ms. #1

Bottle#k

Transf.#2

Transf.#1

Ms. #2

Ms. #1

Ms. #2

Ms. #1

Bottle#2

Transf.#2

Transf.#1

Ms. #2

Ms. #1

Ms. #2

Ms. #1

Between-bottle homogeneity test (2-way layout)

W

B

A

A - between-bottle variation

B - subsampling + transformation variation

W - measurement repeatability


Use of analysis of variance

Allows to extract uncertainty components in experiments– step by step assessment of experimental

performance– extraction of uncertainty components (e.g.

inhomogeneity, instability)Classical statistics– variances sensitive for outliers/stragglers– estimates sometimes too pessimistic


Data requirements

Independence among “ between groups” and “within group” effectsHomogeneity of within group variancesEqual group and subgroup sizesUnimodal distribution


Uncertainty modellingSingle measurement in a homogeneity study

For measurement, only the independent part is relevant– Common contributions do not add to scatter of

results– Calibration (etc.) only relevant for linking

results (in certification)

( ) 222measbbij uuyu +=

( )n

uuyu rpt

bbij

222 +=


TLS Regression curve

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 0,2 0,4 0,6 0,8 1 1,2

amount-of-substance fraction (mmol/mol)

Resp

onse

(a.

u.)

Calibration curve n-hexane


Example

Homogeneity of methane in natural gasMethane is main component, typically x > 0.90 mol/molHomogeneity should be very good: very small sbb

Uncertainty in measurements known (from validation study)


Concept of ANOVA: partitioning sums of squares

Total sum of squares …

… can be partitioned into

( )∑∑= =

−=a

i

n

jijtotal

i

YYSS1 1

2

( )∑∑= =

−=a

i

n

jiijwithin

i

YYSS1 1

2

( )∑=

−=a

iiibetween YYnSS

1

2


Basis for variance estimation:mean squares

Mean square “within group”

Mean square “between groups”

an

SSMS a

ii

withinwithin

−=

∑=1

1−=

aSSMS between

between


Degrees of freedomOften, not all groups have equal membersNumber of group members estimated through

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

−−

=

∑

∑∑

=

=

=a

ii

a

iia

ii

n

nn

an

1

1

2

10 1

1


Example of poor repeatability

Homogeneity data methane

92.092.192.292.392.492.592.692.792.892.993.0

1 2 3 4 5 6 7 8 9 10

Mixture #

amou

nt-o

f-su

bsta

nce

frac

tion

(cm

ol/m

ol)


Homogeneity data of methane fraction in natural gas


92.092.192.292.392.492.592.692.792.892.993.0

1 2 3 4 5 6 7 8 9 10

Mixture #

amou

nt-o

f-su

bsta

nce

frac

tion

(cm

ol/m

ol)


Example of insufficient statistical control


92.092.192.292.392.492.592.692.792.892.993.0

1 2 3 4 5 6 7 8 9 10

Mixture #

amou

nt-o

f-sub

stan

ce fr

actio

n (c

mol

/mol

)


2nd example: homogeneity of Cr in soil

Sample Results1 119,42 124,852 128,54 123,223 121,85 123,214 116,09 121,745 127,30 115,416 118,41 133,297 115,79 120,308 122,65 118,499 119,89 120,92

10 128,80 122,99


Homogeneity of Cr in soil (continued)

Sample Count Sum Average Variance1 2 244,27 122,14 14,742 2 251,76 125,88 14,153 2 245,06 122,53 0,924 2 237,83 118,92 15,965 2 242,71 121,36 70,696 2 251,70 125,85 110,717 2 236,09 118,05 10,178 2 241,14 120,57 8,659 2 240,81 120,41 0,53

10 2 251,79 125,90 16,88


Homogeneity of Cr in soil (ANOVA)

Source of Variation SS df MS FBetween Groups 150.52 9 16.72 0.63Within Groups 263.40 10 26.34

Total 413.93 19

nMSMSs withinbetween

bb−

=2

Repeatability of measurements is poorso that sbb cannot be determined by this formula


Homogeneity of Cr in soilSample Results

1 115.63 114.74 124.39 123.39 124.262 119.55 115.37 120.27 114.82 120.193 133.52 120.25 120.08 122.65 118.424 116.95 116.08 127.81 123.25 115.845 123.54 112.10 115.80 114.79 113.086 113.87 117.95 116.41 123.37 127.507 117.22 135.52 125.11 120.96 135.568 122.18 119.52 134.05 113.76 124.229 118.58 112.24 110.53 117.06 114.28

10 129.25 121.25 130.44 122.24 126.52


Homogeneity of Cr in soil (ANOVA)Source of Variation SS df MS FBetween Groups 723.78 9 80.42 2.67Within Groups 1203.64 40 30.09

Total 1927.41 49

mg/kg 1.42 =−

=n

MSMSs withinbetweenbb

Increase of the number of replicates leads to better repeatability of the sample means ⇒ sbb can now be determined; sr = 5.5 mg/kg, s = 2.5


Outliers

Invalid results should be removed from the dataset (failures in measurement)Not to be rejected on statistical grounds– can be a phenomenon in the batch– results of a homogeneity study inconclusive

Options– analysis of more samples– repetition of the homogeneity study

1

StatisticalHomogeneity Test

Principle of ANOVA

Homogeneity Test

Deviation of a single value from the sample mean value

of all runs:

Li xx − This is a measure of the "method scatter".

(Condition: No surface inhomogeneity!)

Homogeneity Test

Deviation of the sample mean value Lx

from the overall mean x :

xxL −

This is a measure of the "scatter between the samples".

Homogeneity Test

Deviation of a single value from the overall mean:

( ) ( )xxxxxx LLii −+−=−

Homogeneity Test

( ) ( ) ( )∑∑∑∑= ==

−+−⋅=−Z

L

M

kLk,L

Z

LL

N

ii xxxxMxx

1 1

2

1

22

between method

Number of samples Z (index L)

Number of „runs“ M (index k)

Total number of measurements N = M * Z

Homogeneity Test

( )1

1

2

2

−

−⋅=∑=

Z

xxMs

Z

LL

between

2

Homogeneity Test

( )ZN

xxs

Z

1L

M

1k

2LkL,

2method −

−=∑∑= =

∑=

⋅=Z

1L

2L

2method s

Z1s

Homogeneity Test

2method

2between

ssD =

ZNf1Zf

2

1

−=−=

F-test: variance „between“ samples and „method“ variance

Statistic D is the quotient of thevariance „between“ samplesand the „method“ variance

ANOVA Homogeneity Test

2method

2between ss >

ProblemInhomogeneities smaller than the

scatter of the method cannot be detected!

⇒ Scatter of the method has to beas small as possible!

Decision Criterion for anANOVA Homogeneity Test

Only random difference among

sbetween und smethod !

⇒No Inhomogeneity

Systematic difference among

sbetween and smethod !

⇒Inhomogeneity is significant!

2method

2between ss >

Decision Criterion forANOVA Homogeneity Test

ProblemHow large a difference has to be among thevariance of the „method“ and the variance„between“ the samples in order to detect a

significant inhomogeneity?

2method

2between s/sD =

2method

2between ss >

Comparison of the scatter „between“the samples and the „method“ scatter

s(between) > s(method)

s(between) > K · s(method)

K = f(Z) = c · 1/ZZ: number of samples tested

Necessary condition

Sufficient condition

3

To decide whether the material isinhomogeneous, it is NOT sufficient that

sbetween is slightly larger than smethod!

The difference has to beconsiderably larger.

Fuzziness

Decision Criterion forANOVA Homogeneity test

211 f,f,FD α−>

ZNf1Zf

2

1

−=−=

Decision Criterion forANOVA Homogeneity Test

⇒ significant inhomogeneity isdetected!

„Classical“ approach:

Negative result of homogeneity test: ignorance of inhomogeneity,

i.e. its contribution is set equal to zero!

Positive result of homogeneity test: material is discarded or

inhomogeneity „silently“ accepted!

Problem

How to evaluate the result of a homogeneity test?

Evaluation of Homogeneity Test

ProblemDoes the criterion of significance need to be

considered in stating the inhomogeneitycontribution ?

211 f,f,FD α−>

Decision criteria forANOVA Homogeneity test

Possibility AInhomogeneity is only taken into

account in calculating the uncertaintyin case of a „significant“ test result.

Possibility BInhomogeneity contribution is to be

included in the uncertainty calculationin any case.

4

Guidelines for the production and certificationof BCR reference materials

6. HOMOGENEITY TESTING

6.4 Sample size

Notes

Where inhomogeneity is detected but does notexclude certification, the uncertainty associatedwith the inhomogeneity must be taken intoaccount. This uncertainty is expressed as thehalf-width of the 95% / 95% toleranceinterval, i.e. the interval which contains with95% certainty 95% of the population.

Definition of a Tolerance Range for a CRM

k = f (n)

With a risk ofa % are

P % of "all values"of the RM situated

between theselimits

M - k s M M + k s

P %

P %

100 - P %

Account for Inhomogeneitiesin calculating the uncertainty of a

certified value

2inhom

2char snss +=

Problem

The variance of the „method“ is also included in the variance „between“samples!

How to quantify an inhomogeneitycontribution to the uncertainty?

2method

2betweeninhom sss −=

Categories for stating theuncertainty

1. Always state the complete scatter„between“ samples as the contributionof inhomogeneity

2. Always state the difference of thescatter „between“ samples and the„method“ scatter as a contribution to uncertainty

3. State the complete scatter only in caseof significance

4. State the difference only in case of significance

Variances of testing Homogeneity

( )1

1

2

2

−

−⋅=∑=

Z

xxMs

Z

LL

between

( )ZN

xxs

Z

1L

M

1k

2LkL,

2method −

−=∑∑= =

5

Homogeneity test (AlMg4,5Mn) – Cr [%]Nr. Proben 1. DG 2. DG 3. DG 4. DG 5. DG 6. DG M s2(innerhalb)1 3 0,147 0,148 0,149 0,147 0,146 0,146 0,1474 0,00000142 7 0,147 0,149 0,147 0,148 0,151 0,145 0,1480 0,00000493 18 0,148 0,151 0,147 0,146 0,148 0,146 0,1477 0,00000404 23 0,148 0,150 0,147 0,147 0,147 0,146 0,1474 0,00000165 36 0,148 0,149 0,150 0,147 0,149 0,147 0,1486 0,00000136 39 0,147 0,150 0,147 0,148 0,147 0,147 0,1476 0,00000107 41 0,148 0,148 0,148 0,149 0,148 0,146 0,1479 0,00000078 58 0,148 0,151 0,151 0,149 0,149 0,147 0,1489 0,00000259 68 0,148 0,153 0,148 0,148 0,149 0,147 0,1488 0,000003610 72 0,149 0,152 0,150 0,150 0,148 0,149 0,1495 0,000002011 89 0,149 0,151 0,148 0,149 0,148 0,149 0,1490 0,000000712 91 0,149 0,153 0,152 0,150 0,151 0,148 0,1505 0,000003613 97 0,150 0,153 0,150 0,152 0,150 0,151 0,1511 0,000001614 123 0,153 0,153 0,151 0,150 0,152 0,151 0,1517 0,000001915 155 0,149 0,154 0,150 0,151 0,150 0,149 0,1504 0,000003116 177 0,149 0,154 0,150 0,150 0,150 0,151 0,1509 0,000003217 199 0,149 0,153 0,152 0,151 0,151 0,151 0,1512 0,000001618 219 0,151 0,154 0,151 0,150 0,150 0,150 0,1509 0,000002419 232 0,150 0,152 0,150 0,149 0,149 0,149 0,1498 0,000001420 244 0,149 0,154 0,150 0,150 0,150 0,149 0,1502 0,0000035

s2(zwischen) 1,21E-05 2,30669E-06

s2(method)

s2(between)

ISO Guide 35

(3rd Edition)

Certification of reference materials

General and statistical Principles

2006

Stating an inhomogeneitycontribution does no longer depend

on exceeding the critical valueof the test!

ANOVA

( )∑=

−⋅=Z

1L

2Lbetween xxMMS

( )∑∑= =

−=Z

1L

M

1k

2LkL,method xxMS

korr

methodbetweeninhom2

bb nMSMSsu −

=≡

⎥⎦

⎤⎢⎣

⎡⎟⎠

⎞⎜⎝

⎛−⋅

−= ∑∑∑

===

Z

1ii

Z

1i

2i

Z

1iikorr nnn

1Z1n

ubb at sufficient repeatabilityANOVA

( )∑∑= =

−=Z

1L

M

1k

2LkL,method xxMS

4

MS

methodbb

methodν2

nMSu ⋅=

ubb at insufficient repeatability

D < 1:

D = 1

1< D< F1-α, f1, f2

D =F1-α, f1, f2

D > F1-α, f1, f2

s2between < s2

method

s2between = s2

method

s2between > s2

method ; D < F1-α, f1, f2

s2between > s2

method ; D = F1-α, f1, f2

s2between > s2

method ; D > F1-α, f1, f2

Decision Criteria for ANOVAHomogeneity Testing

Evaluation of a Homogeneity Test according to ISO Guide 35

F = 1 F = F_crit

u_bb0

methodbetween2A n

MSMSs −=

4

MS

methodbb

methodν2

nMSu ⋅=

6

Testing Homogeneityis a difficult task !

Wirksamkeit eines Homogenitätstests (95%)szwischen/sVerfahren

00,20,40,60,8

11,21,41,61,8

0 10 20 30 40 50 60Anzahl der Proben

s(zw

isch

en)/s

(Ver

fahr

en)

25

1,27

Wirksamkeit eines Homogenitätstests (99%)szwischen/sVerfahren

00,20,40,60,8

11,21,41,61,8

22,2

0 10 20 30 40 50 60Anzahl der Proben

s(zw

isch

en)/s

(Ver

fahr

en)

25

1,40






ISO Guide 35 –Stability Study


Concepts of stability3.10short-term stability stability of a property of a reference material during transport under specified transport conditions

3.11 long-term stability stability of a property of a reference material under specified storage conditions at the CRM-producer

3.12 life time ⟨of a reference material⟩ time interval during which a reference material may be used

3.13 shelf life ⟨of an RM/CRM⟩ time interval during which the producer of the CRM warrants its stability

NOTE The shelf life is equivalent to the period of validity of the certificate, as described in ISO Guide 31.


Stability of reference materials

Two types– long-term stability (at the shelves of the

producer/reseller)– short-term stability (during transport)

Ideally, transport conditions/material should be established so that short-term stability effects do not exceed those of long-term stability


Stability study

Two experimental layouts– classical– isochronous

Planning may include stability monitoring/testing after certificationLinkage between ults and shelf lifeShort-term stability only relevant if greater than long-term stability


Shelf life versus life time

0

2

4

6

8

10

12

14

Certification period Life time

Tim

e (y

ears

) Certificate #5Certificate #4Certificate #3Certificate #2Certificate #1


Stability studies and life time

-4

-2

0

2

4

6

8

10

12

14

Stability study Life time

Tim

e (y

ears

) PreparationCertificate #5Certificate #4Certificate #3Certificate #2Certificate #1


Stability testingshort-term– behaviour of samples under transport

conditions– ideally, should not show any trend, and should

not exceed long-term stability uncertaintylong-term– behaviour of the samples at the shelves of the

producer– should not show any meaningful trend


Transport conditions

Temperatures may vary between –50 °C and +70 °CHumidity may vary appreciablyCorrosive conditions (e.g., sea water!)Mechanical shocks…


Classical stability study

Measurements after– 3 months– 6 months– 12 months– 18 months– 24 months

Storage of samples at Tstudy


In the beginning …


After 3 months …

MeasurementResult


After 6 months …

MeasurementResult


After 12 months …

MeasurementResult


After 18 months …

MeasurementResult


After 24 months …

MeasurementResult


At the end of the study

Results – after 3, 6, 12, 18, and 24 months– obtained under reproducibility conditions

Data processing– Trend analysis– Trend? → decision whether samples are usable– Uncertainty evaluation


Isochronous stability studyAge of samples– 3 months– 6 months– 12 months– 18 months– 24 months

Samples are stored at TrefSamples to be measured are put at TstudyNo results before the end of the study!


In the beginning …

3 6

18 24

12

T = Tstudy


After 0 months …

3 6

18 24

12

T = Tstudy


After 6 months …

3 6

18 24

12

T = Tstudy


After 12 months …

3 6

18 24

12

T = Tstudy


After 18 months …

3 6

18 24

12

T = Tstudy


After 21 months …

3 6

18 24

12

T = Tstudy


After 24 months …

3

6

18

2412

MeasurementResults


At the end of the study …

Results – after 3, 6, 12, 18, and 24 months– obtained under repeatability conditions

Data processing– Trend analysis– Trend? → decision whether samples are usable– Uncertainty evaluation


Month#0

Month#24

Month#12

Month#6

Month#3

Month#36

Measurement MeasurementMeasurementMeasurementMeasurement Measurement

Month#36

Month#12

Month#24

Month#30

Month#33

Month#0

Storage at RT Storage at RTStorage at RTStorage at RTStorage at RT Storage at RT Measurement

MeasurementStorage at LT

Isochronouslay-out

Classicallay-out


Day #1

Bottle#2

Bottle#1

T+M #2

T+M #1

T+M #2

T+M #1

Day #k

Bottle#2

Bottle#1

T+M #2

T+M #1

T+M #2

T+M #1

Day #2

Bottle#2

Bottle#1

T+M #2

T+M #1

T+M #2

T+M #1

Short-term stability test (2-way layout)

W

B

A

A - between-day trend/variation

B - between bottle variation

W - subsampling/transformation + measurement repeatability

Stability study: 2-way ANOVA layout


Uncertainty from a stability studyUncertainty contribution

Notes– Number of points in time has an impact on the

uncertainty in sstab

– if multiple bottles are used, sstab contains sbb

– if measured under reproducibility conditions, sstab contains slor

( )n

ssyu methodstabi

222 +=


Trend analysisFor most chemical CRMs, ideal case is

Model for trend analysis in stability studies is empiricalFor small effects of instability, straight line suffices

constant=CRMx

taax CRM 10 +=


Mechanisms for instabilityTemperature dependent– Instability at T1 cannot be translated to T2

– Degradation is more rapid at elevated temperaturesLarge differences between parametersDifferences between matricesEffects of– Contaminants/contaminations– Presence of moisture, oxygen (etc.)– …

True mechanism rarely known (!)


Example: H2S in nitrogen

Artefact under study: 1 cylinder (from a set of 6)Measurements taken under reproducibility conditionsCalibration done against own “primary”reference materialsLevel: 850 nmol/mol (nominal)


Long-term stability studyStability study

600

650

700

750

800

850

900

950

1000

0 100 200 300 400 500 600 700 800 900

time (days)

amou

nt-o

f-su

bsta

nce

frac

tion

H2S

(n

mol

/mol

)


Trend analysis Original data:Label x u(x) y u(y) U(x) U(y)Data[0] 7 1 851 27 2 54Data[1] 70 1 794 28 2 56Data[2] 111 1 783 27 2 54Data[3] 118 1 826 28 2 56Data[4] 140 1 806 29 2 58Data[5] 241 1 803 29 2 58Data[6] 265 1 795 35 2 70Data[7] 345 1 826 20 2 40Data[8] 545 1 803 27 2 54Data[9] 661 1 860 29 2 58Data[10] 697 1 736 22 2 44Data[11] 832 1 756 23 2 46

Model equation y = a[0] + a[1]*xRegression coefficients:a[0] 8.25E+02 1.66E+01a[1] -6.69E-02 3.59E-02

Full covariance matrixa[0] 2.74E+02 -4.77E-01a[1] -4.77E-01 1.29E-03


Step 1: Assessment of slope

Test, whether

Ratio equals 0,932 ⇒ no significant slope

( ) 11

1 <⋅ auka

Model equation y = a[0] + a[1]*xRegression coefficients:a[0] 8.25E+02 1.66E+01a[1] -6.69E-02 3.59E-02


Step 2: Uncertainty evaluationStandard deviation of measurements

is to be compared with standard uncertainty of a measurement

Hence

Conclusion: mixture is stable within k*ults

5.35=s

28=measu

8.2122 =−= measlts usu nmol/mol

nmol/mol

nmol/mol


Shelf life

Assignment on basis of – Policy– Experience– Statistics

… but always subject to proof by means of continued stability testing/monitoring


Shelf life (continued)

0 20 40 60 80 100 120 140

A

B

C

D

Stud

y

Time (months)

Logistics Measurement Shelf life


Shelf life (mathematical approach)From uncertainty of slope

Approach relates results of stability study to shelf lifeUncertainty increases with shelf lifeNo guarantee that “predictions” are better than by other means

( ) ( )10 auttxu shelflts ⋅⋅==


Stability monitoringPeriodic measurement of CRM– Requires method with good reproducibility – Metrological traceability should be derived

in a different stepEvaluation– Evaluate uncertainty of measurement – Check condition

– If satisfied, material is (still) stable

22measCRMmeasCRM uukxx +≤−

1

Department I "Analytical Chemistry; Reference Materials"

Wolfram BremserFederal Institute for Materials Research and Testing (BAM), Unter den Eichen 87, 12205 BerlinGermany

Investigating stability for (C)RM expiry date determination

Topics:

• The BAM discussion paper• The ISO Guide 35 procedure • The model-based approach• Conclusions


The BAM discussion paper (1)

Basic assumption:

CRM production cannot be restricted to (absolutely or seemingly) stable materials only.

Controlled instability must be acceptable.

Control is executed through multilevel accelerated ageing and expiry date estimation.



Content:

- proposing stability classes (criteria to be defined)

- scrutinising the ISO Guide 35 approach (applicable to insignificant instabili-ties only)

- proposing tools for handling detected instability (significant or insignifi-cant)



Key word:

First handle detected instability.

Second think about possible uncertainty arising from the above handling (and where and when to include it).


The ISO Guide 35 approach (1)

long-term stability study

short-term stability study

main factor to be tested: temperature


The ISO Guide 35 approach (2)

linear regression

significance test

neglecting cov

neglecting slope

choosing expiry date

2


Problems in the ISO Guide 35 approach

What to do if

? the regression turns out to be significant

? a still insignificant (but observable) trend continues

Finally,

? what is the uncertainty of the slope

? does it cover observable trends

No answer, no handling recommendations !

0.9

0.94

0.98

1.02

0 5 10 15 20 25

time (months)

P> = 0.051

0.9

0.94

0.98

1.02

0 5 10 15 20 25

time (months)

P> = 0.0004


The sources contributing to u(β)

What is u(β)? It is fed by residual scatter:

Consequence:1. The sought-after effect is neglected.2. The residual scatter is retained.3. This is the opposite as in homogeneity testing and obviously an inconsistency.


The model-based approach (time dependence)

-0.2

-0.1

0

0.1

0.2

0 2 4 6 8 10 12

time in months

T = 20 °C

-0.1

-0.05

0

0.05

0.1

0 0.5 1 1.5 2

time in months

T = 70 °C

keff accounts for possible concurring processes


The model-based approach (temperature dependence)

keff determined for several Ticoncurring processes? -> check against model

model: Arrhenius

-12

-9

-6

-3

0

0.0029 0.0031 0.0033 0.0035 0.0037 0.0039inverse temperature [1/K]

acrylamide in crisp bread

ΔE = 60.7 kJ/mol

average for many orga-nic reactions 83 kJ/mol


The model-based approach (full scheme)

minimum layout model: Arrhenius

First step: regression of ln [c(t, T)] over time => k(T) at the nodes

Second step: regression of ln(K) over 1/T => activation energy Ea

Third step: calculation of texp at the desired storage temperature from Δc(texp , T) = Ucrm

The Tref = Tstor problem- no isochronous layout possible- if disregarded: loss of information on the reference temperature- ways out: fitting or inclusion of "other" data (e.g. from homogeneity study)

texp ~ 78 months

overestimation of texp


Conclusions

(C)RM production cannot be limited to stable materials only. "Con-trolled" instability is admissible provided reliable expiry dates are established.ISO Guide 35 recommendations disregard any materials with de-tected instability.

ISO Guide 35 approach to "seemingly" stable materials is insuffi-cient, namely i) it may fail to cover degradation, ii) it retains the wrong part of the decomposed overall uncertainty, iii) it disregards "non-linearities".

Model-based approaches provide deeper insight and more reliable expiry date estimates.

Although also a requirement of Guide 35, usts is not considered, and not included in any of the ERM budgets (so far).

Thank you for your attention!

3


Most common solutions (1)

Food industry = worst-case estimates based on long-term experience and product-specific tests

The potential risks of disregarding or even manipulating expiry dates (when criminal energy is involved) can be seen from the current "nauseous meat" scandal.

Pharmaceutical industry = regulated test procedures (ICH protocols)


Most common solutions (2)


Influential factors and models

Process Modeltemperature-driven chemical Arrhenius equation

decomposition

temperature-controlled activity O´Neill equation

(microbial, biological, etc.)

diffusion Fick´s law

(radioactive) decay exponential law

chemical reaction diverse, complex

(possibly with threshold)


Bits and bites on stability from Guide 35






ISO Guide 35 –Determination of Property Values


Measurement methods

fully validated (before use!)known performance characteristicsif feasible, full evaluation of measurement uncertaintyresults metrologically traceable to

“appropriate references”


Models for metrological traceability

SI unitsunit defined by a standard method; these RMs should be made traceable to a result obtained by strictly following this standard method;other measurement standards or artefacts, including CRMs and RMs


Control over the measurement process

sample weighing,purity of reagents, solvents, “pure materials”,calibration status of common laboratory equipment and glassware,interferences in the measurement signal,appropriate and validated statistical techniques for doing calculations (e.g. calibration curves, interpolations), andcontaminations, losses, flaws in the measurement process


Characterisation models

1. measurement by a single (primary) method in a single laboratory

2. measurement by two or more independent reference methods in one laboratory

3. measurement by a network of laboratories using one or more methods of demonstrable accuracy

4. a method-specific approach giving only method-specific assessed property values, using a network of laboratories


Collaborative study10 – 15 laboratories preferred – allows scrutinisation of data– allows use of outlier tests

At better control, number of labs may be smallerMetrological traceability– assessment of calibrations (including collaborators!)– control over measurement methods (validation of

results!)Agreement between results is necessary, but not sufficient (!)


Metrological traceability

Common reference point needed between– between-bottle homogeneity study– short-term stability study– long-term stability study– characterisation study

Calibration of studies must have common basis!!


Establishment of common reference point

Implementation through– common calibrant– sample calibration standards for all experiments

Comparability of laboratories is not enough– calibration uncertainty in testing often

negligible– comparability has considerable uncertainty!


charltsstsbbwb CRM

Uncertainty from calibrations

prop

erty

val

ue

results of experiments


Characterisation approaches

Property values from – preparation of the CRM (synthetic materials,

dilutions)– single measurement method– multiple measurement methods– collaborative studies

» multiple methods» method-specific approach


Gravimetric procedure of ISO 6142Standard for preparing gravimetric gas mixturesBasic model

∑∑

∑∑

=

=

=

=

⎪⎪⎭

⎪⎪⎬

⎫

⎪⎪⎩

⎪⎪⎨

⎧

⎪⎪⎭

⎪⎪⎬

⎫

⎪⎪⎩

⎪⎪⎨

⎧

==

p

jq

iiij

j

p

jq

iiij

jkj

mix

kk

Mx

m

Mx

mx

nny

1

1

1

1


CertificationCertification model (ISO 6142)

Verification of composition (ISO 6143)

If condition is met

nristabipurityigraviprepi xxxxx ,,,,, Δ+Δ+Δ+=

( ) ( ) ( ) ( ) ( ).,2

,2

,2

,2

,2

nristabipurityigraviprepi xuxuxuxuxu ΔΔΔ +++=

( ) ( ).2 ,2

,2

,, veriprepiveriprepi xuxuxx +≤−

( ) ( ) ( )veriprepiCRMi xuxuxu ,2

,2

, +=


Characterisation on the basis of a calibration curve (ISO 6143)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

0 0.2 0.4 0.6 0.8 1 1.2

amount-of-substance fraction (µmol/mol)

Resp

onse

(a.

u.)


Use of the calibration curve (ISO 6143)

Regression of x on y

Unknown (xunknown) can be calculated from calibration modelUncertainty is calculated by

( ) ( ) ( )

( )∑ ∑

∑−

= +=

=

⎟⎟⎠

⎞⎜⎜⎝

⎛

∂∂

⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

+⎟⎟⎠

⎞⎜⎜⎝

⎛

∂∂

+⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

=

1

0 1

0

2

2

22

2

,2m

i

m

ijji

ji

m

jj

j

bbubg

bg

bubgyu

ygxu

( ) 2210 ybybbygx ++==


Necessary sensitivity coefficientsWith respect to y

With respect to b

ybbyg

21 2+=∂∂

10

=∂∂bg

ybg=

∂∂

1

2

2

ybg=

∂∂


Regression resultsCalculation on an unknown

Reversed regression Covariance matrixb[0] -2.03E-03 1.36E-04 1.86E-08 -5.36E-08 2.68E-08b[1] 5.72E-01 4.11E-04 -5.36E-08 1.69E-07 -8.69E-08b[2] 8.57E-04 2.15E-04 2.68E-08 -8.69E-08 4.61E-08

x = G(y) = b[1] + b[2]*y + b[3]*y*yy(unkn.) 0.451351 0.000087

dG/dy 0.572912dG/db[0] 1dG/db[1] 0.451351dG/db[2] 0.203718


Results unknown mixture

x(unkn.) 0.256383

u(y) u(b[0]) u(b[1]) u(b[2]) u(b[0,1]) u(b[0,2]) u(b[1,2])var(x) 2.48E-09 1.86E-08 3.44E-08 1.91E-09 -4.84E-08 1.09E-08 -1.60E-08

var(x) 3.90E-09u(x) 0.000062 0.024%


Collaborative study

Multiple laboratories (p ≥ 8)Multiple methods/measurement techniquesUncertainty often established from scattering of resultsIdeally, laboratories should state uncertainties


Data requirements

Data should be– state-of-the art– accompanied by a complete uncertainty

statement– thoroughly checked by participants– validated by the statistician– free of outliers– including stragglers


Property valuesSingle method– Result

Multiple methods in a single laboratory– Result of one method, corroborated by

confirmation– (Weighted) mean of the two methods

Collaborative study– Mean– Median– Weighted mean






ISO Guide 35 –Data and Uncertainty Evaluation


Property valuesSingle method– Result

Multiple methods in a single laboratory– Result of one method, corroborated by

confirmation– (Weighted) mean of the two methods

Collaborative study– Mean– Median– Weighted mean


Example: arithmetic mean

Property value given by

Uncertainty given by (xi independent)

∑=

=p

iix

py

1

1

( )psyu =2


Example: weighted meanProperty value given by

Uncertainty given by (xi independent)

(derived from uncertainty propagation formula)

∑=

=p

iii xwy

1

11

=∑=

p

iiwwhere

( ) ( )∑=

=p

iii xuwyu

1

222


Example (data without uncertainty)

810

820

830

840

850

860

870

880

890

0 1 2 3 4 5 6 7 8 9 10

laboratory

resu

lt (a

.u.)


Example (data with uncertainty)

810

820

830

840

850

860

870

880

890

0 1 2 3 4 5 6 7 8 9 10

laboratory

resu

lt (a

.u.)


Example (data with small uncertainties)

810

820

830

840

850

860

870

880

890

0 1 2 3 4 5 6 7 8 9 10

laboratory

resu

lt (a

.u.)


Example (outliers)

750

800

850

900

950

1000

0 1 2 3 4 5 6 7 8 9 10

laboratory

resu

lt (a

.u.)


Outliers and stragglersTechnically invalid results– do not belong to the dataset;– even if they “agree”, they should be rejected

Outliers do not belong to dataset– technical reasons may be the cause– even if no technical reasons, outliers should be

rejectedStragglers should be kept in– avoids underestimation of uncertainty


Uncertainty of a CRM

Uncertainty of a CRM is given by

where uchar is the uncertainty from the characterisation measurements

22222charstsltsbbCRM uuuuu +++=


Expanded uncertainty

Definition [GUM, section 2.3.5]:

“Quantity defining an interval about the result of a measurement that may be expected to encompass a large fraction of the distribution of values that could reasonably be attributed to the measurand”



Notes:the fraction may be viewed as the level of confidence of the intervalsetting up a confidence level requires assumptions about the probability density function of the measurand



Relationship between standard uncertainty, expanded uncertainty, and coverage factor

Coverage factor depends on– assumed distribution of result– level of confidence

cukU ⋅=


Coverage factor

Definition [GUM, section 2.3.6]:

“Numerical factor used as a multiplier of the combined standard uncertainty in order to obtain an expanded uncertainty”


Reporting uncertainty

Uncertainty should be reported as expanded uncertainty (U), stating a value for the coverage factor (k)Uncertainty should be rounded to one (non-zero) digit, unless the first digit is 1 or 2






ISO Guide 35 –Certification


CertificationCertification model (ISO 6142)

Verification of composition (ISO 6143)

If condition is met

nristabipurityigraviprepi xxxxx ,,,,, Δ+Δ+Δ+=

( ) ( ) ( ) ( ) ( )nristabipurityigraviprepi xuxuxuxuxu ,2

,2

,2

,2

,2 Δ+Δ+Δ+=

( ) ( )veriprepiveriprepi xuxuxx ,2

,2

,, 2 +≤−

( ) ( ) ( )veriprepiCRMi xuxuxu ,2

,2

, +=


Indicative values

No formal statusCannot be used for method validation/bias assessment


Minimum information on certificate (1)

the properties of interesttheir valuestheir uncertaintiesa statement concerning metrological traceability of the property values


Minimum information on certificate (2)

general particulars of the certifying body taking responsibility a description of the materialits intended useexpiry date (period of validity) of the certificateinstructions for useappropriate storage conditions


MRA


Regional Groups (CCQM)


International traceability

COOMET

SADM

ET APMP

SIMEUROMET

CCQM


Martina Hedrich and Thomas Steiger

Federal Institute for Materials‘ Research and Testing (BAM)Unter den Eichen 87, D-12205 Berlin, Germany

Implementation of ISO Guidelines –The BAM Approach


The history of RM production at BAM, respectively at its precursors, traces back to 1912, when the Prussian Royal Materials Testing Office released a steel reference material (with certified element content)


Guidelines for reference materialsISO Guide 30: 1992 Terms and definitions used in connection with

reference materials ISO Guide 31: 2000 Reference materials – Contents of certificates

and labelsISO 6141: 2000 Gas analysis – Requirements for

certificates for calibration gases and gas mixtures

ISO Guide 32: 1997 Calibration in analytical chemistry and use of certified reference materials

ISO Guide 33: 2000 Uses of certified reference materialsISO Guide 34: 2000 General requirements for the competence of

RM producersILAC – G12: 2000 Guidelines for the requirements for the

competence of RM producers ISO Guide 35: 2006 Reference materials – General and statistical

principles for certificationBAM I Department of Analytical Chemistry, Reference Materials APLAC RMP Accreditation Training Course, Tsukuba/Japan 2007-06-01 4

Today BAM offers RMs for• Iron and steel products• Non ferrous metals and alloys• Special materials• Primary pure substances• Environmental measurements• Gas mixtures• Elastomeric materials• Optical properties• Porous materials• Layer and surface RMs• Polymer materials• Isotopic reference materials

about 300 different reference materials


European Reference Materials (ERM)

Considerable part of new BAM CRMs are marketed as ERM

IRMM, LGC and BAM have combined forces to produce a new standard in reference materials

European Reference Materials are certified materials, which undergo uncompromising peer evaluation and offer highest quality and reliability


BAM CRMs in CMC claims for amount of substance (QM)

CMC categories:• High purity substances (Cu, Fe, Pb, Ga, Si, Sn, …)

• Inorganic solutions (PTB/BAM CRMs)

• Gases (O2, CH4, CO2, … in nitrogen, methane, …)

• Metals, metal alloys (Pb alloy, Cu, steel, Al, …)

• Advanced materials (Sb in Si, porosimetry)

• Sediments, soils, ores, particulates (OCPs in soil, PAHs in soil, elements in ore, elements in silica bricks)


All RM related issues are addressed by two BAM committees:

Committee for Reference Materials (AK RM)

Certification Committee (ZertKom)

BAM’s specific Reference Materials Committees


Committee for Reference Materials (AK RM)responsible body for all general RM topics and policiesholds regular meetings (at least twice a year)develops BAM policies for homogeneity and stability studiesdevelops generic QM proceduresdiscusses labelling and marketing issues



Certification Committee (ZertKom)monitors all individual RM projectsmeets on demand (several times per year)detailed discussion of all new projects (including technical and application aspects)gives advice for RM project realisationcan be called for guidance at any stage of RM projectsdetailed consideration of any RM project after characterisation and assignment of property valuesrecommends CRM for approval (before the certificate is issued)



Demand and significance (regulatory relevance, cost, competence, …)Specification (matrix, property values to be certified)Feasibility assessment (starting material, measuring capability, collaboration)Production (preparation, packaging, homogeneity, stability, data evaluation)Project planning form (doc. RLB-3.5, A 3)Presentation to Certification CommitteeApproval

Planning


Homogeneity testing and evaluation of associated uncertainty contribution complies with recommendations of ISO Guide 35For ‘batch certification’ (e.g. most solid RMs) a designated (between packaging units) uncertainty contribution is mandatory (ubb)

Assessment of homogeneity


Procedure for stability testing complies with the recommendations of ISO Guide 35 and partially goes beyond these recommendationsEstimating expiring dates from stability study using a degradation model based approach rather than including an uncertainty contribution from (potential) instabilitiesStability monitoring on a regular basis until the CRM is sold out

Assessment of stability


Temporal change investigated:

extent of instabilityconditions for storage and shipping( provisional or final) expiry date

ults and usts are not distinguished

Instability


1) Stability evident - No stability test2) No significant instability detected –

stability test3) Significant instability, kinetics unknown:

stability test4) Significant instability of known kinetics:

certified values specified as a function of time

“Stability classes”


Complete agreement with ISO Guides 34 and 35In-house certification: single reference method, preferable a ‘primary’ method (a single BAM lab)

Example: Gas and isotopic RMsIn-house certification: two or more independent reference methods (several BAM labs)

Example: Environmental RMs, primary pure materialsInterlaboratory certification study: one or more BAM labs and external collaborators ( in general various methods, except for method specific RM )

Example: Ferrous and non ferrous metals and alloys

Characterisation


BAM does not develop RMs by subcontractingExternal cooperation for characterisation measurements (interlaboratory certification study) used for some kinds of RMs (e.g. metallic CRMs)Competence of collaborators assessed by:successful participation in previous certif. studiessuccessful participation in a dedicated qualification studypositive experience from long-lasting co-operationaccreditation to ISO/IEC 17025 for the analytical methods in question

Collaboration


Mainly provided via BAM reference material websites:

http://www.bam.de/en/fachthemen/referenzmaterialien/index.htm

General information on BAM RMs (overview, prices,webshop)Certificates and reportsExperts for all special kinds of RMsRM news

Post distribution service


Task group performed comprehensive CRM management review (work accomplished by midyear 2006)CRM related management processes handled by two committees (Committee for RM, Certification Committee)Guidelines for the Production of BAM Reference Materials (Version 20 June 2006), including RM project flow chart

Strong points


Flow Chart for BAM CRM Projects

Annex B of the

Guidelines for the Production of BAM Reference Materials(Version 20 June 2006)


BAM I Department of Analytical Chemistry, Reference Materials APLAC RMP Accreditation Training Course, Tsukuba/Japan 2007-06-01 21 BAM I Department of Analytical Chemistry, Reference Materials APLAC RMP Accreditation Training Course, Tsukuba/Japan 2007-06-01 22

BAM I Department of Analytical Chemistry, Reference Materials APLAC RMP Accreditation Training Course, Tsukuba/Japan 2007-06-01 23 BAM I Department of Analytical Chemistry, Reference Materials APLAC RMP Accreditation Training Course, Tsukuba/Japan 2007-06-01 24


Customer feedback presently performed with a uniform questionnaire for all BAM services (testing, RM, …) , but for RMs a specific questionnaire seems to be more appropriateAssessment of collaborators:Technical competence approvedInformation on collaborators’ quality management system up to now not enquired and compiled in a uniform manner. Issue to be discussed by BAM’sRM committee (drafting a specific questionnaire?)

Weak points - further improvement needed


Thank you for attention !


Most of the clauses of chapter 4 (management requirements) of ISO Guide 34 are covered by ISO/IEC 17025. In several cases the requirements are somewhat stricter or more specifically expressed with respect to RMsMajor differences between ISO Guide 34 and ISO/IEC 17025 are in chapter 5 (technical requirements), but they should not be overstressed. In some sense ISO Guide 34 is a “specification” of ISO/IEC 17025 for a special “application field”.

Differences between ISO Guide 34 and ISO/IEC 17025


Planning reference material projectsPreparation of the materialAchievement of required homogeneityAssessment of stabilityCharacterisation /assignment of property valuesCollaborationPackaging and storage of RMsPost distribution service /advisory service

Specific requirements of ISO Guide 34 not or only partially covered by ISO/IEC 17025

APLAC RMP Workshop, Tsukuba, Japan 2007

Annex V

Course Feedback

15 participants returned the feedback questionnaire 1. Logistical Arrangements Course Objectives Meals: Excellent: 9; Good: 6 Hotel Condition: Excellent: 8; Good: 7 Remarks: Rooms small but fine Meeting rooms and facilities: Excellent: 11; Good: 3; Not bad: 1 Overall appraisal: Excellent: 13; Good: 2 Remarks: IAJapan did a superb job providing logistical instructions, materials and hosting. Proximity of hotel to meeting room. Train and bus excellent. 2. General Comments on the Design of the Course General comments on the lectures, discussions and group exercises:

Excellent x 2 Very important and well prepared. Discussions put the finer touches on the accreditation infrastructure previously

agreed by members on accrediting RMP. Good x 2.


Projector clarify a little weak but fine/ok; size of room and content very good; could try to have 1 or 2 more group exercises.

Very good. A lot of information and knowledge in RMP obtained during this workshop.

Lecture, discussion and group exercise were good to learn and share experience. Fruitful and useful. Very contribution and productive course. Good arrangement of the issue but for lack of times to present.

Do you think the duration of this course (3 days) is appropriate and adequate. If not, how long would you suggest?

Yes x 9. No. Four or five days x 2. Excellent for 3 days. Time to raise issues, struggles and reach agreement to

decide at least how to move forward. May need more time, e.g. 5 days x 2. Essential introduction RM/RMP.

Do you think the topics covered in this course are appropriate and adequate? If not, what topics are missing or not necessary?

Content of scopes is important and a sessions should be devoted to it. Yes x 8. Hoped to have more discussion about how we apply G34 with 17025 in detail. Fine. Need to briefly review ISO Guide 31. Need more discussion with ISO Guide 34 especially identifying key criteria. As a new comer, interpretation of ISO Guide 34 is needed for some clauses. The topics in ISO Guide 35 is too less especially for statistical estimation.

3. Improvement Suggestions

More study of cases, including some important steps of the assessment. Economic aspects of accreditation of an RMP, specially for developing countries. An APLAC RMP course for ISO Guide 34 technical assessors and an RMP

17011 evaluator course, where possible.


Establish an APLAC list of RMP technical assessors for the various industries. More exercises may be included. Have a half-day technical visit to RMP.

4. Benefits for You and Further Training Needs Do you think this course was beneficial to you and your organisation?

Yes. The class exercises and experiences were very useful in underpinning the presentations.

Of course. I am sure. Yes. The contributions by the participants has been a good effort in streamlining

the RMP assessment process. Yes x 7. Yes. Absolutely, crucial, valuable. Very much. Yes. Very beneficial to me and my organisation for the purpose of having RMP

accreditation program. Yes. Different view to know RMP, especially in microbiology, not focus on

Chemistry always. For your organisation, do you think there is a need for APLAC to provide further training courses on RMP accreditation?

Due to complexity of the subject and numerous possible arrangements it would be helpful to have sessions to provide interactions between ABs rather than work in isolation. Not necessary to have courses that repeat information that has already been covered.

Yes x 5. Maybe, workshops discussing this topic: accreditation of reference materials. Maybe in specific industry sectors, e.g. production of gas CRM/RM, biological

CRMS/RM. Yes, every year or two. Yes. Agree with suggestion that APLAC organise a visit to an RMP facility,

especially representative of AB which hasn’t an RMP accreditation program yet. Yes. To allow more participants (not only one) from each AB. Yes. Specific training for assessment of RMP.


What topics do you think should be covered?

CRMs vs RMs – Traceability; experiences with biological RMs; experiences with physical RMs.

To identify the harmonised criteria for G35. Costs of accreditation of reference material producers; prospective and economic

studies concerning RMP. Biological CRMs/RM. Case study: how should we apply G35 with 17025 in practice? Examples;

harmonisation of standards/practices; certificate and scope formats; CRM vs RM distinctions.

Maybe a visit of RMP (preferably accredited); area other than chemical may be included.

Application/implementation of ISO Guide 34; relevant clause of 17025 in assessment of RMP accreditation; sharing of knowledge/experience of assessment accreditation of RMP.

ISO Guide 31. Any related topics. RMP visit. ISO Guide 34 and 31. Sharing assessment experiences among assessor from different ABs. Topics in ISO G32, 33 which related may mention. About the assessment for the RMP including experience.

What steps should APLAC take to facilitate the harmonisation of RMP accreditation practices?

If additional training or workshops are not planned, then this groups should meet in a forum either annually or biannually (perhaps biennial) to discuss situations and share experiences so as to continue the harmonisation.

See previous comments (course beneficial; need for further training; identify the harmonised criteria for G35.

Promoting workshops about this topic; interaction with different RMOs to disseminate this topic.

Making the relevant parts of TC 008 mandatory could improve the harmonisation process.

Keep doing what they are doing; interact with ILAC, REMCO, etc; training update workshops.


Harmonise the accreditation criteria (ISO G34, 17025 etc.) and its implementation; conduct more course/workshops related to RMP.

Revise APLAC TC 008 to accommodate the outcomes of Tsukuba workshop. For the first time, potential RMP to be invited for training on implementation of

system based on the ISO Guide 34 and 17025. On-site observation to the accredited RMPs.

Presentation slides of Dr. Martina Hedrich

Documents

Transcript of Presentation slides of Dr. Martina Hedrich