MEASUREMENT OF UNCERTAINTY IN ... - NMKL - · PDF fileMeasurement of uncertainty in...

Measurement of uncertainty in quantitative microbiological examination of foods Page: 1 of 33 Version: 4

NMKL PROCEDURE No. 8, 4. Ed. (2008)

Date: September 2008 Approved: Ole Bjørn Jensen

MEASUREMENT OF UNCERTAINTY IN

QUANTITATIVE MICROBIOLOGICAL

EXAMINATION OF FOODS

CONTENTS 1. PREFACE

2. INTRODUCTION

3. DEFINITIONS

4. GENERAL CONSIDERATIONS

5. DESIGN FOR ESTIMATION OF MEASUREMENT UNCERTAINTY

6. ESTIMATION OF MEASUREMENT UNCERTAINTY

7. CONTROL OF THE ESTIMATED MEASUREMENT UNCERTAINTY

8. LITERATURE ANNEX: Comparison of estimations of measurement uncertainty based on different statistical models.

Nordic Committee on Food Analysis




1. PREFACE

The first NMKL Procedure on estimation of uncertainty for microbiological analyses was published already

in 1999. Professor Eystein Skjerve, at the Norwegian School of Veterinary Science was project leader and

arranged courses in the use of the procedure. Skjerve revised the procedure, and the 2. Ed. was published in

2002. In connection with quality assurance and accreditation of microbiological methods, according to EN

ISO /IEC 17025 (1), a calculated value for the uncertainty shall be given in connection with an analytical

result. In cases where it is not possible to estimate a value for the uncertainty, the laboratories shall be able

to identify sources to the uncertainty and provide a reasonable estimate of the proportion of the various con-

tributions.

One technique for estimating the uncertainty is to divide the method in question into its individual working

operations or steps. This can provide the analysts useful information about where the huge sources of error

are to be found and hence give better knowledge to reduce these, if necessary. A general procedure is de-

scribed in GUM (2). For microbiological methods this is described in Niemelä (4). This NMKL Procedure

does not describe estimation of measurement uncertainty for each individual step, however, estimates the

uncertainty based on a total/global assessment. In the previous editions of the NMKL Procedure No 8 the

measurement uncertainty is estimated from the extra variance in addition to the Poisson variance. The Pois-

son variance is the variance connected to the bacteria counting. The extra variance represents the variance in

connection with weighing, homogenisation and pipetting. This edition, which corresponds more with other

guidelines for measurement uncertainty [NMKL Procedure No 5, GUM, Norwegian Accreditation, Sector

Committee P9’s document, NA-S53, ISO / TS 19036] uses the standard deviation calculated from repeat-

ability and reproducibility data.

This 4th version of the NMKL Procedure replaces version 3, released in May 2008, which was only available

a few months. In version 3, the between-series variance (i.e., the variation between analysts, day-to-day and

batch to batch) was left out. Instead of issuing an addendum to Chapter 6, version 4 is released. In the esti-

mation of measurement uncertainty in NMKL Procedure No. 8, the method’s bias (uncertainty in relation to

the true value) it is not considered. It is informed, however, about the use of certified reference materials and

participation in proficiency testing schemes in order to check that the estimated measurement uncertainty is

satisfactory.




The revision of this procedure has been carried out by the following individuals:

Denmark: Erik Dahm, Danish Veterinary and Food Administration, Region North

Iceland: Snorre þórisson Rannsóknaþjónustan Sýni ehf/SYNY Laboratory service

Norway: Marianne Økland, National Veterinary Institute

Sweden: Lennart Larsson, LaVet

NMKL Secretary General: Hilde Skaar Norli, National Veterinary Institute (project leader)

A special thank is given to the Section of Feed and Food Microbiology at the National Veterinary Institute

for providing analytical data and to Joakim Engman, National Food Administration, Sweden for vital contri-

butions to this procedure.

The procedure is available from:

The office of NMKL Secretary General, c/o National Veterinary Institute,

P.Box 750 Sentrum,

N-0106 Oslo, Norway

E-mail: [email protected], Web page: http://www.nmkl.org.

NMKL welcomes any comments and input on the procedure. Comments are requested to be forwarded to

the office of NMKL Secretary General.

©NMKL




2. INTRODUCTION

All quantitative determinations are burdened with measurement uncertainty. The term measurement uncer-

tainty must not be mixed up with error. Error is the difference between measured and true value, while the

measurement uncertainty is an expression for the spread, and is a quantitative expression for the quality of

an analytical result.

The “true” number of microorganisms in a sample is not usually known. Microorganisms can be changed,

multiply or die out in a sample or diluent. In qualitative and quantitative examinations of foods for the oc-

currence of specific microorganisms or groups of bacteria (“heterotrophic bacteria”, “coliform bacteria”),

there are several factors contributing to spread of the results, and to the uncertainty of the obtained analytical

results. These factors must be borne in mind when considering the results. It is important that the measure-

ment uncertainty stated is realistic and that the confidence intervals of the precision are not made too nar-

row.

Lately several certified reference materials are made available at the market, where the levels of microorgan-

isms are given. Certified reference materials and materials used in proficiency testing schemes / ring trials

are useful for estimating the measurement uncertainty of the laboratory. Further participation in proficiency

testing and use of control charts are important to control that the obtained measurement uncertainty of the

laboratory is relevant and satisfactory.

3. DEFINITIONS The definitions are collected from NMKL Procedure No 5 (3), NMKL Procedure No 4 (6), NMKL Proce-

dure No 18 (5) and VIM (7).

Control charts

A control chart is a diagram where the results of analyses from known control samples are continuously

filled in. The control chart is connected to a certain method and matrix (5).

Error (of measurement)

Measured value minus a reference value (7).




Measurement uncertainty

Parameter, which is associated with the result of a measurement, and which characterizes the

dispersion of the values that could reasonably be attributed to the measurand (3).

Notes

1. The parameter can, for instance, be a standard deviation (or a given multiple thereof), or half-width of

an interval having a stated level of confidence.

2. Measurement uncertainty generally includes many components, which are characterized by their stan-

dard deviations estimated from a series of measurements, or from assumed probability distributions

based on experience or other information.

3. It is presumed that the measurement result is the best estimate of the value of the measurand, and that

all elements of uncertainty, including those, which depend on systematic effects (e.g. those which are

connected with corrections and reference standards), contribute to the dispersion.

Repeatability

Repeatability means that the analysis results are obtained using the analytical method on identical samples in

the same laboratory, using the same equipment within a short period of time (6).

Reproducibility

Reproducibility means that the results are obtained by using the analytical method on identical samples in

different laboratories and using different equipment (6).

Internal reproducibility means that the determination is carried out at different times, by different persons

and on different batches of reagents – but in the same laboratory (6).

Precision

Precision is the degree of agreement between independent analysis results obtained under specific circum-

stances (6).

Reference materials

Reference materials contain a specified or agreed amount of certain microorganisms or a mixture of micro-

organisms (5).




Certified reference material (CRM)

Reference material, accompanied by documentation issued by an authoritative body and providing one or

more specified property values with associated uncertainties and traceability, using valid procedures (7).

Standard deviation (SD)

The standard deviation is a measure for the spread around the mean in a collection of several results.

Comments: Expressed mathematically the standard deviation is the square root of the variance.

∑=

−−

=n

1i

2i )x(x

1n1SD

SD = standard deviation, xi = the values of the single results, x = the mean of the results, n = number of

measures.

Standard uncertainty, u(xi)

The uncertainty of a measurement result expressed as a standard deviation (3).

Combined standard uncertainty, u

Standard uncertainty of the result of a measurement when that result is obtained from the values of a number

of other quantities, equal to the positive square root of a sum of terms, the terms being variances or covari-

ances of these other quantities, weighed according to how the measurement result varies with these quantities

(3).

Expanded uncertainty, U

The combined standard uncertainty, uc, multiplied with a coverage factor, k. At a 95 % confidence level

(probability) k ≈ 2, at 99% confidence (probability) k ≈ 3.

U = k · u .




4. GENERAL CONSIDERATIONS 4.1 Traceability To be able to use results from different laboratories in a sensible manner, the results must be comparable.

When it comes to e.g. weighing, the balance must be calibrated against a traceably calibrated set of weights.

This set of weights has been calibrated against a national set of weights, which in turn is calibrated against

the international kilogram prototype. The optimal situation would be to make all measurements that trace-

able, by using calibrated equipment, certified reference material with suitable matrix and a collaborative

validated method.

4.2 Reference materials, proficiency testing

One of the main problems faced by microbiologists in dealing with internal method validation is the lack of

relevant reference materials. Fortunately, more reference materials have been produced lately and there are a

number of freeze-dried materials and matrix related samples, often in connection with proficiency testing /

ring trials. In practice, the real food samples are dealt with rather than freeze-dried materials. Analysing ma-

terial from a freeze-dried ampoule necessarily differs from analysis of a food sample involving a series of

additional steps such as sampling, primary dilution etc. Nevertheless, participation in proficiency tests are

vital in evaluation of the analytical competence of the laboratory at field of interest and to review if the es-

timated measurement uncertainty is adequate. The use of reference materials is described in NMKL Proce-

dure No 18 and hence will not be given much attention in this procedure.

4.3 Official analytical methods

Lately several of the official microbiological methods are validated collaboratively, i.e. they are tested out at

several laboratories on different matrices at different levels of the agent. The requests to the method organi-

sations for including validation data in the methods are increasing. Validation data from a collaborative

study of the method’s performance can be used for estimating the measurement uncertainty at any labora-

tory.




4.4 Large variations between results Most people working with microbiological analysis have learned that variations between results from analy-

ses of the same sample may be considerable. This large variation may be easily exemplified using plating

methods. Results may be obtained that most analysts would consider as improbable without any statistical

calculation, for example: widely deviating parallels, substantial deviation between dilutions, or surprising

combinations of results from a MPN analysis. The causes of these phenomena are often not apparent. It is

crucial for a laboratory to establish whether such variations are due to inaccuracies in laboratory procedures,

or whether such phenomena are simply to be expected when dealing with microorganisms.

When using a plating method, e.g. 88 and 68 colonies counted on two plates from a 10-3 - dilution, most ana-

lysts will view the agreement between the results to be satisfactory, and use them to estimate the «true»

sample level to be 78000/g (7.8·104/g). If repeated, the analysis is likely to yield a different result. If re-

peated on numerous occasions, the analysis will produce a certain dispersion of results. The characteristics

of this dispersion indicate to what extent any one-measurement result may be assumed to be close to the

«true» value. If the shape of the dispersion is unknown, we cannot estimate the uncertainty of single results.

Strictly speaking, the «true» value can only be determined by examining the entire lot, from which the sam-

ples were taken.

The variation observed if analyses are repeated may be due to a number of reasons. Firstly, a stochastic dis-

tribution will always exist with respect to the exact number of microorganisms present in the sample re-

ceived by a laboratory. Even when the sampling method is satisfactory, exactly similar values will never be

attained. The biological nature of the microorganisms is a further contributing factor to the variation. Fi-

nally, in carrying out an analysis, uncertainty and variation are being introduced through inaccuracies in the

method.

4.5 Microbial distribution in sample Ideally, each sample should be selected in such a way that it represents the entire product or lot to be tested.

This may prove to be quite difficult when faced with large lots and solid foods. Liquid foods such as water

and milk are products involving small particle sizes, and as such, the only products from which samples are

likely to be representative for the whole lot or product. Solid foods like meat, cheese etc. may only partly be

relied upon to provide representative samples. Sampling should be optimised to ensure that the final sample

is suitable for quantification of the real content of microorganisms of the product or lot concerned.




The microbial load in a single sample cannot always be expected to be evenly distributed. Microorganisms

may be more numerous on the surface than inside the sample, and different strains of microorganisms may

be active in different areas of the sample.

4.6 Homogenisation In order to obtain an even distribution of a food or matrix, the sample is usually homogenised. This applies

to both liquid and solid samples. The microorganisms in the final sample are more evenly distributed by

homogenising in a stomacher or other instrument. One should be aware, however, of the differences in effi-

ciency of such homogenisation procedures. Heterogeneous foods, those containing fatty particles or other

particles/substances, may also have heterogeneously distributed microorganisms even after homogenisation.

Microorganisms must be considered as "particles" and will, due to spherical effects and electrical charges

often bind to other particles in the sample. Standardising the homogenisation procedure may also prove dif-

ficult, subjective judgement often being used when considering what constitutes an acceptable level of ho-

mogenisation. Each laboratory should have established procedures for the homogenisation of each type of

food. For laboratories running large series on the same kinds of foods, these procedures may prove crucial

for the consistency between analytical results.

4.7 Single microbes or agglomerations? Various microorganisms may appear as single particles or agglomerations (clusters) of particles. Such ag-

glomerations are naturally formed by proliferation of microbes in foods. Microbiological analysis takes this

into account, results often being expressed as Colony Forming Units (CFU) and not in numbers of «mi-

crobes». A Colony Forming Unit can be an agglomeration of bacteria stemming from one or more microbes

originally transmitted, and which have subsequently divided to produce a large number of microorganisms.

A detailed description of the evaluation of colonies is given in NMKL Report No 5 (9).

When sampling, it must be assumed that these agglomerations were originally present in the sample and that

any growth has occurred from these agglomerations. In the further preparation of the sample, some of these

agglomerations are assumed to break up, with the microorganisms or the reduced agglomerations behaving

rather as single particles in a liquid phase. This assumption is the basis for the majority of theories on quan-

tification of microorganisms.




In some foods, growth is believed to occur on a limited scale only, with microorganisms more often being

actually found as single particles. Drinking water with a low content of particles and nutrient salts may serve

as a representative example.

Various microorganisms exhibit wide variations of behaviour when in a liquid phase. Some come close to

the single particle ideal, whereas others typically agglomerate, especially those producing capsules or mu-

cous substances, for example Pseudomonas spp. It is possible to minimise their capability for agglomerating

by adding surface-active compounds into dilutions and media.

4.8 Microbial growth and death Even when sampling takes place under controlled conditions and optimal homogenisation is ensured, the

problem of microbial growth remains. Many of the relevant organisms are able to proliferate in the product,

in the sample following sampling, and to a certain extent throughout the whole procedure until final inocula-

tion takes place.

4.9 The analysts - Procedural differences Differences in the way procedures are being followed may contribute to considerable uncertainty in the final

result. Despite these factors being under good control, some inconsistencies will occur during weighing, pi-

petting etc. The mechanical handling of a sample may also contribute to a gradual change in sample charac-

teristics. Homogenisation and the microbes' ability to agglomerate have been mentioned previously. Fol-

lowing homogenisation, various dilution steps are often used, depending on the parameters to be analysed.

Each step may be instrumental in breaking up existing microbial agglomerations, resulting in an increase of

CFU numbers. Even a process as simple as pipetting may be highly effective in breaking up bacterial ag-

glomerations due to the strong suction exerted by narrow pipette-tips. In addition, mechanical handling may

also injure to the microbes, resulting in reduced vitality, and substances present in the diluent may also re-

duce microbial viability. Inexperienced analytists may place too much faith in the standardisation of results

from microbiological analyses. It is essential to have established a realistic acceptable limit for variations in

results. With reference to quality assurance of methods, small inaccuracies in weighing/pipetting have rela-

tively little significance compared to the potentially large effects of physical treatment of microbe aggre-

gates or temperature deviations, which can allow growth or death. Weighing/pipetting inaccuracy may give

rise to percent error, but inaccuracy in the sample storage and/or incubation temperature or variations in the

routines for dilution can give rise to log10-unit errors.




4.10 Reliability of colony counting

Subjective evaluations will often play a role when counting colonies on an agar plate. Methods to evaluate

the correspondence between an analyst’s repeated counting of a plate and two analysts’ counting of the same

plate should be employed regularly in order to tighten up the criteria for estimating single colonies. A simple

method is published in an earlier NMKL document (9), where it is recommended that the relative variation

for one analyst should not exceed 7.7 % and for two analysts, 18.2%. Error location and optimising of meth-

ods should also include testing of the analysts’ ability to reproduce counts.

5. DESIGN FOR ESTIMATION OF MEASUREMENT UNCERTAINTY

For establishing an estimate of the measurement uncertainty of an analysis at a laboratory, the laboratory’s

internal reproducibility should be used. Internal reproducibility means that a standard deviation is calculated

from results obtained by repeated analyses at the same laboratory at different times, by different persons and

on different batches of reagents. The internal reproducibility of the laboratory will be the standard uncer-

tainty. If, for some reason, a laboratory should not have the capacity to make an internal reproducibility

study, an estimate of the measurement uncertainty may be established based on results from the method’s

performance study. This presupposes that the method is validated collaboratively, and also that the labora-

tory verifies the method’s performance with respect to the repeatability. The following chapters contain ex-

amples of the mentioned possibilities for estimating measurement uncertainty based on

1. the standard deviation for the internal reproducibility from experiments at own laboratory

2. the standard deviation for the reproducibility obtained from a collaborative method performance

study

Alternative 1 is recommended.

In collaborative validation studies within microbiology, robust statistics are usually applied when evaluating

the results. This means that the median is used instead of the mean, and the so called recursive median, Sn of

Rousseeuw, is used instead of the classical standard deviation (13). The median is commonly used in order

to give the so called ”random shots”, deviating results which appear now and then, less influence on the total

result. Robust statistics can be also used to estimate the measurement uncertainty. As most analysts have

more experience with the use of classical statistics, the mean and standard deviation are described in this

procedure. This can be justified as many results are used when establishing measurement uncertainty, and

therefore random shots are of less consequence.




Within chemistry (NMKL Procedure No. 5), the relative standard deviation is usually applied for estimating

the measurement uncertainty. The relative standard deviation means standard deviation (the spread of the

results) divided by the concentration/level. Within microbiology, use of relative standard deviation is not

relevant as the relation between level and standard deviation (calculated on log10 values) is approximately

constant.

The following figures show the relation between the concentration levels and standard deviations of results

from the analysis of 681 parallels (1362 analyses) of different microbes, levels and matrixes estimated in the

following 3 ways:

a) non-logged results, where the standard deviation is calculated from cfu/g

b) log10 -transformed results, where the standard deviation is log10-transformed after the calculations

are carried out

c) log10 transformed results, where the standard deviation is calculated from the log10- values of the re-

sults

a) Relation between level and standard deviation in cfu/g (i.e. non-logged data)

R2 = 0,5638

-5000

0

5000

10000

15000

20000

0 5000 10000 15000 20000 25000 30000 35000

level in cfu/g

stan

dard

dev

iatio

n in

cfu

/g




b) Relation between level and standard deviation when the results are log10 transformed after the calculations

R2 = 0,8051

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

5

0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5

level (log10 cfu/g)

stan

dard

dev

iatio

n (lo

g10

cfu

/g)

c) Relation between level/concentration and standard deviation (in log10 cfu/g)

R2 = 0,04490

0,2

0,4

0,6

0,8

1

1,2

1,4

0,00 0,50 1,00 1,50 2,00 2,50 3,00 3,50 4,00 4,50 5,00

level (log10 cfu/g)

stan

dard

dev

iatio

n (lo

g10

cfu

/g)

For Figure a) (original, non-logged data) there is an increasing spread of the standard deviation with increas-

ing levels. A trend line is included in the figure. However, the linear relation has to be rejected as the regres-

sion coefficient is low (R2 = 0.56). If the results are logged, after the calculations are carried out there is an

obvious linear relation between increasing level and standard deviation (figure b).

Quite often, standard deviations are calculated based on results in log10 cfu/g – the data are logged prior to

the calculations. Figure c), which illustrates results where the calculations are carried out on log10 values,




there is no increase in the standard deviation by increasing level, the relation is almost constant. This illus-

trates that it is important to know whether or not the results are logged-transformed prior to or after the cal-

culations. For the results from the analysis of the 681 parallels, 98% of the results are below 0.5 log10 cfu/g,

96% of the results are below 0.4 log10 cfu/g and 94% of the results are below 0.35 log10 cfu/g. Based on

these results (several 100s of parallels), the standard deviation between two parallels (the repeatability)

should be less than 0.4 log10 cfu/g (at 95% confidence). This can be used as a rule of thumb. However, the

standard deviation depends on the analysis (method, bacteria, materials), and, not least, on the analyst’s

technical skills.




6. ESTIMATION OF MEASUREMENT UNCERTAINTY

The measurement uncertainty is estimated from the standard deviation of internal reproducibility. For calcu-

lation of this standard deviation, analytical results obtained at reproducible conditions are used (see chapter

3 for definitions). The calculations are conducted on log10 transformed results. The results can be put in a

suitable table such as shown below:

Combinations/Series

Analysts;

Batch;

Day

A;1;1 B;2;2 C;3;3

Replicate log10 log10 log10

1 xa11-1 xb22-1 xc33-1

2 xa11-2 xb22-2 xc33-2

3 xa11-3 xb22-3 xc33-3

4 xa114 xb22-4 xc33-4

9 xa11-9 xb22-9 xc33-9

10 xa11-10 xb22-10 xc33-10

In the following, it is described step-by-step how to calculate internal reproducibility (precision within and

between series), and thereby an estimate for the measurement uncertainty.

Step 1: Calculate the precision (the repeatability) within the series / combinations

In order to get a measure of the precision within the series (A, B and C) each of the obtained preci-

sion of the series are combined. This is done by calculating the mean (average) and the standard de-

viation for each combination of analyst / batch / day (A, B and C), and then combining the obtained

standard deviations.

The mean is the sum of the results divided on the number of replicates.




Mean, x = 10

... 10113112111111 −−−−= +++=

∑ xxxxn

xn

ii

n = number of replicates (10 in this example)

Standard deviation, s, is calculated for each combination: 1

)(1

2111

−

∑ −=

−

= n

xxn

is

It is easy to do this in Excel. The results can be presented in a table:

Combination/Series A;1;1 B;2;2 C;3;3

Number of replicates n1 (10) n2 (10) n3 (10)

Mean, x x1 x2 x3

Standard deviation, s s1 s2 s3

3)1()1()1(

321

233

222

211

−++−+−+−

=nnn

snsnsnSr

The combined standard deviation of the precisions (the repeatability), Sr, to the series is calcu-

lated as follows:

If n1 = n2 = n3 (= 10 in this example), the formula becomes

3

23

22

21 sss

Sr++

=

( "3" because it is 3 series (A, B and C), if it were 6 series / combinations the denominator should

have been 6.)

Step 2: Calculate the precision between the series /the combinations

To examine whether there is a major difference in the results between the series A, B and C, calcu-

late the mean of the series’ mean values and the standard deviation thereof. Then use the obtained

standard deviation to calculate the between-series variance.

The mean of a series mean, y = 3

321 xxx ++

The standard deviation of the mean of the series’ mean, Sx = 3

)(1

2∑ −=

n

ii yx




The between-series variance is a measure of the spread of the results between the series, when the

repeatability of the series has been adjusted for.

nSSS r

xL

222 −= (Sr is calculated in step 1, n = number of replicates (the same as in step 1))

Step 3: Calculate the standard deviation of the internal reproducible, SR = standard uncertainty, u,

The standard uncertainty, u, is the standard deviation of the internal reproducibility, SR, and is calcu-

lated by combining the standard deviation of the precision for each of the series (A, B and C) (step

1) and standard deviation between the series, which represents the day-to-day, batch-to-batch and

analyst-to-analyst variations.

22LrR SSSu +==

Step 4: Estimate the measurement uncertainty, U.

In order to make the measurement uncertainty correspond to an interval containing a large

fraction of the expected variation in the results, the combined standard uncertainty is multi-

plied with a coverage factor. The coverage factor is usually 2, but can sometimes be 3. A

coverage factor of 2 corresponds to a confidence level of approx. 95%, and a coverage fac-

tor of 3 corresponds to a confidence level above 99%. It is recommended that a coverage

factor of 2 is used in the following manner: U = 2 · u

U, is half of the measurement uncertainty interval. In order to express the whole measurement uncertainty

interval for a measurement result, y (in log10 units), the following formula is usually used:

• y ± U,

• y log10 [y - U, y + U] or

• y cfu /g or ml [10y (log10)-U, 10 y(log10)+U] NB! not 10y ± 10U




6.1 Estimation of measurement uncertainty from internal reproducibility from analytical resuls of one matrix, one level and several analysts

For estimation of measurement uncertainty from internal reproducibility, each of the current analysts (the

ones that are or will be performing the analysis) analyse 10 replicates (10 plates) for each microorganisms in

question (or group of microorganisms) for each matrix group of interest at different times. If only one ana-

lyst performs the analysis routinely, the analyst has to analyse a set of 10 replicates (10 plates of the same

culture medium) and then repeat the experiment on a later a date.

A way of estimating the measurement uncertainty is to use a sample, preferably a reference material of real

matrix, which are repeatedly analysed by different assigned analysts at the laboratory. The analysts perform

analyses on different days, in order to also include the day-to-day variation and batch to batch variation in

the estimate of the measurement uncertainty.

Example 1

For establishing a measurement uncertainty for determination of aerobic count (NMKL Method No. 184),

the laboratory has used a material from a proficiency test (chocolate pudding mix). Five analysts each ana-

lysed 10 replicates on different days. The results are given in table 1.

Table 1: Results from analysis of aerobic count in chocolate pudding mix, 5 analysts, 10 parallels on dif-

ferent days

Series: Analysts, Batch, Day I II III IV V Replicates log10 cfu/g

1 3.67 3.79 3.61 3.78 3.80 2 3.66 3.76 3.72 3.84 3.77 3 3.72 3.85 3.59 3.73 3.74 4 3.85 3.86 3.68 3.49 3.00 5 3.70 3.90 3.54 3.57 3.96 6 4.02 3.60 3.90 3.51 4.05 7 3.87 3.86 3.96 3.77 3.81 8 3.90 3.89 3.87 3.71 4.08 9 3.74 3.82 4.01 3.80 3.93 10 3.45 3.86 3.86 3.70 3.71




Follow the steps 1 - 4 to establish an estimate for the measurement uncertainty.

Step 1: The precision within in the series

Calculate the mean and the standard deviation of each series. Then combine the obtained standard

deviations.

For the series 1, the mean and the standard deviation is calculated from the results in column 2 in

Table 1, as follows:

The mean of the series I: 76.310

45.3...72.366.367.31 =++++

==∑=

n

xx

n

ii

(log10 cfu/g)

The standard deviation of the series I:

16.09

)76.345.3(....)76.366.3()76.367.3( 222

1

)(1

2

=−++−+−

== −

∑ −=

n

xx

r

n

ii

s

(log10 cfu/g) Calculate the mean and the standard deviation in the same way for the other series in Table 1.

The mean values and the standard deviations of the results in Table 1, is given in Table 2.

Table 2: The mean and the standard deviation of the series I -V from Table 1.

Series: I II III IV V X (log10 cfu/g) 3.76 3.82 3.78 3.69 3.79 sr (log10 cfu/g) 0.16 0.09 0.17 0.12 0.30

The Series V has relatively considerable spread compared to the other series (sr is about twice the

value as for the others). The example is authentic and the analyst running series V is under training

and not authorised to perform the analysis. As the person is not allowed to do contracting analysis,

Series V is excluded in the estimation of the measurement uncertainty.

The combined standard deviation of the precision (the repeatability) Sr, of the series is:

14.04

12.017.009.016.04

222224

23

22

21 =

+++=

+++=

SrSrSrSrSr (log10 cfu/g)

Step 2: The precision between the series

To get a measure of the spread of the results between the series (the analysts, the days and the

batches) calculate the mean of the series means and its standard deviation. Thus, from this standard




deviation calculate the between-series variance.

The mean of the series’ I, II, II and IV mean values, y:

76.34

69.378.382.376.3=

+++=y (log10 cfu/g)

The standard deviation of y, Sx:

3)76.369.3()76.378.3()76.382.3()76.376.3(

1

)( 2222

4

1

2

−+−+−+−=

−

−=∑=

n

yxS i

i

x

= 0.054 (log10 cfu/g)

The standard deviation of the mean of the series mean values (Sx = 0.054 log10 cfu/g), is a measure

for the spread of the series’ mean values.

The between-series variance, SL2:

00096.01014.0054.0

22

222 =−=−=

nS

SS rxL (log10 cfu/g)

The spread of the results between the series is small, the between-series variation, SL, is low and ne-

glected when the standard uncertainty is calculated as the standard deviation of the internal repro-

ducibility.

Step 3: Calculate the standard deviation of the internal reproducible, SR = standard uncertainty, u

Internal reproducibility, SR, is the sum of the variance of the repeatability (from step 1) and variance

of the between-series (from step 2) is as follows:

14.000096.014.0 222 =+=+== LrR SSSu (log10 cfu/g)

Step 4: Estimate the measurement uncertainty, U

If the results are to be given as a 95% confidence interval, the combined uncertainty is multiplied by

a coverage factor of 2, k = 2. Thus, the expanded uncertainty is:

U = 2 · u = 2 · (± 0.14 log10 cfu/g) = ± 0.28 log10 cfu/g




Excel spreadsheet is indeed suitable to make the above calculations and in providing the results in a dia-

gram. The diagram below shows the results in a coordinator system, where the x-axis is the number of paral-

lels and the Y-axis is the results in log10 cfu/g. The bold lines at the top and at the bottom show the expanded

uncertainty, U ± 0.28 log10 cfu/g. The line in the middle shows the mean of all the results.

Aerobic bacteria

3,30

3,40

3,50

3,60

3,70

3,80

3,90

4,00

4,10

4,20

1 2 3 4 5 6 7 8 9 10

number of parallels

log 1

0 cf

u/g

Expression of the result: Example of expression of result: If the obtained analytical result is 6900 cfu/g = 3.84 log10 cfu/g for aerobic

count, the result should be given in one of the following ways:

• 3.84 log10 cfu/g ± 0,28 log10 cfu/g,

• 3.84 log10 cfu/g [3.56 – 4.12] or

• 6900 cfu/g [103.56, 104.12] = 6900 cfu/g [3600, 13200] (NB! not 103.84 ± 100.28 = 6900 ± 2)

• 3.6 ·103 – 1.3·104 cfu/g

6.2 Estimation of measurement uncertainty based on internal reproducibility from different matrixes and levels at the same time.

For the sake of convenience and resources, often one estimate for the measurement uncertainty is used for

different matrixes at different levels. The prerequisite is that the method is applicable for the matrixes and

levels of interest. In the following example it is shown how to establish one value for the measurement un-

certainty of for the determination of Staphylococcus aureus in different matrixes and levels.

Example 2: The laboratory uses NMKL Method No. 66, 4. Ed., 2003: ”Staphylococcus aureus. Determina-

tion in foods.” Two analysts are trained in the method. The matrixes analysed are cheese, lettuce and ham.




and ham, respectively. The materials are

omogenised. The results are given in table 3.

f 10 parallels of cheese, lettuce and ham.

eries /

The content of S. aureus in the samples is enumerated by surface streaking of decimal dilutions of a speci-

fied quantity of the sample on the Baird-Parker with Rabbit plasma fibrinogen agar (BP+RPF). After incu-

bation colonies with typical and atypical appearance are counted.

Both analysts analyse 10 parallels with samples of cheese, lettuce

h

Table 3. Two analysts’ results from analyses o

SAnalyst 1 2 Matrix Cheese Lettuce Ham Cheese Lettuce Ham Sample cfu/g Log10 cfu/g Log10 cfu/g Log10 cfu/g Log10 Cfu/g Log10 cfu/g Log10

1 6500 3.81 3500 3.54 7200 3.86 5800 3.76 3400 3.53 6200 3.79 2 6000 3.78 4000 3.60 7600 3.88 5400 3.73 2800 3.45 6500 3.81 3 5400 3.73 3000 3.48 7100 3.85 6200 3.79 3500 3.54 5900 3.77 4 5000 3.70 2800 3.45 6800 3.83 5500 3.74 2700 3.43 6100 3.79 5 5800 3.76 3600 3.56 7900 3.90 6200 3.79 2600 3.41 6300 3.80 6 6400 3.81 3800 3.58 7100 3.85 5300 3.72 3200 3.51 6000 3.78 7 6200 3.79 2700 3.43 7700 3.89 5100 3.71 3500 3.54 5800 3.76 8 6800 3.83 4300 3.63 7500 3.88 5900 3.77 3900 3.59 6400 3.81 9 6800 3.83 4500 3.65 7800 3.89 4900 3.69 3100 3.49 6300 3.80 10 5600 3.75 3400 3.53 6900 3.84 5700 3.76 3300 3.52 6400 3.81

Follow the steps o is stimate for the measurement uncertainty.

tep 1: The precision within in the series

1 - 4 t establ h an e S

Calculate the mean and the standard deviation of each series. Then combine the obtained standard

Cheese, the mean and the standard deviation are calculated as follows:

deviations.

For Series I,

Mean, x: 78.375.3...73.378.381.31 =++++

==10

∑=

n

xn

i

Standard deviation:

x i i.e. (log10) cfu/g

cfu/glog 9

7.3(....)78.378.3()78.381.3(10

22) 2 ++−+−=

− xi i.e. 044.0)78.35 2(1 =

−=

∑=

xn

is 1−nr

Corresponding calculations are carried out on log10-results of the other matrices for series I and II.

The results are given in table 4.




ard deviation of the results in table 3 Table 4: The mean and the stand

Series/ Analyst

I II

Matrix Cheese Lettuce Ham Cheese Lettuce Ham Mean log10 cfu/g 3.78 3.54 3.87 3.75 3.50 3.79 sr, log10 cfu/g 0.044 0.076 0.023 0.034 0.056 0.016

For estimating the standard deviation of the repeatability for all series, Sr, combine the sr of each ma-

trix and series (given in table 4) as follows:

042.06

016.0056.0034.0023.0076.0044.0 2222221

2

=+++++

==∑=

i

srS

n

ii

r (log10 cfu/g)

Step 2: The precision between the series

To get a measure of the spread of the results between the series (the analysts, the days and the

batches) calculate the mean value of the series means and its standard deviation for each type of ma-

trix (from table 4) as given in table 5.

Table 5: The mean and standard deviation between the series:

Matrix Cheese Lettuce Ham

Mean, 2

21 xx +

(log10 cfu/g)

(3.78+3.75)/2 = 3.76

(3.50 + 3.52)/2 = 3.52

(3.87+3.79)/2 = 3.83

Standard deviation,

s 1n

)x(xn

1i

2i

−

∑ −==

(log10 cfu/g)

0.0231

3.76)(3.753.76)(3.78 22

=−+−

0.031

0.053

Then combine the standard deviations of the series means in order to get a measure for the spread of

their mean value:

035.03

053.0031.0023.0 2221

2

=++

==∑=

i

sS

n

ii

x (log10 cfu/g)

Then calculate the between-series variance, SL2, in order to get a measure for the spread of the re-

sults between the series:




033.010042.0035.0

1022 =−=−= r

xLS

SS (log22

Step 3: C the stand tion o rnal rep ble, S

10 cfu/g)

alculate ard devia f the inte roduci R = unce standard rtainty, u

I ducibil , is the su repeata from step 1) and varian

o serie step 2) is as follows:

nternal repro ity, SR m of the variance of the bility ( ce

f the between- s (from

054.0033.0042.0 2222 =+=+== LrR SSSu (log10 cfu/g)

Step 4: Estimate the measurement uncertainty, U

, U:

the result:

The standard uncertainty is 0.054 log10 cfu/g , and the expanded uncertainty

U = 2 · u = 2 · (± 0.054log10 cfu/g) = ± 0.11log10 cfu/g

Expression of

either as:

If it is preferable to express the result in of log cfu/g, the result is transformed, 103.74 cfu/g =

5495 cfu/ ed

as:

cfu/g [103.74 0 cfu/g [4300, 73 – 7.1·103 cfu/g.

6.3 Estimation of measurement uncertain based on results from collaborative

everal official methods are collaboratively validated, i.e. the methods are tested out on several laboratories

at the

descr ied method when analysing the forwarded samples. The samples are often pre-

fferent levels, i.e. two and two samples are alike, but

that do n ative validation of NMKL Method

No 136 for estimation of the measurement uncertainty for Listeria monocytogenes in foods are shown in

the e

plate pecific isolation medium, ALOA or LMBA or Chromogenic Listeria

If the obtained result is 3.74 log cfu/g, the result can be expressed 10

• 3.74 log10 cfu/g ± 0.11 log10 cfu/g

• 3.74 log10 cfu/g [3.63 – 3.85]

cfu/g in stead 10

g, which rounded to nearest 100 would be 5500 = 5.5 · 103 cfu/g. Then the result can be express

♦

♦ 4.3·10

5500 – 0.11, 103.74 + 0.11] = 550 100]

tymethod validation.

S

same time to check the method’s performance. The participating laboratories follow carefully the

iption of the specif

sented as blind duplicates of different matrixes at di

ot the participants know. How to use results from a collabor

xample below. In the enumeration procedure the initial suspension and/or its dilutions are surface

d on a L. monocytogenes s




presumptive L. monocytogenes colonies

rphological and biochemical tests.

er from

the q

have on and ham (food samples). Table 5 is an extraction

of th 136.

able 5: Extract from table 2, annex 1 of NMKL Method No. 136. Number of participating laboratories and

analyses of cheese, salmon and ham by ALOA.

Matrix

Agar with equal operating principle as ALOA. After incubation

are counted and confirmed using appropriate mo

The collaborative validation showed that there were no significant statistical differences in the results eith

ualitative or the quantitative part when using the different selective media. In the following example we

chosen to use the results on ALOA for cheese, salm

e results of the collaborative validation study of NMKL Method No

Tthe standard deviation of the reproducibility of the

ALOA

No of labs 14 14 12 13

heese

7 0.31 0.15 0.24C sR 0.1

No of labs 14 14 12 13

Salmon sR 0.39 0.24 0.24 0.28

No of labs 14 14 13 14

Ham sR 0.19 0.31 0.17 0.15sR: standard deviation of reproducibility

The estimation of the uncertainty, u, is calculated as follows:

270.0141213131211131312111313 +++++++++++

bined standard uncertain

15.01317.012.......24.01339.01324.01215.01131.01371.013

)1(

(

22222222

=⋅+⋅++⋅+⋅+⋅+⋅+⋅+⋅

−

−=

∑∑

i

i

n

nu

The com ty, u = 0.27 log10 cfu/g.

0.5 d

fro ra-

tor

me ot

po l

pa re-

)1=isR 2

The expanded uncertainty, U = 2 · u = 0.54 log10 cfu/g.

4 log10 cfu/g is a rather high, but realistic estimation of measurement uncertainty, when it is estimate

m a method performance study. The high uncertainty is due to the fact that the spread from all the labo

ies are included in the estimate. When using results from collaborative studies to estimate the laboratory’s

asurement uncertainty, the laboratory has to check that their own obtained values for the precision is n

orer than the results obtained by the participants in the study. This can be performed by analysing severa

rallels in order to check that the repeatability obtained at own laboratory lies within the levels for the

peatability (not the reproducibility) in the method validation and/or by participation in proficiency testing

schemes, which is discussed in chapter 6. Maybe the laboratory will conclude that it may pay to make an

Measurement of uncertainty in quantitative microbiological examination of foods Page: 26 of 33

NMKL PROCEDURE Version: 4 Date: September 2008

No. 8, 4. Ed. (2008)

Approved: Ole Bjørn Jensen

eff than using validation

ate from a collaborative study.

Ex

ort to estimate the measurement uncertainty based on internal reproducibility rather

d

pression of result:

As

ing

3.0 log10 cfu/g ± 0.54 log10 cfu/g

it is preferable to express the result in cfu/g in stead of log10 cfu/g, the result is transformed, 103 cfu/g =

/ The result can the be expressed as

1000 cfu/g [10 103.00 + 0.54] = 1000 cfu/g [288, 3467] = 1000 cfu/g [290, 3500]

02 5· u

sume the obtained result is 1000 cfu/g = 3.0 log cfu/g. The result can be expressed in one of the follow-

options: 10

3.0 log10 cfu/g [2.46– 3.54]

If

3000 cfu g.3.00 – 0.54,

2.9·1 – 3. 1 f02 c /g



No. 8, 4. Ed. (2008)


E ESTIMATED MEASUREMENT UNCERTAINTY

t possible to establish an estimate of the measurement uncertainty for forever all with-

uncertainty is not over or under estimated. The control of the measurement uncertainty can

simplest be p

use

7.1 Use of control charts - co l of

Use of control charts, as described in NMKL Procedure No. 18: (”The use of reference materials, reference

rains and control charts in a food microbiological laboratory”) is a useful tool for maintaining the quality

on the analyses and to check if the estimated measurement uncertainty is realistic.

A control chart is a diagram where the results of analyses from known control samples are continuously

filled in. The control chart is connected to a certain method and matrix. On the control chart the mean, warn-

ing and action limits are marked. The warning limits corresponds to the mean ± the measurement uncer-

tainty with a coverage factor, k = 2, and the action limit corresponds to the measurement uncertainty with a

coverage factor of 3.

Upper and lower warning limits: Mean value ± 2 · u

Upper and lower action limits: Mean value ± 3 · u

Factor 2 and 3 correspond to 95% and 99% significance.

Example of such a control chart is elaborated for aerobic bacteria in chocolate pudding mix from example 5.2.

7. CONTROL OF TH

Unfortunately, it is no

out any form of further follow-ups. As every thing else one have to check that the obtained estimate is real-

istic, that the

erformed by

of control charts, and

participation in proficiency testing schemes

ntro measurement uncertainty

st



No. 8, 4. Ed. (2008)


Control chart

3,8

4

4,2

4,4

g

2,80 2 4 6 8 10 12

time

3

3,2

3,4log

c 3,6fu/

g10 cfu/g, upper warning limit: 4.04 log10 cfu/g; lower warning limit: 3.48 log10 cfu/g

st V). Analysis of the control sample should be run frequently and be plotted on the of the control sample lies within the warning limits, the measurement uncertainty

essing the result is ok. If the result is outside both the warning and the action limit the labora-measurement uncertainty should be expanded. Is

om th hould be considered whether the estimated meas- useful to evaluate any general increasing tenden-

ies of the results. This should be considered even if the obtained results are within the warning limits. An

mean

action limit warning limit

action limit warning limit

Mean: 3.76 lo upper action limit: 4,18 log10 cfu/g, lower action limit: 3.34 log10 cfu/g,

Here are also the results of the analyst under training included (Analyst V table 1). In one case the result is below the lower action limit and at two occasions the result is above the upper warning limit (all 3 results are performed by Analycontrol chart. If the result used for exprtory has to follow up and evaluate whether the area of however the results always far fr e warning limits, it surement uncertainty is too big. The control charts are alsocassessment of such a tendency can prevent sudden results above the warning limit.



No. 8, 4. Ed. (2008)


e .

rement uncer-tainty should be expanded correspondingly. If it is far outside the limits, the laboratory has to check for cal-culation error, sample mistake/problems, reagent, method and / or the routine of the labor In many of the proficiency testing schemes the z-score of the laboratory is given, which is the difference between the result obtained by the laboratory and ”true value” (the median or mean of all the results) di-vided on the standard deviation to all participants. Usually the results are considered satisfactory when the z-score of the laboratory is less than ± 2, which corresponds to upper and lower warning level. Thus the profi-

ency testing results of the laboratory can be plotted in a control chart for each analysis where the lower and

7.2 Participation in proficiency testing schemes – control of the measurement uncertainty

The established value for the measurement uncertainty can also be controlled by participation in proficiency testing schemes. If the laboratory’s obtained results ± the measurement uncertainty includes the mean valuof the scheme (or ”the true value”), the measurement uncertainty estimated by the laboratory is satisfactoryHowever, if the “true value” is just outside the confidence interval, the interval for the measu

atory.

ciupper warning limit is 2 z-scores and the action limits are ± 3 z-scores.



No. 8, 4. Ed. (2008)


8. LITERATURE

ducing the concept of uncertainty of measurement in testing in association

. NMKL Procedure No 4, 2. version 2005: Validation of chemical analytical methods.

7. VIM (2007) International Vocabulary of basic and general standard terms in Metrology. 2007, ISO

Guide 99, 3rd Ed., ISO Geneva 2007.

8. NMKL Procedure No 12, 2002: Guide on sampling for analysis of foods.

9. NMKL rapport nr 5, 2. utg., 1995: Handledning i kvalitetssäkring - för mikrobiologiska laboratorier.

(Available in Swedish and Finnish only)

10. NMKL rapport nr 1, 1983: Statistical evaluation of results from quantitative microbiological exami-

nation.

11. Norsk Akkreditering, Sektorkomité P9 sitt dokument, NA-S53: Måleusikkerhet ved mikrobiologiske

analyser. (Available in Norwegian only)

1. ILAC-G17 (2002): Intro

with the application of the standard ISO/IEC 17025. See www.ilac.org.

2. GUM (1995): Guide to the expression of uncertainty in measurement (1995), ISBN 92-67-10188-9.

3. NMKL Procedure No 5, 2. Ed. 2003: Estimation and expression of measurement uncertainty of

chemical analysis.

4. Niemelä (2003): Uncertainty of quantitative determinations derived by cultivation of microorgan-

isms. MIKES Publication J4/2003.

5. NMKL Procedure No 18, 2006: The use of reference materials, reference strains and control charts

in a food microbiological laboratory.

6



No. 8, 4. Ed. (2008)


gy of food and animal feeding stuffs -- Guidelines for the estimation of

measurement uncertainty for quantitative determinations.

3. or the validation of alterna-

tive methods.

12. ISO/TS 19036 Microbiolo

1 ISO 16140: Microbiology of food and animal feeding stuffs -- Protocol f



No. 8, 4. Ed. (2008)


Comp uncertainty based on dif-erent statistical models.

In the previous editions of NMKL Procedure No. 8, the Poisson distribution was used as the mathematical

model for the estimation of measurement uncertainty. Comments received on the procedure indicate that the

measurement uncertainty was underestimated. Would estimates of the measurement uncertainty be very dif-

ferent if the calculations were carried out according to the old and new version of NMKL Procedure No 8,

respectively?

In this Annex, an example from the previous version of NMKL Procedure No. 8 is used to compare the two

models. The example is based on the estimation of the number of heterotrophic bacteria in a food sample,

where data from 4 countable plates are used. The following formulas are used in the calculations:

Poisson distribution Normal distribution

ANNEX:

arison of estimations of measurementf

Mean, X

VC

V

C

VVVCCC

n

n

n

n

n

n ==++++++

∑

∑

1

1

21

21

....

C = number of colonies

V = dilution

nxxxx

n

xn

n

ii +++=

∑= ...3211

x = single results (cfu/g) or log10 cfu/g

n = number of parallels

(use Excel – Mean (figure1, figure2, figure3,

figure4)) Standard deviation, SD

CV1

1

)(1

2

−

∑ −=

n

xxn

ii

(use Excel – STDEV(figure1, figure2, figure3,

figure4))



No. 8, 4. Ed. (2008)


The results for heterotrophic bacteria are reproduced below as cfu/g and log10 cfu/g.

Results Poisson distribution Normal distribution Normal distribution

cfu/g at dilution cfu/g log10 cfu/g

112 at 10-4 1120000 6.05

95 at 10-4 1200000 6.08

12 at 10-5 950000 5.98

9 at 10-5 900000 5.95

Mean value 41036360.00022

==228

1010101091295112

5544 ++++++

−−−− 6.015 ≈ 1.04 · 106

≈ 1.04 · 106

= 1042500 ≈ 1.04 · 106

Standard deviation 6863522800022.0

1= 141038 0.059

Expanded measurement

uncertainty

U = X ± 2 · SD

1.

I cfu/g:

= 1.04· 106 ± 2· (106.02-

106.02-0.059)

1.04 · 106 ± 2· 0.13·106

1.04 · 106 ± 2· 0.69 · 105

= 1.04 · 106 ± 0.14 · 106

04 · 106 ± 2 · 0.14·106

6.02 ± 0.12

Confidence interval cfu/g [0 [0.78· 106 , 1.3 · 106] [0.90 · 106 , 1.2 · 106]

.76 · 106 , 1.3 · 106]

In this example, the estimates of the expanded measurement uncertainties obtained by the calculations based

a twice the expanded uncertainty calculated using the model from the

Poisson distribution. The lowest level of the confidence interval from 6 cfu/g, is also the

lowest obtained result from the 4 counted plates. This leaves no

value, and therefore the measurement uncertainty could be cons g too narrow.

on Normal distribution, re exactly

Poisson, 0.90 · 10

room for results below the lowest measured

idered bein

MEASUREMENT OF UNCERTAINTY IN ... - NMKL - · PDF fileMeasurement of uncertainty in...

Documents

Transcript of MEASUREMENT OF UNCERTAINTY IN ... - NMKL - · PDF fileMeasurement of uncertainty in...