Microbial Criteria and Testing -Probability, Sampling and Sampling

Plans*Don Schaffner, Ph.D.

Rutgers - The State University of NJ

* Adapted from ICMSF vol 2

About the source text• This book was the only comprehensive publication

on statistically based sampling plans for foods. 2nd ed. (1986). Toronto: University of Toronto Press. ISBN: 0802056938.

• Revised as: Microorganisms in Foods 7: Microbiological Testing in Food Safety Management. Kluwer Academic/Plenum Publishers, 2002. ISBN: 0306472627. Available from Kluwer Publishers

• Part 2 of the old book is available as a free download

• http://www.icmsf.org/pdf/icmsf2.pdf• Excel spreadsheets and other tools

• http://www.icmsf.org/main/software_downloads.html

Sample and Testing - United Fresh - 2019 2 of 31

Spreadsheet screenshot

Probability, populations and samples• The population is everything• The sample is what we observe• Let’s say we run 1000 tests:

• we get a positive result 112 times• we estimate the chance of a positive sample to be 0.112 or

11.2%• We run another 1000 tests:

• positive results 914 times• chance is: 0.914 or 91.4%

Sample and Testing - United Fresh - 2019 4 of 31

Sampling plan• A good sampling plan based on poor data, or poorly

selected samples is worthless…• A simple example plan:

• Take 10 samples• If more than 2 are positive, reject• Why more than? Think about c = 0

• In sampling lingo:• n = 10• c = 2

Sample and Testing - United Fresh - 2019 5 of 31

Operating Characteristic curve

n = 10c = 2

Sample and Testing - United Fresh - 2019 6 of 31

Understanding the OC curve60%

1 0 0 0 0 0 1 0 0 1 0 0 1 0 1 0 1 1 0 1 0 0 1 0 1 0 0 1 1 11 1 1 1 0 0 0 1 1 0 1 0 1 1 1 1 0 0 1 0 0 0 1 1 1 1 1 1 1 11 1 0 1 1 1 0 0 0 1 0 0 0 0 1 0 0 0 1 1 1 0 1 1 1 1 1 0 0 01 0 1 1 1 0 1 0 0 1 1 1 1 1 0 1 0 1 1 0 0 1 1 1 1 0 1 1 1 01 0 1 1 0 0 1 1 0 1 0 1 1 0 0 1 1 0 1 0 0 1 1 0 1 1 1 1 1 10 1 0 1 1 1 1 0 1 1 1 0 1 0 0 1 1 1 1 1 1 1 1 1 0 1 1 0 0 01 1 1 1 1 1 1 0 0 0 0 1 1 0 1 1 1 1 1 1 0 1 0 1 1 0 1 1 1 10 1 1 0 0 1 0 0 1 0 1 1 0 1 1 1 0 0 0 1 1 1 1 0 1 1 0 1 0 11 1 1 1 0 1 0 1 1 0 1 0 1 0 1 1 1 0 1 1 0 0 1 1 0 1 1 0 0 10 1 0 0 0 1 0 0 0 1 1 1 1 1 1 0 1 1 1 1 1 1 0 1 1 0 1 1 0 01 0 1 1 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 0 0 0 1 0 1 1 0 1 0 01 0 0 1 1 0 0 0 1 0 1 1 1 0 1 0 1 1 0 0 0 1 0 1 1 0 1 1 1 11 1 0 1 1 1 0 1 1 1 1 1 0 1 0 1 1 0 1 1 1 0 0 0 1 1 1 0 1 01 1 0 0 1 1 1 1 1 0 0 0 1 0 1 0 1 1 1 0 1 1 1 1 1 0 0 1 1 01 0 0 1 1 1 0 1 0 1 0 1 0 1 1 1 0 1 0 1 1 1 1 1 0 1 0 1 1 10 1 0 1 0 1 1 0 0 0 1 1 1 1 0 1 0 0 1 1 1 1 0 0 1 1 1 0 0 10 0 1 1 1 1 1 0 1 0 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 1 10 1 0 1 1 1 0 0 1 1 0 1 0 0 0 0 1 1 1 1 1 0 1 1 0 0 0 0 0 10 1 1 0 0 0 1 1 1 1 1 1 0 1 1 1 0 0 0 1 1 0 0 1 0 1 0 0 0 10 0 0 1 1 0 1 1 1 0 1 0 1 0 1 1 1 0 0 1 0 1 1 1 0 1 0 0 0 00 1 1 1 0 1 1 1 0 1 1 1 1 0 0 0 1 0 1 1 1 0 0 1 0 0 1 0 1 00 1 0 1 1 0 1 1 0 1 0 1 0 1 1 0 1 1 1 1 1 0 0 1 1 1 0 1 0 01 0 1 1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 0 1 0 1 1 0 1 0 0 0 0 00 1 1 1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 0 0 0 0 0 1 1 1 0 0 1 11 0 1 0 0 0 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 1 1 1 1 0 1 0 11 1 0 0 1 1 1 1 1 0 0 0 0 0 0 1 0 0 1 1 1 1 1 0 0 1 1 1 0 01 0 1 1 1 1 1 1 1 0 0 0 1 0 0 1 0 1 1 1 1 1 1 1 1 0 0 0 1 01 1 0 1 0 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 0 0 1 1 0 0 10 1 0 1 1 1 0 1 0 0 1 0 0 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 01 1 1 0 1 1 0 1 0 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1 0 0 1 1 1 11 1 1 1 0 0 1 0 1 1 1 1 1 1 1 1 0 1 1 0 1 1 0 0 1 1 0 0 1 0

20%0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 11 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 1 0 0 00 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 1 0 0 0 0 01 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 10 0 0 0 0 1 0 1 0 0 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 0 1 0 00 0 1 0 0 0 0 0 1 1 0 0 0 0 1 1 0 0 1 1 0 0 0 0 0 0 1 1 0 00 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0 1 0 0 0 10 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 00 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 00 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 1 00 0 0 0 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 1 1 1 0 00 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 00 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 00 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 11 0 0 0 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 1 1 0 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 1 1 0 1 0 0 0 1 0 1 10 0 0 0 0 0 0 0 0 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 1 0 1 01 0 0 0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 0 1 1 1 1 0 0 0 1 0 0 00 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 1 00 1 1 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 00 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 1 0 1 0 0 1 01 0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 1 1 0 0 0 00 0 0 0 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 00 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 00 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 0

Sample and Testing - United Fresh - 2019 7 of 31

Consumer and Producer risk• Consumers risk is the probability that an unacceptable

lot will be accepted• i.e. lot with high contamination will not be detected

• Producers risk is the probability that an acceptable lot will be rejected

• i.e. lot with low contamination will be rejected• Note that acceptance and reject are relative to a

particular purpose• Animal food may have lower standards than human food• Infant formula may have a higher standard than other foods

Sample and Testing - United Fresh - 2019 8 of 31

Principles of Drawing Samples• Can’t sample the entire shipment• Population

• What we want to make inferences about• Samples

• What we can observe• Probability

• Confidence that what we observe tells us about what we want to know

Sample and Testing - United Fresh - 2019 9 of 31

What is a Lot?• Ideally a lot is a quantity of food produced and handled

under uniform conditions• Rarely occurs in practice• Two or more batches may make up a lot• Temperature differences in a pallet or truck• Heterogeneity of lots makes interpretation difficult• Lot size tradeoff

• more tests will cost more but positive results mean a smaller amount to reject

Sample and Testing - United Fresh - 2019 10 of 31

What is a Sample?• Random sample

• Random number table, random number generator

• Stratified sampling• Useful if a lot is composed of several production batches• Accessed separately, pooled if uniform

Sample and Testing - United Fresh - 2019 11 of 31

Stratified sampling

Lot 1

2030 30

35

200

250

Sub-lot A Sub-lot B Sub-lot C

Sample and Testing - United Fresh - 2019 12 of 31

Sample size matters

375g

*

25g 25g 25g 25g 25g 25g 25g 25g 25g 25g 25g 25g 25g 25g 25g

Sample and Testing - United Fresh - 2019 13 of 31

Compositing matters• N = 60 is 60 times the

cost of N = 1• What if you pool the 60?• Pool, mix and pull

1/60th?

N = 60

*

*

Sample and Testing - United Fresh - 2019 14 of 31

Field sampling

15 of 31

Example from Xu and Buchanan• Nsample ↑ = detection

probability ↑• Not linear• Double n does not

double probability • Contamination sites

↑ = detection probability ↑

• Also not linear

Sample and Testing - United Fresh - 2019 16 of 31

Z vs. other methods• Top is random

• Nine 3*3 areas are missed, others “over” sampled

• Middle is random stratified• No 3*3 areas missed• All have one x

• Bottom is z• Fourteen 3*3 areas are always missed• This has consequences!

Sample and Testing - United Fresh - 2019 17 of 31

Sampling plan has consequences• Random and

Stratified-Random are roughly equal

• Z-pattern is much more variable

• Higher andlower

Sample and Testing - United Fresh - 2019 18 of 31

Practical considerations• Hazard - as the hazard increases, the probability of

accepting a lot which should be rejected should be reduced

• Uniformity - more uniform foods need fewer samples, less uniform foods need more samples

• Stratification - useful for sub-lots• Record of consistency - reduced sampling for

suppliers with good track recordSample and Testing - United Fresh - 2019 19 of 31

Practical considerations• Practicality - perishability

may not allow time for sampling

• The ability to discriminate may not improve much with increased sampling

• Reliability increases as the square root of the number of samples

Sample and Testing - United Fresh - 2019 20 of 31

Two-class plans• Results are

• positive or negative• concentration above or below a critical threshold

• n = number of samples• c = maximum number of unacceptable samples• m = unacceptable concentration

Sample and Testing - United Fresh - 2019 21 of 31

Two-class plans• Example

• n=10, c=2 means:• take 10 samples• if 2 or fewer are positive, accept• if 3 or more are positive, reject

• Stringency increases• as n increases• as c decreases

Sample and Testing - United Fresh - 2019 22 of 31

Two-class plan OC curves

Sample and Testing - United Fresh - 2019 23 of 31

Three-class plans• Three class plans are for situations where

microbiological quality can grouped into three classes: good, marginal and poor

• n = number of samples• m = unacceptable concentration• c = maximum number of unacceptable samples at “m”• M = reject lot if any count greater than M is obtained

Sample and Testing - United Fresh - 2019 24 of 31

Choice of Plan According to Purpose• Type and seriousness of the hazards• Conditions under which the food will be handled

and consumed

Sample and Testing - United Fresh - 2019 25 of 31

Choice of Plan According to Purpose

Moderate, limitedBacillus cereusCampylobacterStaphylococcusYersina

Moderate, extensiveE. coli O157:H7SalmonellaShigella

SevereClostridium bot.Salmonella typhiVibrio cholera

Sample and Testing - United Fresh - 2019 26 of 31

Choice of Plan According to Purpose

Sample and Testing - United Fresh - 2019 27 of 31

Choosing between 2 & 3 class plans

Sample and Testing - United Fresh - 2019 28 of 31

Choosing m and M

Sample and Testing - United Fresh - 2019 29 of 31

Satisfactory “Probability of Acceptance”?• Comparing two different plans

• Lenient 3-class plan (n=5, c=3)• will accept 5% defective (M) and 30% marginal (m) 3 out of 4

(Pa = 0.75) times• one out of four lots rejected - clearly serious enough to compel

change!• Stringent plan (n=20, c=0)

• will accept 5% defective 1 out of 20 (Pa = 0.05) times• will accept 0.5% defective 19 out of 20 (Pa = 0.95) times• even 1/20 reject may be enough to compel change

Sample and Testing - United Fresh - 2019 30 of 31

Sampling and FSMA/Preventive Controls/HACCP

• The current trend is away from microbiological testing to inspect safety into a product

• PC/HACCP focuses on immediate measurements (temperature, pH, etc.) that if in control can ensure safety

• Microbial testing is still useful: • validation of HACCP plans• benchmarking suppliers or processes

• See “FSMA: Testing as a Tool for Verifying Preventive Controls” by Buchanan and Schaffner, Food Protection Trends, 2015 35(3):228-237.

Sample and Testing - United Fresh - 2019 31 of 31