Microbiological Method Validation: How Do We Prove that a Method is Fit for Purpose?
Thomas Hammack
Chief Microbial Methods Development Branch
Division of Microbiology Office of Regulatory Science
Center for Food Safety and Applied Nutrition U.S. Food and Drug Administration
College Park, MD 20740
Equivalence?
24 h, 35C
24 h, 35C and 42C
Does or =
Why validate methods? Benefits public health and world trade False negative results are unacceptable ISO 17025 lab accreditation demands the use of validated methods
Validation Programs AOAC Microbiological Guidelines
ISO 16140:2003
FDAs Guidelines for the Validation of Analytical
Methods for the Detection of Microbial Pathogens in Foods http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf
AOAC International
1884Association of Official and Analytical Chemists Methods Validation for Fertilizers Membership Restricted to Regulatory Chemists
Supported by USDA
Dr. Harvey Wiley -Father of FDA is also considered the Father of AOAC 1892Secretary of AOAC 1898Established AOACs Committee on Food Safety
Housed within FDA
1939Microbiological Sampling of Eggs and Egg products Official Method 939.14 still in use today
Many AOAC methods Official Methods were developed and validated in FDA Labs
1972FDA published acceptance of AOAC Official Methods in the Federal Register Allows FDA to use proprietary rapid methods for the analysis of regulatory samples
1979became an independent non-profit organization no longer tied to FDA, but many
FDA employees serve as volunteers
International Organization for Standardization (ISO)
International non-governmental organization Based in Geneva, Switzerland Comprised of 163 national standards institutes including ANSI in the US
Started in 1926 as the International Federation of National Standardizing Associations (ISA)
Focused on mechanical engineering Disbanded in 1942 Reorganized as ISO in 1946 Mission statement is to provide International Standards for Business, Government and Society
1991 ISO and European Committee for Standardization (CEN) signed the Vienna Agreement
CEN is comprised of the 31 Member States of the European Union Vienna agreement stipulates that CEN and ISO Standards be identical to enable free trade within the
EU
ISO Technical Committee 34 (TC 34) Subcommittee 9 (SC 9) ISO exercises its role in the standardization of food microbiological testing methods through TC
34/SC 9 TC 34/SC 9 started standardizing horizontal reference methods for bacteria in foods in the 1970s
18
FDAs Methods Validation Guidelines The Science and Research Steering Committee (SRSC), of the Office of Foods and Veterinary Medicine (OFVM), approved guidance to be used for validation of microbiological and chemical methods.
Guidelines for the Validation of Analytical Methods for the
Detection of Microbial Pathogens in Foods http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf
Guidelines for the Validation of Chemical Methods for the FDA Foods Program
http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM298730.pdf
Scope
These criteria apply to all FDA laboratories that develop and participate in the validation of analytical food methods for Agency-wide implementation in a regulatory capacity. This includes all research laboratories, and field labs where analytical methods may be developed or expanded for potential
regulatory use. These documents will supersede all other intra-agency documents pertaining to food-related method validation criteria for microbial and chemical analytes. the SRSC will authorize the
formation of a Methods Validation Subcommittee (MVS) to serve as the governing body for all method validation concerns.
FDA and Methods Validation Method validation is a process by which a laboratory confirms by examination, and provides objective evidence, that the particular requirements for specific uses of a method are fulfilled. It serves to demonstrate that the method can detect and identify an analyte or analytes: In one or more matrices to be analyzed In one or more instruments or platforms With a demonstrated sensitivity, specificity, accuracy, trueness, reproducibility, ruggedness
and precision to ensure that results are meaningful and appropriate to make a decision.
Reliably for its intended purpose. Intended purpose categories include, but may not be limited to emergency/contingency operations; rapid screening and high throughput testing; and, confirmatory analyses.
After the method developer has conducted experiments to determine or verify a number of specific performance characteristics that serve to define and/or quantify method performance.
9
RESEARCH
VALIDATION
Micro Method
Validation Sub-group
The Office of Foods and Veterinary Medicine & the SRSC Roadmap for Microbiological Method Development and Validation
IMPLEMENTATION
Organizational Partnerships
BAM Council FERN MCC IFSH ORA Micronauts
VERIFICATION
10
ESTABLISHES validation needs and priorities in consultation with the SRSC-Micro Super-group, FDA Bacteriological Analytical Manual Council (BAM Council), FERN Method Coordinating Committee, ORA micronauts inter-center working groups and others as appropriate ADOPTS procedures to govern all administrative processes needed for emergency and non-emergency method validation proposals and studies. PROVIDES planning, guidance, oversight, and resources to participating laboratories during the method development and validation process; will be the responsible authority for recommendations, evaluations and final approval of all validation studies from planning through field implementation. CONSULTS with other governmental, and independent (commercial, and international) validation bodies to harmonize validation standards where possible and practices
The Method Validation Sub-Group (Microbiology)
11
BROAD REPRESENTATION from CFSAN, ORA, CVM, and NCTR with additional expertise from biostatisticians and QA/QC managers CURRENT MICRO MVS COMPOSITION: ORA Palmer Orlandi (co-Chair), Cathy Burns CFSAN Thomas Hammack (co-Chair), William Burkhardt, Darcy Hanes CVM Beilei Ge NCTR Steven Foley FERN Don Burr NCFST (CFSAN Moffett) Ravinder Reddy
The Method Validation Sub-Group (Microbiology)
Qualitative assays
Analyte
Bacteriological, e.g. Salmonella spp. Pathogenic Escherichia coli Listeria monocytogenes Shigella spp Vibrio spp Campylobacter spp
Microbial toxins
Viral pathogens, e.g. Hepatitis A virus Norovirus Enterovirus
Parasitic protozoan pathogens, e.g. Cryptosporidium Cyclospora cayetanensis
Bioengineered analytes, e.g.
Genetically-modified foods (GMOs)
Applications
Pre- and selective enrichment
Microbial analyte recovery and concentration
Screening, high-throughput, confirmation
Procedures
Phenotypic, e.g. Biochemical characterization Antibiotic resistance traits Antigenic characterization
Genetic, e.g. Nucleic acid isolation/concentration Polymerase Chain Reaction
Conventional, Real-time Reverse transcription
Sequencing, e.g. Whole genome Selective sequencing Single nucleotide polymorphism (SNP) analysis
Strain-typing applications
METHOD VALIDATION SCOPE OF RESPONSIBILITY Pathogens, Genetic Material, Toxins and Antigens
1. Submission of a new or original method, OR,
2. Any significant modification of a method that may alter its performance specifications or changes to the fundamental science of an existing method. Categories include:
Substitutions of reagents/apparatus
Expansion of the scope of an existing method to include additional analytes.
Changes in intended use i.e. screening or confirmatory.
Platform extensions or significant parameter changes e.g. adaptation to another real-time PCR thermal cycler.
Matrix extensions.
Changes to time/temperature incubation periods, or enrichment media.
In cases where the sample preparation and/or the extraction procedure/analytical method is modified from the existing test procedure and protocol, i.e the new method should demonstrate that the modifications do not adversely affect the precision and accuracy or bias of the data obtained.
Modification of a methods performance range e.g. specificity, sensitivity beyond previously validated levels.
Method Validation is Required for
Levels of Validation
Four levels of performance are defined. The hierarchy of scrutiny will provide general characteristics on the methods utility and insights for its intended use, the assessed risk, and the food-borne illness potential for an analyte-matrix pairing.
Not all methods will or should be validated to meet the requirements of a Level 4: full collaborative study.
Method Validation Criteria Level One The lowest level of validation, with all the work done by one lab. Inclusivity and exclusivity testing has been tested, but by a limited number of strains. The analyte was tested at a level based on the intended use of the method, with just normal background flora. There is no aging of the artificially-inoculated samples and no comparison to an existing reference culture method. The expectation would be for the originating lab to continue to conduct further testing to eventually elevate the method to a higher level of validation. Intended Use: Emergency needs. A method developed for the detection of an analyte, or a matrix not previously recognized or identified as a threat to food safety or public health. As the first level in the development of any method designed for regulatory use; performance of the method at this level of scrutiny will determine, in part, whether further validation is useful or warranted. NOTE: Under emergency situations where the rapid development and deployment of a method is needed to immediately address an outbreak event, Level 1 criteria should be followed as closely as the situation will allow. Representatives of the MVC and Agency subject matter experts (SMEs) should be in close consultation with the originating laboratory. Once the crisis has past and it has been determined that there is a need for further validation, procedures outline in this document will be followed.
Method Validation Criteria
Level Two This is a more robust study, with the possibility of regulatory strength depending on the circumstances. The originating lab has done a more comprehensive initial study, with inclusivity/exclusivity levels at the AOAC Collaborative Study level. If possible, a comparison has been done to an existing reference culture method. One other independent laboratory has participated in the collaborative study. Some of the criteria of the study are at a lower level than the full AOAC study, but still appropriate for the developing method at this stage.
Intended Use: Emergency needs. Slightly higher false-positive rates are acceptable as all samples analyzed with methods validated to this level will require confirmatory testing.
Level Three This is a validation level that should have full regulatory strength. Most of the criteria followed by the originating lab are at the AOAC level, including inclusivity/exclusivity, analyte levels, competitor strains, aging, and comparison to existing method when available. The additional collaborating labs follow many of the criteria of an AOAC collaborative study. Intended Use: All methods validated to this level of scrutiny are acceptable for use in any and all circumstances e.g. confirmatory analyses; regulatory sampling, and compliance support. Level Four This validation level has criteria equivalent to the AOAC Collaborative Study. Any method reaching this level of validation should be able to be submitted for adoption by the AOAC as a fully collaborated method.
Method Validation Criteria
Originating Laboratory Criteria
Level One: Urgent usage
Level Two: Independent lab validation
Level Three: Multiple lab collaborative
Level Four: Full collaborative study
AOAC Collaborative Study
# of target organism (inclusivity)
10 (20 for Salmonella)
50 (unless 50 aren't available)a,b
50 (unless 50 aren't available)a,b
50 (unless 50 aren't available)a,b 50
a,b
# of non-target organism (exclusivity) 10 strains 30 strains
c 30 strainsc 30 strainsc 30 strainsc
# of foods 1 or mored 1 or mored 1 or mored 1 or mored Up to 20 foodse
# of analyte levels/food matrixf set level based on intended use and negative control
One inoculated levelf and uninoculated level
One inoculated levelf and uninoculated level
One inoculated levelf and uninoculated level
One inoculated levelf and uninoculated
level
Replicates per food at each level tested 20 20 20 20 20
Aging of inoculated samples prior to testing No Yes Yes
g Yesg Yesg
Addition of competitor strainh Normal background flora In 1 food at +1 log>analyte at
fractional positivef analyte level
In 1 food at +1 log>analyte at
fractional positivef analyte level
In 1 food at +1 log>analyte at
fractional positivef analyte level
In 1 food at +1 log>analyte at
fractional positivef analyte level
Comparison to recognized method No Yes, if available Yes, if available Yes, if available Yes, if available
Table 1. ORIGINATING Laboratory Requirements
I. General Qualitative Guidelines for Microbial Analytes
Microbiology Validation Category Examples
Collaborating Laboratory Criteria Level Two: Independent lab validation Level Three: Multiple lab
collaborative Level Four: Full
collaborative study AOAC Collaborative Study
# of laboratories providing usable data 2 5 10 10
# of foods 1 or morea 1 or morea 1 or morea 1 to 6 foodsb
# of strains of organism 1 per food 1 per food 1 per food 1 per food
# of analyte levels/food matrixc 2 levels: One inoculated levelc
and uninoculated level
3 levels: One inoculated levelc one at 1 log higher and
uninoculated level
3 levels: One inoculated levelc, one at 1 log higher and
uninoculated level
3 levels: One inoculated levelc one at 1 log higher and
uninoculated level
# of replicate samples/food 8 per analyte level 8 per analyte level 8 per analyte level 6 per analyte level
Aging of inoculated samples prior to testing Noe Yesd Yesd Yesd
Comparison to Recognized Method Yes, if available Yes, if available Yes, if available Yes, if available
Microbiology Validation Category Examples, continued Table 2 - General Qualitative Guidelines for Microbial Analytes-Collaborating Laboratory Requirements
Originating Laboratory Criteria
Category One: Urgent
usage
Category Two:
Independent lab
validation
Category Three:
Multiple lab collaborativ
e
Category Four: Full
collaborative study
Replicates per strain 3 3 8 8 Comparison to
recognized methoda No Yes, if
available Yes, if
available Yes, if
available
Collaborating Laboratory Criteria
Level One: Urgent usage
Level Two: Independent lab
validation
Level Three: Multiple lab
collaborative
Level Four: Full collaborative study
# of laboratories providing usable datab n/a 2 5 10
Replicates per strain n/a 3 8 6 Comparison to
Recognized Methoda n/a Yes, if available Yes, if
available Yes, if available
II. General Qualitative Guidelines for Food-borne Microbial Pathogens That Present Unique Isolation and/or Enrichment Challenges
Microbiology Validation Category Examples, continued
Table 1. ORIGINATING Laboratory Requirements
Table 2. COLLABORATING Laboratory Requirements
III. CRITERIA AND GUIDANCE FOR THE VALIDATION OF FDA- DEVELOPED MOLECULAR-BASED ASSAYS
Inclusivity and exclusivity Target gene(s) and controls (positive and negative). Comparison to the Reference Method
IV. FOOD MATRIX EXTENSION FOR VALIDATED MICROBIOLOGY METHODS
The validation of a method for a food matrix not previously included in a validation study is necessary to assure that the new matrix will produce neither high false positive (matrix is free from cross reactive substances) no high false negative rates (matrix is free of inhibitory substances)
Guidance to Support Field Laboratory Expedience Guidance for the Formal Expansion of Validated Food Matrices
Microbiology Validation Category Examples, continued
V. PLATFORM EXTENSION
VI. CRITERIA AND GUIDANCE FOR THE VERIFICATION AND VALIDATION OF COMMERCIALLY- AVAILABLE MICROBIOLOGICAL DIAGNOSTIC KITS AND PLATFORMS
1. For commercially-available microbiological diagnostic kits whose performance parameters have been fully validated in a collaborative study monitored and evaluated by an independent accrediting body e.g. AOAC-OMA, AFNOR, etc.
2. For commercially-available microbiological diagnostic kits whose performance parameters are supported by data obtained through an abbreviated validation protocol and evaluated by an independent accrediting body e.g. AOAC-RI.
Validation of an Alternative method: Demonstration that adequate confidence is provided when the results obtained by the alternative method i.e. the commercially-available kit, are comparable to or exceed those obtained using the reference method using the statistical criteria contained in the approved validation protocol. Verification: The confirmation by examination and the provision of objective evidence that specified requirements have been fulfilled. Criteria For Kits Fully Validated in a Collaborative Study Monitored by an Independent Accrediting Body
Microbiology Validation Category Examples, continued
23
Current Micro MVS Priorities I. Hepatitis A Virus a. Real-time RT-qPCR assay to detect Hepatitis A virus b. In a food matrix (green onions) Status: Final report nearing completion 2. Non-O157:H7 STEC Screening method to detect non-O157:H7 STECs using the BioPlex Status: Multi-lab collaborative study has been completed.. 3. Salmonella Enteritidis Isolation and Detection of Salmonella Enteritidis (SE) from Whole Shell Eggs-Cultural and Molecular Applications 4. Norovirus
a. Real-time RT-qPCR assay to detect Noroviruses b. In a food matrix (molluscan shellfish)
Status: planning phase underway in collaboration with the ORA-CFSAN virology working group
24
Method Authors Andrew Lin Teresa Lee
Laurie Clotilde Julie A. Kase Insook Son
J. Mark Carter Carol R Lauzon
Validation of an Identification Method for Shiga-toxigenic E. coli Somatic (O) Groups using the
BioPlex Suspension Array System
25
STECs are a significant public health concern
Non-O157 STECs are responsible for over 60% of STEC infections or an estimated 112,000 illnesses in the U.S. each year
Over 74.2% of STEC infections in the U.S. are caused by serogroups O26 (23.9%), O45 (7.8%), O91 (2.3%), O103 (16.7%), O111 (12.6%), O121 (7.5%), and O145 (3.4%)
O26, O103, O111, O121, and O145 are known to cause HC and HUS, and O45 is associated with HC
Other serogroups that may cause HC and HUS, but are less commonly isolated, are O91, O113, and O128
Validation of a STEC Molecular Serotyping Method
26
BioPlex Assay Overview Must be performed on Pure Cultures Bead-based assay that can be multiplexed
Conventional and real-time PCR assays for STEC O serogroup - limited by resolution of band sizes
on a gel, or the # of fluorescent channels 96-well plates Targets = nucleic acid, proteins Ab, antigens, cytokines Each bead = a separate assay 100 different color-coded beads (magnetic or polystyrene)
Unique color comes from different ratios of two distinct flourophores Two lasers, fluidics, optics, detectors Automated, High-Throughput, Fast, Expandable
Validation of a STEC Molecular Serotyping Method
27
Method Status In-House Validation Study Completed
Paper LIB
5 Lab Collaborative Study Successfully Completed
Approved by the BAM Council
Validation of a STEC Molecular Serotyping Method
28
In-house Validation Results Inclusivity Panel
Validation of a STEC Molecular Serotyping Method
40
Exclusivity Panel
Validation of a STEC Molecular Serotyping method
BioPlex Results
Fluorescence signals are quantified for each micro bead
Signal to background ratios are calculated, where background is measured using all no template reactions
Samples were considered to be positive when signal to background ratio was greater than 5.0
30
31
BioPlex Results
Validation of a STEC Molecular Serotyping method
32
Development and Validation of a Method for the Detection and Isolation of Salmonella Enteritidis (SE)
from Whole Shell Eggs
Guodong Zhang Eve Thau
Eric W. Brown Thomas Hammack
Validation of a Method for Salmonella in Whole Shell Eggs
33
Background SE is the second most commonly isolated Salmonella serotype
in the United States 14.18% (1968-1998)
SE is most commonly associated with whole shell eggs and egg products
FDAs egg rule (74 FR 33030) recommends various preventive control measures including sampling and sample analysis
FDAs BAM reference culture method for the detection of SE in whole shell eggs takes 9 days to complete
There are no AOAC Official Methods of Analysis methods for the detection of SE in whole shell eggs
Validation of a Method for Salmonella in Whole Shell Eggs
34
Clean surface Surface disinfection
20 eggs to a container Hand homogenize sample
Hold at room Temp (20-24oC) for 96 h
25 ml eggs to 225 ml TSB + ferrous sulfate; 24 h at 35oC
1 ml to 10 ml TT 24 h at 35oC
Streak on XLD, BS, HE 24 h at 35oC
TSI, LIA 24 h at 35oC
Serological Test
0.1 ml to 10 ml RV 24 h at 42oC
Streak on XLD, BS, HE 24 h at 42oC
TSI, LIA 24 h at 35oC
Serological Test
CURRENT BAM METHOD
Clean surface Surface disinfection
20 eggs + 2L TSB 24 h at 35oC
1 ml to 10 ml TT 24 h at 35oC
Streak on XLD, BS, HE 24 h at 35oC
TSI, LIA 24 h at 35oC
Serological Test
0.1 ml to 10 ml RV 24 h at 42oC
Streak on XLD, BS, HE 24 h at 35oC
TSI, LIA 24 h at 35oC
Serological Test
PROPOSED BAM METHOD
Validation of a Method for Salmonella in Whole Shell Eggs
35
Validation of a Method for Salmonella in Whole Shell Eggs
36
Validation of a Method for Salmonella in Whole Shell Eggs
37
Method Status
In-House Validation Study of Culture Method Complete Manuscript in preparation Report to BAM Council in preparation
In-House Validation Study of qPCR Methods in Progress ABI SE qPCR Method Promising Evaluation of an internal qPCR method in progress
Validation of a Method for Salmonella in Whole Shell Eggs
38
Collaborative Studies
Tentative Schedule
Culture Method Alone Fall 2012
Culture Method Plus qPCR Method Spring 2013
Validation of a Method for Salmonella in Whole Shell Eggs
Microbiological Method Validation: How Do We Prove that a Method is Fit for Purpose?Equivalence?Slide Number 3Validation ProgramsAOAC InternationalInternational Organization for Standardization (ISO)FDAs Methods Validation GuidelinesFDA and Methods ValidationSlide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20Slide Number 21Slide Number 22Slide Number 23Slide Number 24Slide Number 25BioPlex Assay OverviewMethod StatusIn-house Validation Results Inclusivity PanelExclusivity PanelBioPlex ResultsBioPlex ResultsDevelopment and Validation of a Method for the Detection and Isolation of Salmonella Enteritidis (SE) from Whole Shell EggsBackgroundSlide Number 34Slide Number 35Slide Number 36Method StatusCollaborative Studies
Top Related