CS4723 Software Validation and Quality Assurance Lecture 11 Static Bug Detection and Verification.
Validation, Verification and Quality Control
description
Transcript of Validation, Verification and Quality Control
![Page 1: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/1.jpg)
Validation, Verificationand Quality Control
Thomas Novicki Ph.D. D(ABMM)
Marshfield Clinic
Marshfield WI
![Page 2: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/2.jpg)
Today’s Topics
1.Validation and Verification of ID and AST systems
2.Quality Control (QC) of Identification (ID) and Antimicrobial susceptibility test (AST) systems
(We will not be discussing proficiency testing due to time constraints)
2
![Page 3: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/3.jpg)
Disclosures I have nothing relevant to this
presentation to disclose
3
![Page 4: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/4.jpg)
VALIDATION AND VERIFICATION
4
![Page 5: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/5.jpg)
Validation and Verification Validation: Documenting in vitro
diagnostic device performance in your lab
Verification: An ongoing demonstration of the vendor’s performance claims using a variety of tools
5
![Page 6: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/6.jpg)
Validation and Verification(or is it Verification and Validation…?)
Note that some entities (e.g. CLSI, CLIA) reverse the definitions Initially Verify Then periodically Validate
The definitions used don’t matter, but their consistent use in your lab is important.
What does your accrediting body use?
6
![Page 7: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/7.jpg)
Who Guides Us? Vendors of commercial systems The Clinical and Laboratory Standards
Institute (CLSI) The Feds
CLIA – Clinical labs FDA – Instruments
Your accrediting body (COLA, CAP, JC)(Did I miss anything…?)
7
![Page 8: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/8.jpg)
Validation and Verification
The Centers for Medicare and Medicaid Services (CMS), (a part of the US Department of Health and
Human Services) regulates clinical laboratory testing through the Clinical Laboratory Improvements
Amendments (CLIA) that became effective in 2003.
(Recall that it all began in 1988)
8
![Page 9: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/9.jpg)
Validation
For FDA cleared and approved tests, the lab demonstrates performance claims by the device manufacturer, typically found in the product label (‘package insert’)
For lab-developed tests (LDTs), a more rigorous set of studies demonstrating performance is necessary
9
![Page 10: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/10.jpg)
Validation of LDTs Will not discuss further except to say that any
modification of an FDA IVD makes it an LDT
Be wary of modifying the more critical FDA-approved test (e.g. TB, HIV IVD devices )
Qualitative tests easier to modify than quantitative
Simple cleared tests (e.g. MRSA chromo-agars for additional anatomical sites) are reasonably modified
10
![Page 11: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/11.jpg)
iClick? 1 CLIA requires a lab to validate which of
the following for every new diagnostic test1. Accuracy
2. Precision
3. Reportable and Reference ranges
4. All of the above
5. 1 and 2
11
![Page 12: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/12.jpg)
iClick? 1 - Discussion CLIA specifies that the user of an FDA IVD device of
Moderate/High complexity will Validate accuracy, Precision, Reportable range, and Reference range
You must satisfactorily complete validation of the device before beginning its use.
You must document all validation study data and maintain it for at least 2 years after test retirement.
12
![Page 13: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/13.jpg)
Validation – AST Two methods for an unmodified FDA-
cleared instrumentA. Compare to a reference methodB. Compare to an existing system
Method B probably the best fit for most labs with an existing system For method A, see Cumitech 31A
(Cumitech 31A, Verification and validation of procedures in the clinical microbiology laboratory, ASM Press 2009)
13
![Page 14: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/14.jpg)
14
DisclaimerWhat I’m about to say on validation…
‘… is more what you’d call guidelines than actual rules’
![Page 15: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/15.jpg)
Validation – AST Source of samples/isolates to test
> 30 well-characterized clinical isolates per panel that reflects the mix common to your population and also some with unusual AST patterns
Can include proficiency or commercial validation panels, but be wary: Reflective of your population? Incorrect sample matrix? If from the device manufacturer, a self-fulfilling prophecy?
Note that with care a single panel can be used for both AST and ID validations
15
![Page 16: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/16.jpg)
Validation – ASTHow to deal with discrepancies when two non-reference systems are compared?
No reference method, so grade results, then apply predetermined limits on the number and
types of discrepancies
16
![Page 17: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/17.jpg)
Validation – AST For each bug/drug result, calculate %s for…
Categorical (S/I/R) Errors Minor Error I vs S or I vs R Major Error S vs R or R vs S No Very Major Error category: ‘True’ result is not known
Essential Errors (> + 1 MIC dil. difference) Can only score when both methods generate discrete
numbers (i.e. do not score < or > values)
17
![Page 18: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/18.jpg)
Validation – ASTUsing the total number of evaluable bug/drug data points, an acceptable validation has:
< 5% Major Errors AND < 10% Major + Minor Errors AND < 10% Categorical Errors AND < 10% Essential Errors
Also look for trends in a particular drug or bug which may point towards a problem area
Consult your vendor rep with any validation problems
18
![Page 19: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/19.jpg)
Validation – AST Precision (reproducibility)
Test five isolates X3 for three to five days
Calculate intra-and inter-run categorical and essential % agreements
Isolates should have known susceptibility profiles, such as ATCC strains
Use both Gram positive and Gram negative strains if validating GP and GN panels
Acceptable precision > 95%agreement
19
![Page 20: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/20.jpg)
Validation – ID Test a range of Gram positive and Gram negative organisms ID’d by reference or current method(s)
Minimum of 20 patient isolates; larger labs test more. Additionally, test QC strains
< 10% discrepancies between reference and new IDs
20
![Page 21: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/21.jpg)
Validation – ID Level of agreement may need to be adjusted
e.g. Old ID of ‘Coag-negative Staphylococcus sp’ New instrument ID’s to species level
Precision Test 2-4 isolates X3 for three to five days Calculate % intra- and inter-run agreements Accept > 95% agreements
21
![Page 22: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/22.jpg)
Validation What to do if validation fails? Do not use
the new test and either…
Withdraw test from further consideration OR
Develop and implement a corrective plan with the manufacturer. Then, re-validate
22
![Page 23: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/23.jpg)
Verification – Component of the QA Process
An ongoing, multi-faceted process that assures the validated test continues to perform at the expected level
Personnel competency Proficiency challenges
QC organisms or analytes Comparison with other labs
Instrument PM/calibration Trend analysis
Discrepant resolution Pre- and post-analytic controls
Written procedures documenting the process
23
![Page 24: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/24.jpg)
Quality Control
I think, therefore I control quality
24
![Page 25: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/25.jpg)
iClick? 2You have an instrument that uses micro-broth dilution MIC plates in your lab. This system incubates and reads plates automatically but also has a manual plate reader. Your tech rep tells you to read a plate manually if QC fails on the instrument. If QC then passes, your QC is acceptable for the entire system and no follow-up action is needed.
1.True2.False
25
![Page 26: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/26.jpg)
iClick? 2: discussionQC is performed to assure the accuracy of a test result by assessing all analytical components
Reagents Instrument (where applicable) Test precision (i.e. repeatability) Operator technique Data interpretation
26
![Page 27: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/27.jpg)
CAP on AST QC‘For AST of either disk or dilution type,
control organisms are tested with each new lot or batch of antimicrobials or media and each day the test is performed thereafter.’
(Note that >3 failures per month per drug/bug combo is unacceptable)
27
![Page 28: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/28.jpg)
CAP on AST QC‘However, the frequency of test monitoring
may be reduced to weekly …if the laboratory can document satisfactory performance with
daily control tests as suggested by CLSI guidelines.’
The CAP checklist then goes on to briefly describe how to do so
28
![Page 29: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/29.jpg)
AST QC ResourcesCOLA, JC have similar verbiage, but is the checklist guidance enough?
IMHO*, No!
You need CLSI AST documents in your lab.
(*In my humble opinion.)
29
![Page 30: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/30.jpg)
CLSI AST Documents M2 (disk diffusion) and M7 (MIC), 3 year cycle
Test performance QC (including daily-to-weekly QC)
M100, annual cycle Recommended drugs to test and report, interpretive breakpoints QC strains, QC troubleshooting, when to re-validate QC
Others documents for other classes of microorganisms
(Explore these and more at http://www.clsi.org/; choose Microbiology from the Shop drop down menu)
30
![Page 31: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/31.jpg)
iClick? 3How many of you have these CLSI documents in your lab:
1. 2012 M2 (disk diffusion), M7 (MIC) and M100 (Breakpoints, QC)
2. 2012 M7 and M1003. 2012 M2 and M1004. 2012 M2 and M75. Any older versions of these documents6. None of the above
31
![Page 32: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/32.jpg)
Conversion to Weekly AST QCWhen day-of-use QC is being successfully performed at least once a week, a lab may consider weekly AST QC by a validation process:
Test QC strains 20 or 30 consecutive days If 0-1 failures per drug-bug combo in 20 days
OR 2-3 failures in 30 days
THEN Weekly QC may be implemented
32
![Page 33: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/33.jpg)
Weekly AST QC Perform QC testing once per week and
with any new reagent If weekly AST QC fails
If readily identifiable error (e.g. wrong QC strain; see M2 or M7 for more examples) correct error and retest once. If QC passes, document findings and continue weekly QC
If no error is found…
33
![Page 34: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/34.jpg)
Weekly AST QC Re-validationPerform 5 consecutive days of QC on the failed drug-bug combo
(We automatically take out a fresh QC strain from the freezer at this point)
If ALL 5 days pass, document and resume weekly testing
If a failure occurs again, STOP TESTING and thoroughly evaluate the process for error
34
New QC strain New lot of reagent
Correct process errors Consult the manufacturer
![Page 35: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/35.jpg)
Weekly AST QC Re-validation Once a likely error has found, re-validate
Do not include data from the prior 5-day sequence Must again pass these criteria
0-1 failures per drug-bug combo/20 days OR 2-3 failures per drug-bug combo/30 days
Remember that, during re-validation the lab is on a daily QC schedule-a failure means that that drug’s results are not released
35
![Page 36: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/36.jpg)
AST QC References Disk diffusion
M02-A11 Section 15 M100-S22 Tables 3C, 3D
MIC M07-A9 Section 16 M100-S22 Tables 4F, 4G
36
![Page 37: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/37.jpg)
AST QC Special Considerations
M100-S22 Tables 3C & 4F, ‘Reference Guides to QC Frequency’
When and how much re-validation to perform when a system change occurs (e.g. AST instrument repair, Abx formula change & Etc)
Release of patient results when QC fails? CLSI says OK with many qualifiers, but our
lab generally just says No
37
![Page 38: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/38.jpg)
iClick? 4You are performing a weekly AST QC re-validation for drug X on your favorite automated ID/AST instrument when you incur a failure on the same AST panel for drug Y. What do you do?
1. Ignore the failure of drug Y and press on with the 5-day testing of drug X
2. Now treat the failure of drug Y as a second failure and start a 5 day test of drug Y
3. Stop testing, evaluate the whole process, then perform a new 20/30 day validation on the whole AST panel
38
![Page 39: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/39.jpg)
iClick? 4 discussionThe correct answer isn’t clear, but Marshfield Labs treats the failure of the second drug as a failure and starts a 5 day test on that drug also
Why? Subsequent failures may reflect a larger problem
39
![Page 40: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/40.jpg)
ID QCCLIA requires a lab to show positive and negative reactions for each new lot and/or shipment of biochemical.
This includes kit and automated instrument ID methods!
40
![Page 41: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/41.jpg)
RapID ANA II Identification System Quality Control Each new lot number and/or shipment is to be tested with the following Lot#/Exp Date ________________________________ organisms to provide at least one positive and one negative reaction Set-up Date/Time/Init. __________________________ for each biochemical within the ID system. Read Date/Time/Init. ___________________________ Results Acceptable? ___________________________ Expected Results:
Before Reagent Addition After reagent addition
Organism URE BLTS ARA ONPG GLU GLU GAL FUC NAG PO4 LGY GLY PRO PAL ARG SER PYR IND
Clostridium sordellii
ATCC 9714
E + - - - - - - V - - - V + V V V V +
A
Bacteroides distasonis
ATCC 8503
E - + V + + + + - + + + + - + + + + -
A
Bacteroides uniformis
ATCC 8492
E - + + + + + + + + + + - - - - - - +
A
+ positive, - negative, V variable, (-) usually negative, (+) usually positive E = Expected Reactions A = Actual R
41
![Page 42: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/42.jpg)
ID QCThere is a QC solution for automated ID
instrument users:
Streamlined QC
Stay Tuned for the Next Talk!
42
![Page 43: Validation, Verification and Quality Control](https://reader030.fdocuments.net/reader030/viewer/2022033018/56814553550346895db221ae/html5/thumbnails/43.jpg)
The End
43