Post on 07-Feb-2017
Xpert MTB/RIF EQA panel evaluation
1
A multi-centre feasibility study to assess External Quality Assessment panels for Xpert® MTB/RIF 1 assay in South Africa 2
Running title: Xpert MTB/RIF EQA feasibility study 3
Authors: 4
Lesley Scott1#, Heidi Albert3, Chris Gilpin4, Heather Alexander5, Kyle DeGruy5, Wendy Stevens1,2 5
Author affiliations: 6
1Department of Molecular Medicine and Haematology, School of Pathology, Faculty of Health 7 Sciences, University of the Witwatersrand, Johannesburg, South Africa 8
2National Health Laboratory Service, Johannesburg, South Africa 9
3Foundation for Innovative New Diagnostics (FIND), Geneva, Switzerland 10
4 Global TB Programme, World Health Organization, WHO, Geneva, Switzerland 11
5Division of Global HIV/AIDS, U.S. Centers for Disease Control and Prevention, Atlanta, USA 12
Institution where work was performed: Department of Molecular Medicine and Haematology, 13 School of Pathology, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, 14 South Africa 15
Corresponding author#: 16
Lesley Scott 17
7 York Road Parktown 18
Department of Molecular Medicine and Haematology, School of Pathology, Faculty of Health 19 Sciences, University of the Witwatersrand, WITS Medical School, 20
Parktown, 2193 21
Johannesburg, South Africa 22
lesley.scott@nhls.ac.za 23
24
JCM Accepts, published online ahead of print on 30 April 2014J. Clin. Microbiol. doi:10.1128/JCM.03533-13Copyright © 2014, American Society for Microbiology. All Rights Reserved.
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
2
Abstract 25
External quality assessment (EQA) for Xpert MTB/RIF is part of the quality system required for 26
clinical and laboratory practise. Five newly developed EQA panels using the following different 27
matrices: lyophilised sample (Vircell, Granada, Spain), dried tube specimen (CDC), liquid (Maine 28
Molecular Quality Control, Inc,Scarborough, Maine), artificial sputum (GLI) and dried culture spot 29
(NHLS) were evaluated at eleven GeneXpert testing sites in South Africa. 30
Panels comprised Mycobacterium tuberculosis complex (MTBC) negative, MTBC positive (including 31
rifampicin [RIF] susceptible and resistant) and non-tuberculosis mycobacteria material inactivated 32
and safe for transportation. Twelve qualitative and quantitative variables were scored as acceptable 33
(1) or unacceptable (0); overall panel performance score for Vircell, CDC, GLI and NHLS was 9 of 12 34
and while MMQCI- scored 6 of 12 (owing to the need for cold chain maintenance). All panels 35
showed good compatibility with Xpert MTB/RIF testing and none showed PCR inhibition. Liquid or 36
dry matrix did not appear to be a distinguishing criterion as both had reduced scores on insufficient 37
volumes, need for extra consumables and ability to transfer to the Xpert MTB/RIF cartridge. 38
EQA is an important component of the quality system required for diagnostic testing programmes, 39
but must be complemented by routine monitoring of performance indicators and instrument 40
verification. The study aims to introduce EQA concepts for Xpert MTB/RIF testing and evaluate five 41
potential EQA panels. 42
Words: 196 43
44
Key words: Xpert MTB/RIF, External Quality Assessment, five panel evaluation 45
46
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
3
Introduction: 47
The endorsement of the Xpert MTB/RIF (Xpert) assay by the World Health Organization (WHO) in 48
December 2010 (1) led to an unprecedented commitment by tuberculosis (TB) programs and donors 49
worldwide to implement this technology in efforts to improve the diagnosis of and initiate a 50
continuum of care to reduce the burden of disease. South Africa leads the implementation with 51
more than 2,400 000 sputum specimens tested using Xpert MTB/RIF (November 2013) on 284 52
instruments in 207 smear microscopy testing laboratories country wide, with a further 6 Gx 80 53
instrument placements in the pipelineThe South African experience to date has highlighted several 54
areas for program strengthening realised during their national phased implementation: the need for 55
improved technical and clinical training; the need for laboratory information systems for data 56
management and the need for quality system components such as an external quality assessment 57
(EQA) program, to name a few (2, 3)). The quality system needs to encompass the pre-analytical, 58
analytical and post analytical process to ensure on-going test quality, and an EQA program is one of 59
the internationally recognised tools in this armamentarium (4, 5). In this context for Xpert MTB/RIF, 60
a panel must require intact MTB that can be processed outside a BSL3 (or even BSL2) laboratory 61
infrastructure. 62
Currently no EQA program exists for the Xpert MTB/RIF test which represents a paradigm shift in 63
molecular testing for TB. A methodology using dried culture spots (DCS) has been developed in 64
South Africa for verifying GeneXpert(6) instruments on installation to ensure instrumentation is fit 65
for purpose, and has been used as an EQA for a clinical trial with the AIDS Clinical Trial Group 66
(ACTG) in sites in Africa, Brazil, Peru and the United States. This study therefore investigated the 67
performance of 5 EQA panels being developed in response to this need. The five EQA panels 68
consisting of varying matrices developed specifically for Xpert EQA were donated from both 69
commercial and non-commercial manufacturers: dried culture spots (DCS) developed in South 70
Africa for the National Health Laboratory Services (NHLS), National Priority Program (NPP)(6); dried 71
tube specimens (DTS) developed by the Centers for Disease Control and Prevention (CDC)(7); 72
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
4
artificial sputum developed by the Global Laboratory Initiative (GLI)/technical expert advisory group 73
to WHO and Stop TB Partnership; lyophilized samples developed by Vircell (Granada, Spain) in 74
conjunction with the Foundation for Innovative New Diagnostics (FIND) and liquid matrix specimens 75
developed by Maine Molecular Quality Controls Inc. (MMQCI, Scarborough, Maine)(8) as shown in 76
Figure 1. The performance evaluation of all five panels was conducted in South Africa at 11 Xpert 77
testing sites. One reference laboratory tested each of the panels and the results served as the 78
reference standard against which the quantitative results from the 11 Xpert testing sites were 79
compared. 80
81
Materials and methods: 82
Panel preparation and distribution 83
Examples of the five panels are illustrated in Figure 1. The NHLS DCS panel consisting of a 84
characterised MTBC RIF susceptible isolate (H37Rv) grown in bulk single cell culture which had been 85
inactivated using Xpert MTB/RIF sample reagent (SR) buffer (a kit component), followed by 86
quantification by flow cytometry and spotted with blue dye onto Whatman 903 filter cards (Merck) 87
(6). For the present study the same technique was used to create an EQA panel consisting of 4 88
samples: 1 MTBC RIF susceptible, 1 MTBC RIF resistant, 1 non-tuberculosis mycobacterium (NTM) 89
and 1 negative (water), each spotted onto a separate Whatman filter paper card. The CDC panels 90
were created by dilutions from known cultures grown in Mycobacteria Growth Indicator Tube 91
(MGIT) (Becton Dickenson, Sparks, Maryland)media followed by chemical inactivation with Xpert 92
MTB/RIF Sample Reagent (SR), glass bead disruption and drying in a Class II biosafety cabinet(7). This 93
CDC panel consisted of 4 samples: 1 MTBC RIF susceptible, 2 MTBC RIF resistant, and 1 negative. The 94
GLI panels were composed of artificial sputum containing heat killed bacilli and consisted of 4 95
samples: 1 MTBC RIF susceptible, 2 MTBC RIF resistant, and 1 negative. A lyophilised panel from 96
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
5
Vircell was developed in conjunction with the Foundation for Innovative New Technologies (FIND) 97
and consisted of 4 samples: 1 MTBC RIF susceptible, 1 MTBC RIF resistant, 1 non-tuberculosis 98
mycobacterium (NTM) and 1 negative. The MMQCI-liquid panel was composed of transfected MTBC 99
DNA in Escherichia coli chemically fixed and killed from Maine Molecular Quality Inc. (MMQCI). The 100
MMQCI panel consisted of 3 samples: 1 MTBC RIF susceptible, 1 MTBC RIF resistant, and 1 negative. 101
All panels were centrally received and re-packaged according to WHO safety protocols (9) using gas 102
impermeable and sealed transport bags and boxes and couriered together to a convenience sample 103
of 11 randomly selected NHLS Xpert laboratories. Testing sites represented a variety of testing 104
volumes and levels of the TB laboratory network. All panels where transported at room 105
temperature except the MMQCI panel which required cold-chain maintenance. The Xpert sites 106
varied both in geography and service level and were visited prior to the commencement of the study 107
to establish feasibility and willingness of participants. The reference laboratory was the Research 108
Diagnostic Laboratory in the Department of Molecular Medicine and Haematology, University of the 109
Witwatersrand, Medical School in Johannesburg. To blind study participants to the panel 110
compositions, barcodes were generated, encoding the panel matrix as well as the contents of each 111
sample. Each panel box also contained panel specific testing instructions produced by the developer. 112
Extra consumables were required for the NHLS panel including a universal container/Nunc 113
tube/sputum jar into which the perforated spot is placed and a pipette tip. An extra pipette was also 114
required for the NHLS, GLI and MMQCI panels to dispense the SR buffer into the containers. These 115
were not supplied by the EQA program as they are standard laboratory consumables. 116
Technologists were provided bench aids and received training on how to upload the results onto 117
www.tbgxmonitor.com. TBGx Monitor® is a web based application developed by the Department of 118
Molecular Medicine and Haematology of the NHLS and the University of the Witwatersrand team to 119
manage their DCS verification program of all GeneXperts implemented in the field(6). An additional 120
tab was included to allow sites to upload the CSV (comma separated value) file format generated by 121
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
6
the GeneXpert software after each EQA sample was analysed on the GeneXpert instruments. The 122
sites were also provided with a testing schedule, describing the order in which the tests were to be 123
performed, to minimise any confounding variables. A qualitative questionnaire accompanied the 5 124
panels sent to each site and contained questions relating to the condition of the material, label 125
clarity, ease of operation, sufficient material and ease of protocol use. A general comments section 126
allowed for narrative capture. The processing of the panels took place over two weeks, with three 127
panels being performed in the first week and the remaining two panels, in the second. This was done 128
in order to minimise interruption of daily work-flow at these routine testing sites. The GeneXpert 129
modules used for testing each sample are randomly assigned by the GeneXpert software and 130
therefore not determined by the operators. 131
Panel evaluation and scoring 132
A measure of overall panel performance was determined using a scoring system across the 133
qualitative and quantitative variables. A score of 1 was awarded for acceptable performance and a 134
score of 0 was awarded where issues were documented. Performance was calculated based on 10 135
qualitative and 2 quantitative evaluation criteria, and summarised as the most frequent score 136
reported across 11 testing sites. The qualitative criteria (also termed discrete variable) were: (1) 137
correct results reported for each panel compared to the result of the sample performed at the 138
reference centre; (2) instrument errors related to sample volume (too little volume/too viscous/fluid 139
transfer failed) testing such as 5006 and 5007; (3) the need for cold chain maintenance during 140
transportation of a panel and therefore the need for additional packaging; (4) the need for extra 141
consumables to perform a panel test; (5) questionnaire observations related to condition of goods 142
received, (6) clarity of the standard operating procedures (SOPs), (7) ease of opening samples, (8) 143
ease of handling samples, (9) ease of rehydration of sample, and (10) ease of transferring sample to 144
Xpert testing cartridge. For example: (a) small packaging and therefore lower courier cost received 145
a score of 1 while large packaging and higher courier cost received a 0; (b) the need for cold chain 146
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
7
maintenance was scored as 0 while panels with no cold chain requirement received a score of 1; (c) 147
in the case of the quantitative variables, a cycle threshold (Ct) SD similar to the majority group was 148
awarded 1 and an outlier (higher than the majority group) was awarded 0 for any probe (A-E); an 149
invalid result was scored 0. These criteria may need to be refined with time and experience In 150
addition to the qualitative reported result (MTB detected, MTB not detected) generated by the 151
Xpert MTB/RIF assay, a semi-quantitative range (very low, low, medium, high) based on the value of 152
the Ct is also reported (10). The Ct is the threshold at which fluorescence from the hybridization 153
probes increases as the target region is amplified. This is further explained by the assay’s molecular 154
characteristics: Xpert is an automated molecular technology that amplifies the rpoB gene of 155
Mycobacterium tuberculosis and detects this target using molecular beacons (11, 12). Resistance to 156
Rifampicin (RIF) is determined by delayed or drop out hybridization of the molecular beacon probes 157
(5 probes span the rpoB target and 1 probe hybridizes to the internal control). Delayed hybridization 158
occurs when there is >4 cycle threshold (Ct) difference (for assay version G4) between any probes 159
and drop out is reported when the Ct is zero (13). The mean Ct value and the standard deviation 160
(SD) of the Ct for each probe (A-E) of each panel were calculated. The amount of variability in the Ct 161
values of each probe (A-E) across the panels was used as a quantitative evaluation criterion as a 162
potential measure of bacterial consistency in each EQA matrix. For example a qualitative Xpert 163
result of RIF resistance could be generated by a delayed or a dropout probe(s). Their final outcome 164
result of RIF resistance is the same, but the SD for values ranging from 0 (drop out) to 40 (highest Ct 165
value that could indicate delayed hybridization) would be high. 166
A sample from each panel was also processed at the reference laboratory and referred to as the 167
reference result. Bar charts were used to visualize the SD of each probe set across all panels. PCR 168
inhibition of the internal control (SPC) was also reported if any sample yielded an invalid result. 169
Quantitative criteria for scoring included (1) probe Ct SD and (2) PCR inhibition of any sample, 170
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
8
Several other observations were also noted during the study that would be considered important in 171
managing a national EQA program but were not used to compare panel performance in this study. 172
These were number and proportion of returned results (categorised by results uploaded onto TBGx 173
Monitor, results returned by email, results received as paper returns, and results not returned) and 174
issues such as unreadable samples (bar scans errors) or incorrect labels. 175
Statistical Analysis 176
Summary descriptives and performance indicators were tabulated for each EQA panel. The mean 177
and standard deviation (SD) were calculated for the quantitative variable of cycle threshold (Ct) for 178
each probe (A-E, and the internal control [SPC]). The consensus (across all panels) SD of the Ct was 179
determined through visualization on a bar chart and used to separate the panel Ct values into the 180
consensus and outlier groups. MS Excel was used for all calculations. Performance scores are 181
provided in binary format. 182
Results: 183
Panel compositions and general observations 184
MTBC positive RIF resistant, MTBC positive RIF susceptible, and negative samples were included in 185
all panels. NHLS and Vircell were the only panels to also include an NTM sample. Two of the five 186
panels (MMQCI and GLI) used a liquid matrix and the remaining three (Vircell, CDC and NHLS) used 187
dried matrix formats. MMQCI was the only panel with 3 samples (only three were purchased for this 188
study); while all the other panels consisted of 4 samples (Table 1). Results were returned for 96% 189
(201/209) of samples. All (100%) results were returned for the CDC, MMQCI and NHLS panels, but 190
91% of GLI panel results and 82% of Vircell panel results were returned. The “no returns” came from 191
two sites which could not read the 2D bar-code labels affixed to the samples. All participants were 192
encouraged to make use of the TBGx Monitor website for submission of their CSV files, however only 193
33% of the MMQCI CSV files and 40-43% of the CSV files from all other panels were submitted on 194
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
9
line, suggesting some difficulty in use of the TBGx Monitor. This was similarly noted in the narrative 195
comments of the questionnaire. The Vircell panel had 4 unreadable 2D bar scans. Two sites would 196
have yielded non-conformances in an EQA program: one site did not return the questionnaire and 197
one site mixed all the labels across the panels and would have generated incorrect results. The latter 198
samples were correctly identified through their CSV files, and therefore were included in the data 199
analysis. 200
Qualitative panel performance 201
Expected results were observed for the CDC, GLI and NHLS panels. Using the Vircell panels, one 202
negative sample yielded a false-positive result and with MMQCI, one positive (with RIF resistance) 203
was missed. Using the CDC panels no GeneXpert run errors were observed. Vircell and GLI panels 204
each had one sample where the instrument generated a 2008 error which relates to syringe pressure 205
and therefore not related to the tested material. MMQCI panels showed two insufficient volume 206
errors (5007, 5006) in spite of being a liquid matrix, and GLI and NHLS each had 1 insufficient volume 207
error (Table 1). 208
Of the qualitative questionnaire responses: GLI had no issues; NHLS had 1 issue (“not easy to open”); 209
CDC had 2 issues (“not easy to open” and “not easy to handle”). Vircell and MMQCI had three issues 210
each namely “not easy to transfer”, “material insufficient to test”, and “protocol not easy to follow”. 211
The most expensive panel to transport was the MMQCI panel as this required cold chain 212
maintenance. This was also the heaviest and largest package, due to the addition of cold packs. 213
Courier costs were dependant on weight, distance and whether road and/or air transportation was 214
needed. The laboratory, to which delivery was most expensive, was the most remote, requiring both 215
air and road transportation. 216
Quantitative performance of panels 217
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
10
Table 2 presents the average and standard deviation (SD) of each probe’s Ct and SPC Ct across all 218
panels. The sample size across panels per probe analysed, ranged from 7 to 16. The SD values for 219
each probe across all panels are represented in two bar charts in Figure 2 for MTB detected RIF 220
resistant samples and RIF susceptiblesamples. The Vircell followed by the MMQCI panels had the 221
lowest SD on all probes compared to the other panels. The panel with the most variability (SD 222
greater than the majority consensus SD of all panels [SD>2.3) for the RIF susceptible samples was the 223
CDC panel (probes A-E SD>2.3) followed by GLI (probes A and B SD >2.3). The SD on the probes of 224
the RIF resistant sample was highest across 4 probes for the GLI (probes A-D SD>2.3), and 1 probe 225
for CDC (probe C SD>2.3). Although the CDC, GLI and NHLS panels showed more variability on their 226
SPC, no invalids (SPC inhibition) were reported for any sample across all panels indicating the panel 227
materials did not cause any PCR inhibition. 228
Overall panel score 229
The overall performance of the five panels is summarised in Table 3 using the scoring system for the 230
main variables examined. Overall the Vircell, CDC, GLI and NHLS panels all generated the same score 231
of 9 out of a possible score of 12. The MMQCI was the only panel with a lower score of 6. In more 232
detail: Vircell yielded an incorrect result, standard operating procedure (SOP) was unclear and 233
material was not easy to handle; CDC was not easy to open or handle and there was greater 234
variability on amplification and probe hybridization; the GLI panel had insufficient volume, required 235
extra consumables and had greater variability on amplification and probe binding; NHLS had 236
insufficient volume, required extra consumables and was not easy to open; MMQCI yielded an 237
incorrect result, had insufficient volume, required cold chain and extra consumables, SOP was 238
unclear and material was not easy to transfer to the Xpert cartridge. 239
240
Discussion: 241
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
11
The implementation of a new assay in settings which are often not experienced with laboratory 242
testing creates many challenges in the “quality system”(14), even though the test in question is 243
designed for users with minimal expertise. The “quality system”, as described by CLSI (Clinical 244
Laboratory Standard Institute) is intended to ensure quality of the overall testing process (specimen 245
collection to result reporting for patient management); to detect and reduce errors; to improve 246
consistency within and between laboratories; contain costs and ensure customer satisfaction. This 247
encompasses twelve elements such as organization, personnel, equipment, quality control, 248
assessment, facility and safety (15). 249
Many quality system components have often been lacking in resource poor settings in general (16, 250
17) and quality assessment programmes for Xpert MTB/RIF have been no exception. In particular 251
there are three components that appear critical to ensure the quality and accuracy of Xpert MTB/RIF 252
testing: (1) verification of each GeneXpert module (on installation or repair) to ensure the 253
instrument is “fit for purpose” before testing and reporting on clinical specimens; (2) external quality 254
assessment to ensure the entire testing process (pre-analytical, analytical and post-analytical) is 255
quality managed and (3) continuous performance monitoring (error rate, potential for 256
contamination, usage, etc.). Verification should be performed on each module and although to date, 257
published evidence on the use of EQA panels for instrument verification is available only for the 258
NHLS panel (6), the other panels evaluated in the current study may also prove suitable for 259
verification. 260
This feasibility study on five EQA panels addresses the gap in EQA for Xpert MTB/RIF and showed 261
little overall differences in scores between the Vircell, CDC, GLI or NHLS panels indicating that any of 262
these panels are likely to be suitable for use in an EQA programme. The MMQCI-liquid panel 263
received a lower score as it was the only panel that required cold chain maintenance and the 264
material was not easy to transfer into the Xpert cartridge. This cold chain requirement is likely to be 265
a challenge in many low resource settings. All panels were received in good condition and therefore 266
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
12
good for shipping across distances, and all were easy to rehydrate and dissolve fully in the SR buffer, 267
showing good compatibility with the Xpert testing process. In addition none of the panel materials 268
caused any PCR inhibition. The matrix requirement (liquid or dry) for EQA material did not appear to 269
be a distinguishing criterion as both liquid and dried formats received reduced scores due to 270
insufficient volumes, the need for extra consumables, and ease of transfer to the Xpert cartridge. 271
The increased variation (increased SD) in the Ct (due to some Xpert MTB/RIF tests reporting delayed 272
probe hybridization and some reporting drop out probes with no hybridisation) of the CDC and GLI 273
panels may be attributed to the use of MGIT bulk stock which may have increased clumping of 274
bacterial cells compared to the bulk stock manufactured in single cell format used in the NHLS- 275
product, or other factors inherent to their specific preparation. Minimal variation may, however, be 276
more attractive to an EQA program monitoring RIF resistance rates using differences in probe drop 277
out or probe delayed hybridisation as well as differences in Ct values across the reported semi-278
quantitative categories, since studies are already describing the potential use of Xpert for patient 279
monitoring using Ct values(13, 18). 280
In keeping with international recommendations for EQA that a preferred matrix should bear 281
resemblance to the clinical sample and one that is relatively cost effective to produce, easy, stable 282
and safe to transport, as well as being accurate and precise (15, 19, 20), a liquid specimen format for 283
Xpert MTB/RIF EQA may prove more suitable. Xpert programs would also need guidance on EQA 284
testing schedules and perhaps could draw from other existing molecular EQA schemes: the NHLS 285
National Priority Program in South Africa, for example, subscribes to international schemes that 286
provide three to four panels per year for molecular testing such as HIV viral load. 287
Factors such as SOP clarity, label bar-code scanning, and use of the web based program for result 288
entry highlight the need for any EQA program to be accompanied by training and on-going 289
improvements. Although the web based program, TBGx Monitor was under development during the 290
study and raised concerns of difficulty in use by some South African Xpert users, TBGx Monitor was 291
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
13
also used in a concurrent pilot study conducted in collaboration with ACTG experienced clinical trial 292
sites. The ACTG trial included 6 sites in USA, South Africa, Brazil and Peru. The NHLS panels were 293
tested and despite English not being the first language in some sites, none of the sites reported 294
difficulty in uploading their results. Efficient use of both the GeneXpert instrument and the TBGx 295
Monitor software require basic computer skills. Thus, additional computer training may be required 296
for technicians with limited previous experience with computers. 297
The third component in the quality system identified as important to Xpert testing is continuous 298
performance monitoring, since EQA only provides a snap shot. Connectivity standards(21) have 299
made possible several commercial products for hospital, clinic and laboratory information systems 300
as well as instrument information systems, and these should be applied to all testing platforms. In 301
South Africa, for example, seventeen national HIV viral load molecular testing laboratories 302
comprising two platforms (CAP/CTM from Roche Molecular Systems, Branchburg, NJ and 303
m2000sp/m2000rt Abbott Molecular Inc., Des Plaines, Illinois, USA) are centrally monitored 304
in real time for instrument down time, utility, error rate, contamination, etc., which results in rapid 305
response to problems. A similar concept should apply to the GeneXpert as part of managing the 306
entire quality system. Such a remote monitoring system is under development by the GeneXpert 307
manufacturer, Cepheid (Sunnyvale, CA, USA). 308
309
Acknowledgement 310
Disclaimer: The opinions expressed herein are those of the authors and do not necessarily represent 311
the official position of the U.S. Centers for Disease Control and Prevention. Financial support: This 312
project has been funded in part by the President’s Emergency Plan for AIDS Relief (PEPFAR) through 313
the U.S. Centers for Disease Control and Prevention, under the terms of (3U2GPS0001328-04). The 314
Grand Challenges Canada, grant # 0007-02-01-01-01 (Wendy Stevens) also provided support. The 315
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
14
authors wish to thank the South African National Health Laboratory Service GeneXpert participating 316
sites. Lesley Scott also wishes to disclose she is the inventor of the Dried Culture Spot product which 317
is undergoing PCT application (PCT/IB2011/052016) through the University of the Witwatersrand. 318
319
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
15
320
Table 1: Description of the panels and summary of qualitative data 321
Table 2: Qualitative results describing the mean values and SD per probe compared to the reference. 322
The probes that identified the result as MTBC detected RIF resistant are highlighted in grey 323
Table 3: Overall performance score of the five EQA pilot panels. 324
Figure 1: Representation of the five EQA panels designed for the Xpert MTB/RIF assay. 325
326
Figure 2: Bar charts of SD for each probe and internal control (SPC) across all panels for two groups 327 of samples: (a) MTB detected RIF susceptible and (b) MTB detected RIF resistant. The dotted line 328 separates the SD values into two groups >2.3 and <2.3 329
330
331
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
16
332
References 333
1. http://www.who.int/tb/laboratory/mtbrifrollout/en/index.html. 2012. WHO monitoring 334 of Xpert MTB/RIF roll-out. World Health Organisation. 335
2. Stevens W. 2012. Analysis of needed interface between molecular diagnostic tests and 336 conventional mycobacteriology: Country experience in South Africa. In 43rd Union World 337 Conference on Lung Health SoAmdwasatipc (ed.), Kuala Lumpur, Malaysia, November. 338
3. Boehme C. April 2012. Xpert MTB/RIF: Technology update, Workshop on TB diagnostics and 339 laboratory services: Actions for Care delivery and sustainability Global Laboratory Initiative, 340 WHO, Annecy, France. 341
4. Parsons LM, Somoskovi A, Gutierrez C, Lee E, Paramasivan CN, Abimiku A, Spector S, 342 Roscigno G, Nkengasong J. 2011. Laboratory diagnosis of tuberculosis in resource-poor 343 countries: Challenges and opportunities. Clin Microbiol Rev 24:314-350. 344
5. Palamountain KM, Baker J, Cowan EP, Essajee S, Mazzola LT, Metzler M, Schito M, Stevens 345 WS, Young GJ, Domingo GJ. 2012. Perspectives on introduction and implementation of new 346 point-of-care diagnostic tests. J Infect Dis 205 Suppl 2:S181-190. 347
6. Scott LE, Gous N, Cunningham BE, Kana BD, Perovic O, Erasmus L, Coetzee GJ, Koornhof H, 348 Stevens W. 2011. Dried Culture Spots for Xpert MTB/RIF External Quality Assessment: 349 Results of a Phase 1 Pilot Study in South Africa. J Clin Microbiol 49:4356-4360. 350
7. DeGruy K, Zilma R, Alexander H. 2012. Dried Tube Specimens (DTS) for Xpert MTB/RIF 351 Proficiency Panels., ASLM (African Society for Laboratory Medicine) conference, Cape Town, 352 South Africa, December. 353
8. http://www.mmqci.com/qc-m110.php. INTROL TB panel M110. MMQCI, accessed April 354 2014. 355
9. http://www.who.int/ihr/infectious_substances/en/. Accessed May 2013. A Guide for 356 Shippers of Infectious Substances, 2013, International Health Regulations (IHR), World 357 Health Organisation. 358
10. Helb D, Jones M, Story E, Boehme C, Wallace E, Ho K, Kop J, Owens MR, Rodgers R, Banada 359 P, Safi H, Blakemore R, Lan NT, Jones-Lopez EC, Levi M, Burday M, Ayakaka I, Mugerwa RD, 360 McMillan B, Winn-Deen E, Christel L, Dailey P, Perkins MD, Persing DH, Alland D. 2009. 361 Rapid detection of Mycobacterium tuberculosis and rifampin resistance by use of on-362 demand, near-patient technology. J Clin Microbiol 48:229-237. 363
11. Boehme CC, Nabeta P, Hillemann D, Nicol MP, Shenai S, Krapp F, Allen J, Tahirli R, 364 Blakemore R, Rustomjee R, Milovic A, Jones M, O'Brien SM, Persing DH, Ruesch-Gerdes S, 365 Gotuzzo E, Rodrigues C, Alland D, Perkins MD. 2010. Rapid molecular detection of 366 tuberculosis and rifampin resistance. N Engl J Med 363:1005-1015. 367
12. Helb D, Jones M, Story E, Boehme C, Wallace E, Ho K, Kop J, Owens MR, Rodgers R, Banada 368 P, Safi H, Blakemore R, Lan NT, Jones-Lopez EC, Levi M, Burday M, Ayakaka I, Mugerwa RD, 369 McMillan B, Winn-Deen E, Christel L, Dailey P, Perkins MD, Persing DH, Alland D. 2010. 370 Rapid detection of Mycobacterium tuberculosis and rifampin resistance by use of on-371 demand, near-patient technology. J Clin Microbiol 48:229-237. 372
13. Blakemore R, Nabeta P, Davidow AL, Vadwai V, Tahirli R, Munsamy V, Nicol M, Jones M, 373 Persing DH, Hillemann D, Ruesch-Gerdes S, Leisegang F, Zamudio C, Rodrigues C, Boehme 374 CC, Perkins MD, Alland D. 2011. A multisite assessment of the quantitative capabilities of 375 the Xpert MTB/RIF assay. Am J Respir Crit Care Med 184:1076-1084. 376
14. Parekh BS, Kalou MB, Alemnji G, Ou CY, Gershy-Damet GM, Nkengasong JN. 2010. Scaling 377 up HIV rapid testing in developing countries: comprehensive approach for implementing 378 quality assurance. Am J Clin Pathol 134:573-584. 379
15. CLSI. Accessed May 2013. http://www.clsi.org/wp-380 content/.../LabGuidelinesStandardsGP26Jan2012.pdf , Quality management system: 381
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Xpert MTB/RIF EQA panel evaluation
17
Development and managment of laboratory documents; approved guidelines, 6th Edition. 382 The CLSI (Clinical Laboratory Standards Institute) Quality System, 2011, An Overview of the 383 Essential Elements of a PT/EQA Program. 384
16. Nkengasong JN, Nsubuga P, Nwanyanwu O, Gershy-Damet GM, Roscigno G, Bulterys M, 385 Schoub B, DeCock KM, Birx D. 2010. Laboratory systems and services are critical in global 386 health: time to end the neglect? Am J Clin Pathol 134:368-373. 387
17. Petti CA, Polage, C. R., Quinn, T. C., Ronald, A. R. & Sande, M. A. 2006. Laboratory medicine 388 in Africa: a barrier to effective health care. Clin Infect Dis 42:377-382. 389
18. Theron G, Pinto L, Peter J, Mishra HK, Mishra HK, van Zyl-Smit R, Sharma SK, Dheda K. The 390 use of an automated quantitative polymerase chain reaction (Xpert MTB/RIF) to predict the 391 sputum smear status of tuberculosis patients. Clin Infect Dis 54:384-388. 392
19. Noble MA. 2007. Does external evaluation of laboratories improve patient safety? Clin Chem 393 Lab Med 45:753-755. 394
20. MA. N. Accessed May 2013. http://wwwn.cdc.gov/mlp/pdf/GAP/Noble.ppt, An Overview of 395 the Essential Elements of a PT/EQA Program. 396
21. CLSI. 2008. Point of care connectivity: Approved Standard - Second edition: POCT1-A2 397
Clinical and Laboratory Standards Institute 26. 398
399
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Vircell CDC MMQCI GLI DCS
matrix Lyophilised Dried Tube
specimen Liquid (E.coli) Artificial sputum Dried Culture Spot
negative 1 1 1 1 1
MTB negative, NTM 1 0 0 0 1
MTB positive, RIF sensitive 1 1 1 1 1
MTB positive, RIF resistant 1 2 1 2 1
number of testing sites 11 smear microscopy sites, now performing Xpert MTB/RIF, 1 Reference site
number of samples sent to sites, n 44 44 33 44 44
Return of results
Total returned results (%) 36 (81.8) 44 (100) 33 (100) 40 (90.9) 44 (100)
CSV files uploaded onto tbgxmonitor (%) 18 (40.9) 19 (43) 11 (33) 18 (40.9) 18 (40.9)
CSV returned by email that could be uploaded (%) 16 (36.4) 16 (36.4) 15 (45) 16 (36.4) 20 (45)
Paper returns 2 (4.5) 9 (20) 7 (21) 6 (13.6) 6 (13.6)
no returns 8 (18) 0 0 4 (9) 0
Qualitative results
True negative (n/total) (17/18) (11/11) (10/10) (10/10) (10/10)
False negative (n/total) (1/18)* (0/11) (0/10) (0/10) (0/10)
(* MTB positive, RIF
sensitive)
True MTB detected, RIF sensitive (n/total) (8/8) (11/11) (10/10) (10/10) (11/11)
False MTB detected, RIF sensitive (n/total) (0/8) (0/11) (0/10) (0/10) (0/11)
True MTB detected, RIF resistant (n/total) (9/9) (22/22) (10/11) (18/18) (11/11)
False MTB detected, RIF resistant (n/total) (0/9) (0/22) (1/11)# (0/18) (0/11)
(# MTB not detected)
True negative NTM (n/total) na na na na (11/11)
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
False negative, NTM (n/total) (0/11)
Errors 1 0 2 2 1
2008 (instrument) 5006 (insufficient
volume) 2008
(instrument) 5007 (insufficient volume)
5006 (insufficient
volume) 5006 (insufficient
volume)
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from
Vircell CDC MMQCI GLI DCS Reference mean
MTB detected, RIF sensitive mean SD mean SD mean SD mean SD mean SD Vircell CDC MMQCI GLI DCS
n 7 10 8 8 10 1 1 1 1 1
Probe A, mean (SD) 19.6 1.7 22.0 2.9 24.1 2.1 15.7 2.3 19.4 2.1 20.4 22.2 20.9 12.2 21.9
Probe B, mean (SD) 21.4 1.6 23.6 3.0 25.5 2.1 17.6 2.5 21.0 2.2 21.8 23.5 22.5 13.5 22.9
Probe C, mean (SD) 20.0 1.8 22.3 3.0 24.5 2.1 16.0 2.2 19.8 2.2 20.7 22.6 21.2 12.4 22.0
Probe D, mean (SD) 21.2 2.0 23.5 2.8 25.5 1.9 17.4 2.3 20.8 2.2 21.6 23.5 22.3 13.3 23.0
Probe E, mean (SD) 20.9 1.6 23.3 2.8 25.2 2.3 17.0 2.2 20.8 1.9 22.3 23.9 22.6 13.9 23.7
SPC, mean (SD) 24.8 2.1 26.8 3.2 25.7 2.1 27.8 2.1 27.6 3.1 26.7 24.3 26.6 28.7 30.5
MTB detected, RIF resistant mean SD mean SD mean SD mean SD mean SD
n 8 8 8 16 10 1 1 1 2 1
Probe A, mean (SD) 19.3 2.4 20.9 1.8 0.0 0.0 16.4 4.9 20.9 2.6 18.4 33.9 0.0 12.0 11.7 17.4
Probe B, mean (SD) 20.9 2.5 21.9 1.8 0.0 0.0 18.1 4.9 22.7 2.6 19.7 33.7 0.0 12.2 13.2 19.0
Probe C, mean (SD) 19.9 2.5 19.9 21.3 22.7 2.3 16.9 5.0 21.2 2.7 18.9 0.0 20.4 12.5 12.3 17.7
Probe D, mean (SD) 21.1 2.5 22.5 1.5 0.0 0.0 18.1 4.7 0.0 0.0 20.2 33.8 0.0 13.4 13.1 0.0
Probe E, mean (SD) 0.0 0.0 22.3 1.9 23.2 2.2 0.0 0.0 22.1 2.6 0.0 35.6 21.6 0.0 0.0 18.9
SPC, mean (SD) 26.4 3.5 26.0 2.3 25.0 2.0 29.4 3.3 27.4 3.4 23.6 31.7 28.3 26.9 28.4 27.0
MTB detected, RIF resistant mean SD mean SD mean SD mean SD mean SD
n na 9 na na na 1
Probe A, mean (SD) 26.4 2.6 27.6
Probe B, mean (SD) 27.8 2.4 27.9
Probe C, mean (SD) 26.8 2.7 27.0
Probe D, mean (SD) 0.0 0.0 0.0
Probe E, mean (SD) 27.6 2.8 28.5
SPC, mean (SD) 27.3 1.9 25.6
MTB not detected, NTM mean SD mean SD mean SD mean SD mean SD
n all values negative na na na 9 1
Probe A, mean (SD) 0.0 0.0 0.0
Probe B, mean (SD) 0.0 0.0 0.0
Probe C, mean (SD) 24.4 18.4 0.0
Probe D, mean (SD) 0.0 0.0 0.0
Probe E, mean (SD) 0.0 0.0 0.0
SPC, mean (SD) 27.1 1.7 28.5 *two sample sin the GLI panel were the same and analysed together since they had the same Rif resistant
profile
on April 6, 2018 by guest
http://jcm.asm
.org/D
ownloaded from