Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet...

9
Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo B. Pinheiro,* ,Helen OBrien, Julian Druce, § Hongdo Do, Pippa Kay, Marissa Daniels, # Jingjing You, × Daniel Burke, Kate Griths, and Kerry R. Emslie National Measurement Institute (NMI), Lindeld, Sydney, New South Wales 2070, Australia Research and Development, Australian Red Cross Blood Service, Kelvin Grove, Queensland 4059, Australia § Victorian Infectious Diseases Reference Laboratory, Melbourne, Victoria 3000, Australia Olivia Newton-John Cancer Research Institute, Translation Genomics and Epigenomics Laboratory, Heidelberg, Victoria 3084, Australia Agri-Bio Molecular Genetics, Biosciences Research Division, Bundoora, Victoria 3083, Australia # The Prince Charles Hospital University of Queensland, Thoracic Research Centre, Chermside, Queensland 4032, Australia × Save Sight Institute, Sydney Eye Hospital, Sydney Medical School, University of Sydney, Sydney, New South Wales 2000, Australia * S Supporting Information ABSTRACT: Use of droplet digital PCR technology (ddPCR) is expanding rapidly in the diversity of applications and number of users around the world. Access to relatively simple and aordable commercial ddPCR technology has attracted wide interest in use of this technology as a molecular diagnostic tool. For ddPCR to eectively transition to a molecular diagnostic setting requires processes for method validation and verication and demonstration of reproducible instrument performance. In this study, we describe the development and characterization of a DNA reference material (NMI NA008 High GC reference material) comprising a challenging methylated GC-rich DNA template under a novel 96-well microplate format. A scalable process using high precision acoustic dispensing technology was validated to produce the DNA reference material with a certied reference value expressed in amount of DNA molecules per well. An interlaboratory study, conducted using blinded NA008 High GC reference material to assess reproducibility among seven independent laboratories demonstrated less than 4.5% reproducibility relative standard deviation. With the exclusion of one laboratory, laboratories had appropriate technical competency, fully functional instrumentation, and suitable reagents to perform accurate ddPCR based DNA quantication measurements at the time of the study. The study results conrmed that NA008 High GC reference material is t for the purpose of being used for quality control of ddPCR systems, consumables, instrumentation, and workow. O ver the past few years, use of digital polymerase chain reaction (PCR) technology has rapidly expanded and diversied into a multitude of areas within life sciences. The expansion in application of digital PCR technology is evident from literature reports in various areas of research including pathogen detection, 14 monitoring of food and water safety, 57 and microbial ecology. 8 Because digital PCR can resolve small dierences in the amount of nucleic acids, the largest uptake of this technology is in clinical research such as biomarker quantication, 911 detection of genetic variants in cancer patients, 12,13 detection of copy number alterations in stem cells, 14 monitoring transplant recipients, 15 quantication of massively parallel sequence libraries, 1618 and genome editing. 19, 20 Recognition that digital PCR can provide unprecedented levels of precision, accuracy, and resolution for quantication of nucleic acids, together with development and availability of aordable instrumentation, have been the main reasons for the rapid expansion of digital PCR. Transitioning digital PCR technology from a research laboratory to a clinical setting requires processes for method validation and verication and demonstration of method reproducibility and instrument performance. Many DNA reference materials have been prepared for validation and calibration of specic real-time quantitative PCR (qPCR) assays. Because digital PCR is a primary measurement method that does not require a calibrator and is in principle a counting technique, digital PCR has been used for property value assignment of some DNA reference materials with traceability to the International System of Units. 2125 Reference materials have also been prepared to assess biases in specic steps of a qPCR process such as biases that may arise in Received: December 19, 2016 Accepted: October 2, 2017 Published: October 2, 2017 Article pubs.acs.org/ac © XXXX American Chemical Society A DOI: 10.1021/acs.analchem.6b05032 Anal. Chem. XXXX, XXX, XXXXXX Cite This: Anal. Chem. XXXX, XXX, XXX-XXX

Transcript of Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet...

Page 1: Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo

Interlaboratory Reproducibility of Droplet Digital Polymerase ChainReaction Using a New DNA Reference Material FormatLeonardo B. Pinheiro,*,† Helen O’Brien,‡ Julian Druce,§ Hongdo Do,∥ Pippa Kay,⊥ Marissa Daniels,#

Jingjing You,× Daniel Burke,† Kate Griffiths,† and Kerry R. Emslie†

†National Measurement Institute (NMI), Lindfield, Sydney, New South Wales 2070, Australia‡Research and Development, Australian Red Cross Blood Service, Kelvin Grove, Queensland 4059, Australia§Victorian Infectious Diseases Reference Laboratory, Melbourne, Victoria 3000, Australia∥Olivia Newton-John Cancer Research Institute, Translation Genomics and Epigenomics Laboratory, Heidelberg, Victoria 3084,Australia⊥Agri-Bio Molecular Genetics, Biosciences Research Division, Bundoora, Victoria 3083, Australia#The Prince Charles Hospital University of Queensland, Thoracic Research Centre, Chermside, Queensland 4032, Australia×Save Sight Institute, Sydney Eye Hospital, Sydney Medical School, University of Sydney, Sydney, New South Wales 2000, Australia

*S Supporting Information

ABSTRACT: Use of droplet digital PCR technology (ddPCR) is expandingrapidly in the diversity of applications and number of users around the world.Access to relatively simple and affordable commercial ddPCR technology hasattracted wide interest in use of this technology as a molecular diagnostic tool.For ddPCR to effectively transition to a molecular diagnostic setting requiresprocesses for method validation and verification and demonstration ofreproducible instrument performance. In this study, we describe thedevelopment and characterization of a DNA reference material (NMINA008 High GC reference material) comprising a challenging methylatedGC-rich DNA template under a novel 96-well microplate format. A scalableprocess using high precision acoustic dispensing technology was validated toproduce the DNA reference material with a certified reference value expressedin amount of DNA molecules per well. An interlaboratory study, conductedusing blinded NA008 High GC reference material to assess reproducibility among seven independent laboratories demonstratedless than 4.5% reproducibility relative standard deviation. With the exclusion of one laboratory, laboratories had appropriatetechnical competency, fully functional instrumentation, and suitable reagents to perform accurate ddPCR based DNAquantification measurements at the time of the study. The study results confirmed that NA008 High GC reference material is fitfor the purpose of being used for quality control of ddPCR systems, consumables, instrumentation, and workflow.

Over the past few years, use of digital polymerase chainreaction (PCR) technology has rapidly expanded and

diversified into a multitude of areas within life sciences. Theexpansion in application of digital PCR technology is evidentfrom literature reports in various areas of research includingpathogen detection,1−4 monitoring of food and water safety,5−7

and microbial ecology.8 Because digital PCR can resolve smalldifferences in the amount of nucleic acids, the largest uptake ofthis technology is in clinical research such as biomarkerquantification,9−11 detection of genetic variants in cancerpatients,12,13 detection of copy number alterations in stemcells,14 monitoring transplant recipients,15 quantification ofmassively parallel sequence libraries,16−18 and genomeediting.19,20 Recognition that digital PCR can provideunprecedented levels of precision, accuracy, and resolutionfor quantification of nucleic acids, together with developmentand availability of affordable instrumentation, have been themain reasons for the rapid expansion of digital PCR.

Transitioning digital PCR technology from a researchlaboratory to a clinical setting requires processes for methodvalidation and verification and demonstration of methodreproducibility and instrument performance.Many DNA reference materials have been prepared for

validation and calibration of specific real-time quantitative PCR(qPCR) assays. Because digital PCR is a primary measurementmethod that does not require a calibrator and is in principle acounting technique, digital PCR has been used for propertyvalue assignment of some DNA reference materials withtraceability to the International System of Units.21−25 Referencematerials have also been prepared to assess biases in specificsteps of a qPCR process such as biases that may arise in

Received: December 19, 2016Accepted: October 2, 2017Published: October 2, 2017

Article

pubs.acs.org/ac

© XXXX American Chemical Society A DOI: 10.1021/acs.analchem.6b05032Anal. Chem. XXXX, XXX, XXX−XXX

Cite This: Anal. Chem. XXXX, XXX, XXX-XXX

Page 2: Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo

measurement of methylated DNA after bisulphite conversion.26

To date, such reference materials often comprise a plasmidDNA solution with an assigned copy number concentration orcopy number ratio.Even if a validated PCR protocol is followed, instrument

related factors can introduce error or bias to results. Accuratethermal cycling conditions during PCR amplification are criticalfor reproducible results. Inaccurate thermal cycling temper-atures or poor uniformity of temperature across heating blockunits can result in no amplification or low amplificationefficiency in wells where optimal temperatures are not reached.PCR assays targeting guanine and cytosine (GC)-rich DNAsequences are especially prone to inefficient amplificationresulting from suboptimal denaturation and annealing temper-atures.27 This was highlighted in a recent study involving aqPCR based measurement targeting a methylated, GC-richDNA sequence in which diagnostic results were incorrectlyinterpreted due to instrument derived denaturation temper-ature differences between samples within 96-well plates.28

In this study, we describe the development, characterization,and interlaboratory validation of a DNA reference materialunder a 96-well format designed for digital PCR instrumentvalidation, monitoring of instrument performance, and trainingin digital PCR instrumentation and procedures. The referencematerial is produced using a “scalable” high precision roboticacoustic droplet ejection (ADE) (Labcyte) technology.29 ADEis an ultrasound-based liquid ejection technology with severaladvantages over traditional pipetting techniques. ADE does notrequire tips or pins29 for dispensing and is capable of deliveringprecise nanoliter volumes of solution, thus utilizing muchhigher DNA concentrations than required for dispensing thesame amount of DNA using traditional microliter volumetechniques. Together, these features reduce the risk of DNAloss due to adsorption to the surface of the pipet tip or storagetube. Unlike many other DNA reference materials, thereference value is expressed in DNA copy number amountper well and assigned by droplet digital PCR (ddPCR). DNAreference materials in this format, in addition to verifyinginstrument performance, could also be used to supportanalytical validation of specific PCR methods.

■ EXPERIMENTAL SECTIONDNA Materials. The procedure used for production of

DNA materials is described in Supporting Information S-1.Briefly, DNA materials were produced by end point PCRamplification of human genomic DNA (Human placental DNA,Sigma-Aldrich product number D 4642) followed bydeproteination, ethanol precipitation, high-pressure liquidchromatography (HPLC), fractionation and ultrafiltration.CDKN2A_550 is a 550 base pair (bp) amplicon correspondingto part of the human cyclin-dependent kinase inhibitor 2A(CDKN2A) gene promoter region. CDKN2A_183 wasprepared by digestion of CDKN2A_550 using MspI restrictionenzyme (New England BioLabs) followed by the samepurification procedure used for materials produced by end-point PCR amplification. TLX3_304 is a 304 bp ampliconcorresponding to part of the human T-cell leukemia homeobox3 gene promoter region. Methylated CDKN2A_550(mCDKN2A_550 ) a nd me t h y l a t e d TLX3_304(mTLX3_304) were prepared by in vitro methylation usingM.SssI CpG Methyl transferase (New England BioLabs).NMIA NA008 Copy Number Verification Plate, High GCDNA reference material (hereafter referred to as NA008 High

GC reference material) consists of a defined number of copiesof mTLX3_304 acoustically dispensed into each well of a 96-well PCR microplate. All dilutions of DNA were performedusing 1× TE0.1 buffer (10 mM Tris, 0.1 mM EDTA, pH 8.0).

Instrumentation. Digital PCR. QX100 and QX200 DropletDigital PCR systems (Bio-Rad Laboratories Pty Ltd., Australia)including DG8 cartridges for droplet generators (Bio-RadLaboratories Pty Ltd., Australia) were used by all laboratoriesparticipating in the interlaboratory study for ddPCR measure-ments, except for one laboratory (Laboratory 4) which used anautomated droplet generator (AutoDG) and DG32 cartridgesfor AutoDG (Bio-Rad Laboratories Pty Ltd., Australia). MostDroplet Digital PCR systems included a C1000 Touch ThermalCycler (Bio-Rad Laboratories Pty Ltd., Australia) for thermalcycling of droplets in ddPCR assays, except for threelaboratories participating in the interlaboratory study; Labo-ratory 2 used an Arktik Thermal Cycler Type 5020 (ThermoScientific) and Laboratories 5 and 6 used a T100 ThermalCycler (Bio-Rad Laboratories Pty Ltd., Australia). Forexperimental work to identify DNA materials sensitive tothermal cycler temperature uniformity, Mastercycler epgradient S (Eppendorf), Nexus (Eppendorf), and C1000(Bio-Rad Laboratories Pty Ltd., Australia) thermal cyclerswere employed. The first two of these individual thermalcyclers were chosen as they were known to have inaccuratethermal profile from prior calibration verifications usingtemperature probes (data not shown). ddPCR data acquisitionwas performed using QuantaSoft software (Bio-Rad Labo-ratories Pty Ltd., Australia). The procedure used for digitalPCR quantification of DNA materials using QX100 DropletDigital PCR system is given in Supporting Information S-2.Primers and probes sequences used for digital PCR assays ofDNA materials are given in Table S-1.

Liquid Handling. Acoustic liquid dispensing of DNA intowells of a 96-well plate was undertaken using an automatedAccess workstation (Labcyte Inc.) comprising an Echo 550Liquid Handler (Labcyte Inc.), a 4-axis Precise PF-400 Robotplate handler, and a PlateLoc Thermal Microplate Sealer(Agilent Technology, Australia) inside a custom built HEPAfiltered air enclosure (LabSystems Pty Ltd., Australia). ATempo automation control software (Labcyte Inc.) was usedfor protocol workflow definition and running the Accessworkstation. For ddPCR quantification of DNA in wells of a96-well plate during development work and batch productionof the DNA reference material, ddPCR mix was dispensed intowells using a CAS-1200N liquid handler (Corbett). Calibratedpipets and calibrated analytical balance XP205 (Mettler-Toledo) with 0.01 mg resolution were used for gravimetricdilutions of DNA and preparation of digital PCR mixes.

High-Pressure Liquid Chromatography (HPLC). A prom-inence SPD-M20A HPLC (Shimadzu) fitted with a Gen-PakFAX column (Waters) was used to fractionate ampliconrestriction fragments by ion-exchange HPLC. QuantitativeHPLC measurements of DNA materials were performed usingan Ultimate 300 rapid separation liquid chromatography(RSLC) nano system (Dionex) with a PepSwift monolithiccolumn (Dionex). Details of analytical methods used for bothion-exchange HPLC fractionation and quantitative HPLCmeasurements of DNA materials have been previouslydescribed.30

Capillary Electrophoresis. Amplicon size (bp) was verifiedon a 2100 Bioanalyzer (Agilent Technologies, Australia) using aDNA 1000 kit as per manufacturer’s instructions.

Analytical Chemistry Article

DOI: 10.1021/acs.analchem.6b05032Anal. Chem. XXXX, XXX, XXX−XXX

B

Page 3: Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo

Optical Microscopy. An optical microscope (LeicaDM6000M) with digital CCD camera (Leica DFC490) and 1μ-Slide VI flat uncoated microscopy chambers (IBID Germany)was used to image droplets generated from the ddPCR systemdroplet generators (Bio-Rad) to determine droplet volume aspreviously described.31

Acoustic Dispensing of DNA Combined with ddPCRQuantification. The procedure of combining high precisionacoustic dispensing of a DNA solution with ddPCRquantification of number of dispensed DNA moleculescomprised the following steps: (A) quantification andconfirmation of identity of high concentration stock of DNAmaterial, gravimetric dilution of DNA to a Source Stock attargeted dispensing concentration, and confirmatory digitalPCR quantification of Source Stock, (B) transfer of SourceStock to individual wells of an Echo qualified 384-well SourcePlate (Labcyte Inc.), robotic-driven acoustic dispensing ofSource Stock into 96-well × 0.2 mL PCR microplates(BIOPlastics B, Landgraaf Netherlands) and robotic-drivenfoil sealing of dispensed plates under HEPA filter airenvironment, (C) ddPCR quantification of the number ofDNA molecules present in individual wells of dispensed PCRmicroplates (Figure 1).The acoustic dispensing procedure involved transfer of

defined nanoliter volumes of Source Stock into individual wells

of each 96-well PCR microplate. The acoustic dispensing liquidhandling system ejects single or multiple 2.5 nL droplets (≤8%coefficient of variation (CV) accuracy; ≤5% CV precision)(manufacturer’s specification) from a Source Plate into thewells of an inverted plate (Destination Plate) that is suspendedabove the Source Plate. The nanoliter volume of DNA solutionevaporates immediately after acoustic dispensing, resulting inPCR microplates containing dry DNA molecules at the bottomof individual wells. The sealed plates are stored at −20 °C untilused.To assay DNA in a Destination Plate, a defined volume

between 20 and 25 μL of ddPCR mix containing appropriateprimers and probe was either manually pipetted or roboticallydispensed (CAS-1200N) into each well of the microplate. TheDNA contents in each well were then redissolved by incubatingthe Destination Plate at 62 °C for 2 min, followed by mixingusing a plate vortex (MixMate PCR 96 Eppendorf) andcentrifugation (Centrifuge 5430 Eppendorf) to collect thesolution to the bottom of the wells. The QX100/200 ddPCRworkflow was then followed for quantification of the number ofDNA molecules present in the wells. A simplified DNAredissolving procedure, which excluded the heat incubationstep, was used for the interlaboratory study. The simplifiedmethod consisted of pipetting 25 μL of ddPCR mix containingappropriate primers and probes into each well of a microplate,

Figure 1. Schematic workflow of combining acoustic dispensing with ddPCR quantification used in the production of NA008 High GC referencematerial. (A) High concentration DNA material is quantified by HPLC and expected sequence identity and size verified by Sanger DNA sequencingand capillary electrophoresis, respectively. High concentration DNA material is gravimetrically diluted to generate Source Stock at suitableconcentration for acoustic dispensing. Source Stock concentration of DNA material is verified by ddPCR. (B) Samples from Source Stock of DNAmaterial are transferred to Labcyte Echo qualified Source Plate wells. Single 2.5 nL aliquots of Source Stock of DNA material are acousticallydispensed into individual wells of 96-well PCR microplates and foil sealed using a robotic-driven automated workstation under HEPA filter airenclosure. (C) Preparations of ddPCR mix containing primers and probe are either manually pipetted or robotically dispensed into each well of themicroplate. The DNA contents in each well are redissolved, mixed, and collected to the bottom of wells by centrifugation and quantified followingstandard BioRad QX100/200 ddPCR workflow. The number of positive droplets and number of accepted droplets obtained from BioRadQuantaSoft software output is used to calculate the number of DNA molecules in each well using an NMI-designed Microsoft Excel proforma.

Analytical Chemistry Article

DOI: 10.1021/acs.analchem.6b05032Anal. Chem. XXXX, XXX, XXX−XXX

C

Page 4: Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo

mixing by pipetting the reaction solution up and down fivetimes, brief centrifugation of the plate to collect the solution atthe bottom of each well followed by the standard QX100/200ddPCRworkflow.

■ RESULTS AND DISCUSSIONIdentification of DNA Materials Sensitive to Thermal

Cycler Temperature Uniformity. The first evidence of aneffect of well position on results from ddPCR quantificationwas noticed when analyzing mCDKN2A_550 using an assay(Assay F) targeting a 70 bp sequence with 71% GC contentand 9 CpG methylation sites (Table S-1 and Figure S-1).Results from ddPCR quantification of mCDKN2A_550 usingassay F showed poor repeatability among replicate assaysplaced in wells positioned at the edge of the thermal cyclerheating block. To further investigate the thermal cycler wellposition effect in the assays, a bulk ddPCR mix containingprimers, probe, and mCDKN2A_550 sufficient for 96 wells wasprepared, manually dispensed across an entire 96-well plate (3replicate plates), and analyzed using ddPCR. Thermal cyclingof each plate was undertaken on a different thermal cycler.Depending on the thermal cycler unit used, the measured copynumber concentration showed up to 56% relative standarddeviation (RSD) across the 96 wells of the plate and up to 4-fold difference between adjacent wells at the edge of the heatingblock. The thermal cycler unit with the lowest well to wellvariation (thermal cycler C) resulted in 3.0% RSD in themeasured concentration of mCDKN2A_550 across the entire96 wells (Figure S-2).To verify if the thermal cycler well position effect could be

detected using other high GC content DNA, a second series ofassays was performed using mTLX3_304. First, a preparationof mTLX3_304 was diluted and then quantified by ddPCR toapproximately 1 × 106 copies per μL using TLX3 assay 1 thattargets a 112 bp sequence of TLX3_304 with 66% GC contentand 13 CpG methylation sites (Supporting Information FigureS-3). The standard procedure for acoustic dispensing followedby digital PCR quantification was used to dispense 10 nL (four2.5 nL droplets) of mTLX3_304 preparation into each well of a96-well PCR microplate. Digital PCR quantification ofmTLX3_304 plates using TLX3 assay 1 (Table S-1) showeda similar thermal cycler well position pattern as detected formCDKN2A_550. Thermal cycler A, which previouslyproduced the poorest uniformity in measured values acrosswells containing mCDKN2A_550, showed 71% RSD inmeasured DNA concentration across the plate with consistentlylower values around the edge of the plate whereas there was a3.6% RSD across the plate when using thermal cycler C (FigureS-4).These results prompted us to consider the possibility of

preparing reference plates predispensed with a defined numberof copies of methylated DNA material to be used for verifyingthermal cycler uniformity and reproducibility of ddPCRsystems.Validation of Acoustic Dispensing for Production of

DNA Reference Materials. Validation of acoustic dispensingof DNA was performed using accurately quantified preparationsof CDKN2A_183. A HPLC fraction corresponding toCDKN2A_183 was collected and quantified to an estimatedconcentration of 1.05 × 1010 copies per μL by quantitativeHPLC. CDKN2A_183 was gravimetrically diluted to producethree Source Stocks: levels 1, 2, and 3 containing 400, 40, and 4DNA molecules per nL, respectively. The standard procedure

for acoustic dispensing followed by digital PCR quantificationwas used to dispense either 2.5, 5, or 10 nL of CDKN2A_183preparation into each well of a 96-well PCR microplate inreplicates of seven. This resulted, theoretically, in sets of sevenwells containing either 4 000, 2 000, 1 000, 400, 200, 100, 40,20, or 10 copies in each well. Three replicate plates weredispensed. The dispensing format was designed to assess theacoustic dispensing precision of both single and multipleejections of the 2.5 nL volume (2.5, 5, and 10 nL) for deliveryof different numbers of DNA molecules. Quantification ofCDKN2A_183 in the wells was performed by ddPCR usingCDKN2A assay F as the 183 bp fragment spans the assay Ftarget region within CDKN2A_550.For data analysis, wells with less than 10 000 accepted

droplets were excluded. This comprised a complete columnplus two additional wells on one of the plates. Data analysis ofthe number of CDKN2A_183 molecules in each well across thethree plates indicated excellent linearity across the range from10 to 4 000 molecules (r2 = 0.99995) between the nominalnumber expected assuming an acoustic ejection volume of 2.5nL (manufacturer’s specification) and the number measured byddPCR (Supporting Information Figure S-5). The relativeexpanded uncertainty of the number of CDKN2A_183molecules in each well ranged from 7.0% for 2 500 dispensedmolecules to 48% for 10 DNA molecules (Figure 2). The

increase in uncertainty observed when dispensing very lownumbers of DNA molecules is predominantly due to stochasticeffects31,32 rather than increased variability in the acousticdispensing process. The results indicate that either single ormultiple 2.5 nL aliquots can be acoustically dispensed with highprecision allowing for a flexible dispensing design format using96-well PCR microplates. Overall, the results demonstrated thatthe process of acoustic dispensing of DNA molecules incombination with quantification of dispensed molecules usingddPCR is fit for the purpose of producing DNA referencematerials.

Figure 2. ddPCR analysis of acoustically dispensed CDKN2A_183.Each data point corresponds to the average of either 6 or 7 replicatesfor each of 3 plates except for the data point with a nominal 10 copiesper well which was based on 2 plates. Error bars represent expandeduncertainty calculated by multiplying the combined standarduncertainty by a coverage factor of between 2.10 and 2.18 dependingon the number of wells analyzed. This provides a level of confidence of95% in the expanded uncertainty. Row (A) corresponds to the numberof acoustically dispensed droplet ejections and row (B) corresponds tothe Source Stock dilutions used: levels 1, 2, and 3 containing 400, 40,and 4 DNA molecules per nL, respectively.

Analytical Chemistry Article

DOI: 10.1021/acs.analchem.6b05032Anal. Chem. XXXX, XXX, XXX−XXX

D

Page 5: Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo

Production and Characterization of AcousticallyDispensed DNA Reference Material. mTLX3_304 wasselected for batch production of DNA reference materialbecause, in addition to its generic potential as quality controlfor digital PCR, mTLX3_304 shows high sensitivity to theaccuracy and uniformity of thermal cycler temperature. Forpreparation of the acoustic dispensing Source stock, apreparation of mTLX3_304 was gravimetrically diluted toapproximately 1.2 × 107 copies per μL, with concentrationverified by ddPCR using TLX3 assay 1. This dilution waschosen with the objective of acoustically dispensing single 2.5nL aliquots for delivery of approximately 30 000 copies ofmTLX3_304 into each well of a 96-well PCR microplate. Thetarget of approximately 30 000 DNA molecules per well wasselected to minimize the contribution to uncertainty from thePoisson model assuming the ddPCR would compriseapproximately 20 000 droplets31 and, thus, optimize conditionsfor identifying instrument-related factors that could affectddPCR performance. The dispensing workflow of a single 2.5nL aliquot per well maximizes the number of plates that can bedispensed from a single well of Source Stock in the sourceplate. Acoustic dispensing was programmed to consecutivelydispense a single aliquot in each well of each columncommencing at A01, moving down column 1 to H01 andthen returning to A02 etc until the last aliquot was dispensedinto the final well, H12. An automated workflow wasprogrammed for the Access workstation using the Temposoftware for dispensing a batch of 300 microplates in sixsequential sets of 50 plates. Each set of 50 plates was dispensedfrom a single Source well. The automated dispensing processinvolved barcoding and foil heat sealing each plate afteracoustically dispensing DNA into each of the 96 wells. Afterdispensing, the batch of plates of NA008 High GC referencematerial was transferred to storage at −20 °C.Homogeneity Assessment. The first and one of the most

important steps for determining success in the production of abatch of reference materials is homogeneity assessment.Assessment of the interplate homogeneity of the acousticallydispensed batch of NA008 High GC reference material wasconducted in accordance with ISO Guide 35.33 The measurandwas defined as the average number of mTLX3_304 DNAmolecules per well across the eight wells of a column.Homogeneity data was obtained from ddPCR quantificationof the number of mTLX3_304 molecules present in each of 96-wells using seven plates randomly selected from the batch.Initial assessment of the data revealed a slightly higher numberof molecules (approximately 10% higher) detected for the firstfew wells of the first column of each dispensed plate. The trendpossibly resulted from a deviation in the volume of theacoustically dispensed aliquot caused by effects from fluidsurface tension on the meniscus in the source well at the start ofthe dispensing cycle for each individual plate (Labcyte, personalcommunication).In order to assess homogeneity while taking into account the

dispensing trend, data analysis was performed by dividing eachplate into four quadrants according to the dispensing sequenceof columns. Plate quadrants 1, 2, 3, and 4 comprised columns 1to 3, 4 to 6, 7 to 9, and 10 to 12, respectively. ANOVA of datafrom each of the four quadrants was used to separate thevariation due to precision (within-plate relative standarddeviation, Sw,rel) and homogeneity (between-plate relativestandard deviation, Sb,rel) and these factors were used for therelative standard uncertainties for precision and homogeneity,

respectively. The relative standard uncertainties for bothhomogeneity (ub,rel) and precision (uw,rel) for all four quadrantswere below 3% (Table 1).

Stability Assessment. Stability of the produced batch ofmaterial was assessed using a classical stability study designfollowing ISO guide 3533 recommendations. Short-termstability was conducted at 20 and 40 °C for up to 4 weeks tosimulate the most extreme conditions that may arise undertransport. Long-term stability at −20 °C has been undercontinuous monitoring for 76 weeks at the time of theinterlaboratory study. Both short- and long-term stability datawere obtained from ddPCR quantification of the number ofDNA molecules per well from three randomly selected plateunits at each time point tested. As for assessment ofhomogeneity, data was collected and analyzed across the fourquadrant regions comprising columns 1 to 3, 4 to 6, 7 to 9, and10 to 12. Results were assessed for trends in the measurednumber of DNA molecules using a t test and the regressionslope over time. There was no significant linear trend over timefor any of the quadrants when plates were stored at either 20 or40 °C for up to 4 weeks (p-values ranging from 0.128 to 0.779).Estimates for the relative standard uncertainty for short-termstability at 40 °C for 4 weeks, an unlikely worst possibletransport condition, were below 1.2% per week. There was nosignificant linear trend over time for any of the quadrants whenplates were stored at −20 °C for 76 weeks (p-values rangingfrom 0.257 to 0.852). ANOVA evaluation of the long-termstability data revealed a significant time point effect (p-values<0.001 for all quadrants). Since a classical approach was used,the long-term data also incorporates different equipment andreagent conditions (QX100 and QX200 instruments, twodifferent C1000 thermal cyclers, different reagent batches,EvaGreen and Taqman assays). We concluded the long-termdata, while indicating the reference material was stable, wasreflecting [time+equipment]-different intermediate precision34

and a term for this (uI(TE),rel) was included in the uncertaintybudget to capture this component of uncertainty.

Property Value. Measurement uncertainties were estimatedaccording to international guidelines.35 The measurementuncertainty associated with batch characterization comprisesseveral aspects related to the procedure used to quantify thenumber of DNA molecules in plate wells. It includes theuncertainty associated with method precision (uw, rel), as wellsas uncertainty associated with gravimetric dilutions (uva,rel) andType B uncertainty of a droplet volume (uvd, rel), as previouslydescribed.31 The measurement uncertainty associated withhomogeneity (ub,rel) accounts for the variation between plateunits, while the measurement uncertainty associated with short-

Table 1. Factors Contributing to the Relative StandardUncertainty Estimate for NA008 High GC ReferenceMaterial

platecolumns

uw,rela

(%)uva,rel

b

(%)uvd, rel

c

(%)ub,rel

d

(%)usts, rel

e

(%)urepr,rel

f

(%)

1 to 3 2.8 0.17 0.96 1.1 0.7 3.44 to 6 2.2 0.17 0.96 1.1 0.9 3.27 to 9 1.2 0.17 0.96 2.3 1.2 3.610 to 12 1.8 0.17 0.96 2.6 0.8 3.2

aMethod precision. bAssay volume. cDroplet volume. dHomogeneity.eShort-term stability. f[time + equipment]-different intermediateprecision.

Analytical Chemistry Article

DOI: 10.1021/acs.analchem.6b05032Anal. Chem. XXXX, XXX, XXX−XXX

E

Page 6: Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo

term stability (usts, rel) accounts for transport of the material.The measurement uncertainty associated with [time+equip-ment]-different intermediate precision (uI(TE),rel) accounts forlong-term stability and intermediate precision when measure-ments are made at different times and using different batches ofreagents and instruments. The relative standard uncertaintieswere combined to give the combined relative standarduncertainty of the property value (eq 1). A summary of factorscontributing to the relative standard uncertainty estimate arepresented in Table 1.

=

+ + + + +

u

u u u u u u

T, rel

w,rel2

va,rel2

vd,rel2

b,rel2

sts,rel2

I(TE),rel2

(1)

The combined standard uncertainty for each quadrant of theplate was expanded to provide a level of confidence of 95%using a coverage factor calculated from the Welch-Satterthwaiteequation. The property value assigned to each well in NA008High GC reference material as well as coverage factors andexpanded uncertainties is given in Table 2. Results fromcharacterization and assessment of NA008 High GC referencematerial indicated the material was suitable for use in aninterlaboratory study.Interlaboratory Reproducibility Using the Produced

DNA Reference Material. One of the objectives of aninterlaboratory study is to assess the performance oflaboratories using samples of which the concentration orquantities have been assigned. Reproducibility is defined byISO 5725-136 as the precision under conditions where testsresults are obtained with the same method on identical testitems in different laboratories with different operators usingdifferent equipment.An interlaboratory study of the NA008 High GC reference

material was conducted with participation of seven laboratories.Each laboratory received a set of materials consisting of threeblinded NA008 High GC reference material plates, stocks ofprimers, and probe mix and disposables sufficient to run theassays. All participating laboratories were required to performddPCR using all 96 wells of each plate following prescribedinstructions (Supporting Information S-3). The data generatedusing QX100/200 droplet readers (Bio-Rad) and QuantaSoftsoftware from all plates was submitted to NMI. The number ofpositive droplets and number of accepted droplets in the rawdata output were used to calculate the number of DNAmolecules in each well from Poisson statistics using an NMI-designed Microsoft Excel proforma. The prescribed 25 μLvolume of ddPCR mix per well, the type of ddPCR Supermixused for the assays, and associated average droplet volume weretaken into account in calculation of DNA amount per well. Onthe basis of droplet volume measurements performed at NMI,average droplet volumes of 0.795 nL and 0.833 nL were usedfor assays using Supermix for probes no dUTP and Supermixfor probes containing dUTP, respectively.

Initial assessment of data was based primarily on the averagenumber of DNA molecules measured per well in each columnof the plate (n = 8 for a complete column of wells) and thecorresponding standard deviation. Wells for which ddPCRassays resulted in less than 10 000 accepted droplets wereexcluded from the assessment. A minimum of six wells in acolumn was required for inclusion of the column in dataassessment. Using these criteria, all columns in each of threeplates (total of 36 columns) were included in data assessedfrom Laboratories 4, 5, and 7 while one column from a singleplate was excluded for both Laboratories 1 and 3, one columnfrom each of two plates was excluded for Laboratory 2 and 11columns across all three plates excluded for Laboratory 6(Table S-2).Mandel’s h and k statistics were used to assess between-

laboratory and within laboratory consistency, respectively.36,37

For each laboratory, i, Mandel’s h statistic, hi, was determinedfor each column of the plate (hi,c) as the ratio of the differencein the average number DNA molecules measured per well forthat column in a set of up to three plates (yi,c) and the mean ofall data sets for that column from all laboratories (yc), and thestandard deviation of the means from all data sets from alllaboratories (sdm) (eq 2).

= −

hy y

sdmii

,c,c c

(2)

For each laboratory, i, Mandel’s k statistic was determined foreach column (ki,c) as the quotient of the pooled standarddeviation for a set of up to three plates from that laboratory andthe mean standard deviation of all sets of data from alllaboratories. Mandel’s h and k plots were prepared for eachcolumn of the plate (Figure 3A,B). In agreement with the initialvisual inspection of raw data, Mandel’s h plot revealed thatLaboratory 6 had large variability in data across the columnswhile Mandel’s k plot reflected poor repeatability within somecolumns for Laboratories 2, 5, 6, and 7. Laboratory 4, the onlylaboratory using an AutoDG instrument for generating thedroplet emulsions, demonstrated the best repeatability (Figure3B).Investigation of the 15 data points that fell outside the 5%

significance level for Mandel’s k statistic (Figure 3B) revealedthat, except for data points from Laboratory 6, a very lownumber of copies of mTLX3_304 was measured (between 0and 20 000 molecules) in either one or two wells within the setof eight wells in a column. This resulted in a large standarddeviation across the column and, consequently, a largeMandel’s k statistic. As only 25% of these wells were at theedge of the plate and, in addition, 35% of these wells showedtwo positive populations or a significant downshift influorescence intensity for the negative and positive dropletpopulations, we concluded that the low measured copy numberwas not due to thermal cycler edge effect but was more likelydue to a technical problem, for example, formation of thedroplets. For further data analysis, wells with a measured

Table 2. NA008 High GC Reference Material Property Values, Coverage Factors, and Expanded Uncertainties

platecolumns

reference value(copies/well)

combined relative standarduncertainty (%)

coverage factor(95%)

relative expandeduncertainty (%)

expanded uncertainty(copies/well)

1 to 3 32 000 4.6 2.07 9.6 3 0004 to 6 31 000 4.2 2.08 8.8 2 6007 to 9 30 500 4.7 2.10 9.9 2 90010 to 12 30 200 4.7 2.07 9.7 2 900

Analytical Chemistry Article

DOI: 10.1021/acs.analchem.6b05032Anal. Chem. XXXX, XXX, XXX−XXX

F

Page 7: Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo

number of mTLX3_304 molecules either less than 75% ormore than 125% of the assigned property value (1.3% of thetotal wells analyzed) were excluded (Figure 3C,D). Followingexclusion of these wells, all laboratories, except for Laboratory6, demonstrated good precision (Figure 3D). Cochran’s testrevealed that Laboratory 6 results for columns 1 and 4 werestragglers (C = 0.37 and 0.39; Cochran’s 5% critical value =0.338) and for columns 6 and 9 were outliers (C = 0.41 and0.57; Cochran’s 1% critical value = 0.391). Because of thenumber of excluded columns (Table S-2) and poor precisionfor Laboratory 6, this Laboratory was excluded from theestimation of repeatability and reproducibility variance.The repeatability relative standard deviation, sr,rel, and

reproducibility relative standard deviation, sR,rel, for the averagenumber of mTLX3_304 per well within a column in eachquadrant were between 2.6 and 3.4% and between 3.8 and4.4%, respectively. The repeatability standard deviation wasonly slightly higher than the uncertainty associated withmethod precision but less than the uncertainty associatedwith intermediate precision (Table 1), while the reproducibilitywas slightly higher than the uncertainty associated withintermediate precision (Table 1).For feedback to laboratory participants, performance accept-

ance criteria were established, applied to each plate, andpresented in a proforma (Supporting Information S-7). A platepassed the accuracy criterion if the average number of copiesper well in at least 10 columns fell within the expandeduncertainty of the assigned property value. A plate passed theprecision criterion if the relative standard deviation of thenumber of copies per well was equal to or less than 7% for atleast 10 columns. The 7% precision limit was chosen based onthe estimated repeatability standard deviation multiplied by twoto cover for 95% confidence interval. The proforma includes aplot of the average number of DNA molecules per well againstplate column number and highlights excluded wells. In total, 17

of the 21 plates analyzed in the interlaboratory study passedboth the accuracy and precision criteria.One plate from Laboratory 3 failed the accuracy criterion as

the measured values for three columns were lower than theexpanded uncertainty value (Figure S-6). All three plates fromLaboratory 3 returned measured values for the number of DNAmolecules per well which were close to the lower limit of thereference range across the entire set of plates. This consistentlylow measurement could have resulted from pipettinginaccuracy. Accuracy of the measured number of molecules ina well can be significantly affected by accuracy of the reactionmix volume used to redissolve the DNA in the well. Onemicroliter inaccuracy of the prescribed 25 μL reaction mixvolume for the assays results in 4% inaccuracy in the measurednumber of DNA molecules per well.For Laboratory 6, two plates had insufficient data for

assessment (either 20 or 49 wells excluded) while theremaining plate failed both the accuracy and precision criteria.In total, 26% of the wells in these three plates had a lownumber of accepted droplets suggesting an issue with theddPCR measurement workflow in that laboratory. Laboratory 6assays were performed by an inexperienced user of ddPCR(personal communication).Overall the data obtained from the interlaboratory study,

using a challenging methylated template DNA, demonstratedthat most of the participating laboratories had appropriatetechnical competency, fully functional instrumentation, andsuitable reagents to perform accurate ddPCR based DNAquantification measurements at the time of the study. Mostimportantly, the study confirmed the NA008 High GCreference material to be fit for the purpose of being used forquality control of ddPCR systems, consumables, instrumenta-tion, and workflow.

Figure 3.Mandel’s between laboratory consistency statistic, h, (A, C) and within laboratory consistency statistic, k, (B, D) for the average number ofmeasured mTLX3_304 molecules per well within a column of the NA008 High GC reference material plate grouped by laboratory. Wells for whichddPCR assays resulted in less than 10 000 accepted droplets were excluded from the assessment (A, B) and, in addition, wells with a measurednumber of mTLX3_304 molecules either less than 75% or more than 125% of the assigned property value were excluded (C, D). Data for eachlaboratory is plotted for each of the 12 columns in order from column 1 to column 12. The solid and dotted lines are the 1% and 5% significancelevels, respectively. Each laboratory analyzed three plates providing up to three data points per column. A minimum of six wells in a column wasrequired for inclusion of the column in data assessment.

Analytical Chemistry Article

DOI: 10.1021/acs.analchem.6b05032Anal. Chem. XXXX, XXX, XXX−XXX

G

Page 8: Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo

■ ASSOCIATED CONTENT*S Supporting InformationThe Supporting Information is available free of charge on theACS Publications website at DOI: 10.1021/acs.anal-chem.6b05032.

Preparation and in vitro methylation of ampliconmaterial, quantification of DNA materials using Bio-Rad QX100 Droplet Digital PCR system, prescribedplate assay method for the interlaboratory study usingNA008 High GC reference material, primers and probesused in the study, interlaboratory data excluded fromassessment of reproducibility, variation in measured copynumber concentration from 96-well ddPCR assaystargeting mCDKN2A_550 (assay F) using differentthermal cyclers, map of TLX3_304 showing % GCcontent and relative position of TLX3 Assay 1, variationin measured copy number concentration from 96-wellddPCR assays (TLX3 Assay 1) targeting mTLX3_304using different thermal cyclers, linearity between theexpected number of acoust ica l ly d ispensedCDKN2A_183 molecules and the number of copiesmeasured by ddPCR, interlaboratory results fromanalysis of NA008 High GC reference material, examplesof NA008 High GC reference material data analysisproforma and examples from pass (A) and fail (B) plateanalysis results, and dMIQE checklist (PDF)

■ AUTHOR INFORMATIONCorresponding Author*E-mail: [email protected]. Phone: +612 84673735.

ORCIDLeonardo B. Pinheiro: 0000-0002-6854-8465Author ContributionsThe manuscript was written through contributions of allauthors. All authors have given approval to the final version ofthe manuscript.

NotesThe authors declare no competing financial interest.

■ ACKNOWLEDGMENTSWe thank Michael Forbes-Smith for his invaluable help withinstrument troubleshooting during the production of acousti-cally dispensed DNA materials. The National MeasurementInstitute declares that it has no competing interests and byconducting this study does not imply endorsement of thetechnologies described in the manuscript.

■ REFERENCES(1) Devonshire, A. S.; Honeyborne, I.; Gutteridge, A.; Whale, A. S.;Nixon, G.; Wilson, P.; Jones, G.; McHugh, T. D.; Foy, C. A.; Huggett,J. F. Anal. Chem. 2015, 87, 3706−3713.(2) Pond, M. J.; Nori, A. V.; Patel, S.; Laing, K.; Ajayi, M.; Copas, A.J.; Butcher, P. D.; Hay, P.; Sadiq, S. T. Sexually transmitted infections2015, 91, 165−170.(3) Strain, M. C.; Lada, S. M.; Luong, T.; Rought, S. E.; Gianella, S.;Terry, V. H.; Spina, C. A.; Woelk, C. H.; Richman, D. D. PLoS One2013, 8, e55943.(4) Yang, R.; Paparini, A.; Monis, P.; Ryan, U. Int. J. Parasitol. 2014,44, 1105−1113.(5) Cao, Y.; Raith, M. R.; Griffith, J. F.Water Res. 2015, 70, 337−349.

(6) Floren, C.; Wiedemann, I.; Brenig, B.; Schutz, E.; Beck, J. FoodChem. 2015, 173, 1054−1058.(7) Morisset, D.; Stebih, D.; Milavec, M.; Gruden, K.; Zel, J. PLoSOne 2013, 8, e62583.(8) Kim, T. G.; Jeong, S. Y.; Cho, K. S. Appl. Microbiol. Biotechnol.2014, 98, 6105−6113.(9) Day, E.; Dear, P. H.; McCaughan, F.Methods 2013, 59, 101−107.(10) Jennings, L. J.; George, D.; Czech, J.; Yu, M.; Joseph, L. J. Mol.Diagn. 2014, 16, 174−179.(11) Regan, J. F.; Kamitaki, N.; Legler, T.; Cooper, S.; Klitgord, N.;Karlin-Neumann, G.; Wong, C.; Hodges, S.; Koehler, R.; Tzonev, S.;McCarroll, S. A. PLoS One 2015, 10, e0118270.(12) Heredia, N. J.; Belgrader, P.; Wang, S.; Koehler, R.; Regan, J.;Cosman, A. M.; Saxonov, S.; Hindson, B.; Tanner, S. C.; Brown, A. S.;Karlin-Neumann, G. Methods 2013, 59, S20−23.(13) Sanmamed, M. F.; Fernandez-Landazuri, S.; Rodriguez, C.;Zarate, R.; Lozano, M. D.; Zubiri, L.; Perez-Gracia, J. L.; Martin-Algarra, S.; Gonzalez, A. Clin. Chem. 2015, 61, 297−304.(14) Gao, S.; Zheng, C.; Chang, G.; Liu, W.; Kou, X.; Tan, K.; Tao,L.; Xu, K.; Wang, H.; Cai, J.; Tian, J.; Gao, S. Nat. Commun. 2015, 6,6318.(15) Beck, J.; Bierau, S.; Balzer, S.; Andag, R.; Kanzow, P.; Schmitz,J.; Gaedcke, J.; Moerer, O.; Slotta, J. E.; Walson, P.; Kollmar, O.;Oellerich, M.; Schutz, E. Clin. Chem. 2013, 59, 1732−1741.(16) Aigrain, L.; Gu, Y.; Quail, M. A. BMC Genomics 2016, 17, 458.(17) Laurie, M. T.; Bertout, J. A.; Taylor, S. D.; Burton, J. N.;Shendure, J. A.; Bielas, J. H. BioTechniques 2013, 55, 61−67.(18) Robin, J. D.; Ludlow, A. T.; LaRanger, R.; Wright, W. E.; Shay, J.W. Sci. Rep. 2016, 6, 24067.(19) Findlay, S. D.; Vincent, K. M.; Berman, J. R.; Postovit, L. M.PLoS One 2016, 11, e0153901.(20) Miyaoka, Y.; Chan, A. H.; Conklin, B. R. Cold Spring HarborProtoc. 2016, 2016, pdb.prot086801.(21) Haynes, R. J.; Kline, M. C.; Toman, B.; Scott, C.; Wallace, P.;Butler, J. M.; Holden, M. J. J. Mol. Diagn. 2013, 15, 177−185.(22) White, H.; Deprez, L.; Corbisier, P.; Hall, V.; Lin, F.; Mazoua,S.; Trapmann, S.; Aggerholm, A.; Andrikovics, H.; Akiki, S.; Barbany,G.; Boeckx, N.; Bench, A.; Catherwood, M.; Cayuela, J. M.; Chudleigh,S.; Clench, T.; Colomer, D.; Daraio, F.; Dulucq, S.; Farrugia, J.;Fletcher, L.; Foroni, L.; Ganderton, R.; Gerrard, G.; Gineikiene, E.;Hayette, S.; El Housni, H.; Izzo, B.; Jansson, M.; Johnels, P.; Jurcek,T.; Kairisto, V.; Kizilors, A.; Kim, D. W.; Lange, T.; Lion, T.; Polakova,K. M.; Martinelli, G.; McCarron, S.; Merle, P. A.; Milner, B.;Mitterbauer-Hohendanner, G.; Nagar, M.; Nickless, G.; Nomdedeu, J.;Nymoen, D. A.; Leibundgut, E. O.; Ozbek, U.; Pajic, T.; Pfeifer, H.;Preudhomme, C.; Raudsepp, K.; Romeo, G.; Sacha, T.; Talmaci, R.;Touloumenidou, T.; Van der Velden, V. H.; Waits, P.; Wang, L.;Wilkinson, E.; Wilson, G.; Wren, D.; Zadro, R.; Ziermann, J.; Zoi, K.;Muller, M. C.; Hochhaus, A.; Schimmel, H.; Cross, N. C.; Emons, H.Leukemia 2015, 29, 369−376.(23) Baoutina, A.; Bhat, S.; Partis, L.; Dobeson, M.; Zheng, M.;Alexander, I. E.; Emslie, K. R. Gene Ther. 2016, 23, 708−717.(24) Dong, L.; Meng, Y.; Sui, Z.; Wang, J.; Wu, L.; Fu, B. Sci. Rep.2015, 5, 13174.(25) Deprez, L.; Corbisier, P.; Kortekaas, A. M.; Mazoua, S.; BeazHidalgo, R.; Trapmann, S.; Emons, H. Biomolecular detection andquantification 2016, 9, 29−39.(26) Yu, H.; Hahn, Y.; Yang, I. PLoS One 2015, 10, e0137006.(27) Mamedov, T. G.; Pienaar, E.; Whitney, S. E.; TerMaat, J. R.;Carvill, G.; Goliath, R.; Subramanian, A.; Viljoen, H. J. Comput. Biol.Chem. 2008, 32, 452−457.(28) von Kanel, T.; Gerber, D.; Wittwer, C. T.; Hermann, M.; Gallati,S. Anal. Biochem. 2011, 419, 161−167.(29) Hadimioglu, B.; Stearns, R.; Ellson, R. J. Lab. Autom. 2016, 21,4−18.(30) Burke, D. G.; Dong, L.; Bhat, S.; Forbes-Smith, M.; Fu, S.;Pinheiro, L.; Jing, W.; Emslie, K. R. Anal. Chem. 2013, 85, 1657−1664.

Analytical Chemistry Article

DOI: 10.1021/acs.analchem.6b05032Anal. Chem. XXXX, XXX, XXX−XXX

H

Page 9: Interlaboratory Reproducibility of Droplet Digital ...Interlaboratory Reproducibility of Droplet Digital Polymerase Chain Reaction Using a New DNA Reference Material Format Leonardo

(31) Pinheiro, L. B.; Coleman, V. A.; Hindson, C. M.; Herrmann, J.;Hindson, B. J.; Bhat, S.; Emslie, K. R. Anal. Chem. 2012, 84, 1003−1011.(32) Bhat, S.; Herrmann, J.; Armishaw, P.; Corbisier, P.; Emslie, K. R.Anal. Bioanal. Chem. 2009, 394, 457−467.(33) ISO. Guide 35, Reference materials - General and statisticalprinciples for certification, International Standardization Organization:Geneva, Switzerland, 2006.(34) ISO. ISO 5725−3:1994 Accuracy (trueness and precision) ofmeasurement methods and results - Part 3: Intermediate measures of theprecision of a standard measurement method, International Stand-ardization Organization: Geneva, Switzerland, 1994.(35) ISO. ISO Guide 34:2009 General requirements for the competenceof reference material producers, International Standardization Organ-ization: Geneva, Switzerland2009.(36) ISO. ISO 5725−1:1994 Accuracy (trueness and precision) ofmeasurement methods and results -- Part 1: General principles anddefinitions, International Standardization Organization: Geneva,Switzerland, 1994.(37) ISO. ISO 5725−2:1994 Accuracy (trueness and precision) ofmeasurement methods and results - Part 3: Basic method for thedetermination of repeatability and reproducibility of a standardmeasurement method, International Standardization Organization:Geneva, Switzerland, 1994.

Analytical Chemistry Article

DOI: 10.1021/acs.analchem.6b05032Anal. Chem. XXXX, XXX, XXX−XXX

I