20140711 6 s_munro_ercc2.0_workshop
15
Standard Performance Metrics for Gene Expression Experiments with the “erccdashboard” Sarah Munro
-
Upload
external-rna-controls-consortium -
Category
Science
-
view
102 -
download
2
Transcript of 20140711 6 s_munro_ercc2.0_workshop
Standard Performance Metrics for Gene Expression Experiments with the “erccdashboard”
Sarah Munro
Feature A_1 A_2 A_3 B_1 B_2 B_3
T1 1 5 4 0 2 3
T2 200 204 199 101 97 103
T3 142 153 147 149 130 155
ERCC-‐0001 5 8 10 20 23 19
…
Method ValidaOon with erccdashboard R package
erccdashboard Package Vignette
Sarah A. Munro
May 4, 2014
This vignette describes the use of the erccdashboard R package to analyze External RNA Control Con-sortium (ERCC) spike-in control ratio mixtures in gene expression experiments. If you use this package formethod validation of your gene expression experiments please cite our publication:
Please cite our paper when you use the erccdashboardpackage for analysis. This is a placeholder citation,because our manuscript is still under review.
Munro SA, Lund S, Pine PS, Binder H, Clevert D,Conesa A, Dopazo J, Fasold M, Hochreiter S, Hong H,Jafari N, Kreil DP, A ,Aabaj PP, Liao Y, Lin S, MeehanJ, Mason CE, Santoyo J, Setterquist RA, Shi L, ShiW, Smyth GK, Stralis-Pavese N, Su Z, Tong W, WangC, Wang J, Xu J, Ye Z, Yang Y, Yu Y, Salit M (UnderReview, 2014). Assessing Technical Performance inGene Expression Experiments with External Spike-inRNA Control Ratio Mixtures.
A BibTeX entry for LaTeX users is
@Article{,title = {Assessing Technical Performance in Gene Expression Experiments with External Spike-in RNA Control Ratio Mixtures},author = {Munro SA and Lund S and Pine PS and Binder H and Clevert D and Conesa A and Dopazo J and Fasold M and Hochreiter S and Hong H and Jafari N and Kreil DP and A ,Aabaj PP and Li S and Liao Y and Lin S and Meehan J and Mason CE and Santoyo J and Setterquist RA and Shi L and Shi W and Smyth GK and Stralis-Pavese N and Su Z and Tong W and Wang C and Wang J and Xu J and Ye Z and Yang Y and Yu Y and Salit M},journal = {Under Review},volume = {0},pages = {0},year = {2014},
}
Munro SA, Lund S, Pine PS, Binder H, Clevert D, Conesa A, Dopazo J, Fasold M, Hochreiter S, Hong H,Jafari N, Kreil DP, 0141abaj PP, Li S, Liao Y, Lin S, Meehan J, Mason CE, Santoyo J, Setterquist RA, ShiL, Shi W, Smyth GK, Stralis-Pavese N, Su Z, Tong W, Wang C, Wang J, Xu J, Ye Z, Yang Y, Yu Y, SalitM (Under Review, 2014). Assessing Technical Performance in Gene Expression Experiments with ExternalSpike-in RNA Control Ratio Mixtures.
Analysis is shown for two types of samples spiked with ERCC control ratio mixtures from the SEQCproject
• Rat toxicogenomics treatment and control samples for di↵erent drug treatments
• Human reference RNA samples from the MAQC I project, Universal Human Reference RNA (UHRR)and Human Brain Reference RNA (HBRR)
1
−10
−5
0
5
10
−10 −5 0 5 10Log2 ERCC Spike Amount(attomol nt/µg total RNA)
Log2
Nor
mal
ized
ERC
C C
ount
s
Ratio4:11:11:1.51:2
SampleCTLMET
Ratio4:1
1:1.51:2
AUC1.0000.9500.971
Detected161616
Spiked232323
0.00
0.25
0.50
0.75
1.00
0.00 0.25 0.50 0.75 1.00FPR
TPR
Ratio4:11:1.51:2
log(rm)
Weighted Mean
−0.07014
(+/−) Weighted Standard Error
0.1495
−4
−3
−2
−1
0
1
2
3
4
−10 −5 0 5 10Log2 Average of Normalized Counts
Log2
Rat
io o
f Nor
mal
ized
Cou
nts
Ratio4:11:11:1.51:2
1e−05
1e−04
1e−03
1e−02
1e−01
1e+00
10 1000Average Counts
DE
Test
P−v
alue
s
Ratio4:11:11:1.51:2
Ratio4:1
1:1.51:2
LODR Estimate26Inf280
90% CI Lower Bound19NA140
90% CI Upper Bound33NA410
• Open-‐source R package – erccdashboard
• Assess technical performance of a gene expression experiment
• Compare results – Within a single laboratory
– Between laboratories
Method ValidaOon with erccdashboard R package
• Open-‐source R package – erccdashboard
• Assess technical performance of a gene expression experiment
• Compare results – Within a single laboratory
– Between laboratories
ERCC RaOo Mixture Design
Example Experiments for erccdashboard Method ValidaOon
Universal Human Reference RNA (UHRR) vs. Human Brain Reference RNA (HBRR)
Library PreparaOon Replicates
Microarrays RNA-‐Seq
Dynamic Range
−2
0
2
4
−5 0 5 10Log2 ERCC Spike Amount(attomol nt/µg total RNA)
Log2
Nor
mal
ized
Fluo
resc
ent I
nten
sity
Sample
HBRR
UHRR
Ratio
4:1
1:1
1:1.5
1:2
−10
−5
0
5
10
−10 −5 0 5 10Log2 ERCC Spike Amount(attomol nt/µg total RNA)
Log2
Nor
mal
ized
ERC
C C
ount
s
Sample
HBRR
UHRR
Ratio
4:1
1:1
1:1.5
1:2
Microarray RNA-‐Seq
DiagnosOc Performance
Ratio4:11:1.51:2
AUC0.9330.9290.917
Detected141416
Spiked232323
0.00
0.25
0.50
0.75
1.00
0.00 0.25 0.50 0.75 1.00FPR
TPR
Ratio
4:1
1:1.5
1:2
Ratio4:11:1.51:2
AUC0.9660.8260.894
Detected222223
Spiked232323
0.00
0.25
0.50
0.75
1.00
0.00 0.25 0.50 0.75 1.00FPR
TPR
Ratio
4:1
1:1.5
1:2
Microarray RNA-‐Seq
LODR: Limit of DetecOon of RaOos
1e−12
1e−10
1e−08
1e−05
1e−04
1e−03
1e−02
1e−01
1e+00
100 1000 10000Average Fluorescence Intensity
DE
Test
P−v
alue
s
Ratio4:11:11:1.51:2
Ratio4:1
1:1.51:2
LODR Estimate<150200150
90% CI Lower Bound<150<160<130
90% CI Upper Bound<150210160
1e−12
1e−10
1e−08
1e−051e−041e−031e−021e−011e+00
1e+01 1e+03 1e+05Average Counts
DE
Test
P−v
alue
s
Ratio4:11:11:1.51:2
Ratio4:1
1:1.51:2
LODR Estimate2117037
90% CI Lower Bound5.97222
90% CI Upper Bound2721040
Microarray RNA-‐Seq
RaOo DetecOon, Bias, and Variability
log(rm)
Weighted Mean0.207
(+/−) Weighted Standard Error0.09446
−4
−3
−2
−1
0
1
2
3
4
−2 0 2 4Log2 Average of Normalized Intensity
Log2
Rat
io o
f Nor
mal
ized
Inte
nsity
Ratio
4:1
1:1
1:1.5
1:2
log(rm)
Weighted Mean0.2836
(+/−) Weighted Standard Error0.03764
−4
−3
−2
−1
0
1
2
3
4
−10 −5 0 5 10Log2 Average of Normalized Counts
Log2
Rat
io o
f Nor
mal
ized
Cou
nts
Ratio
4:1
1:1
1:1.5
1:2
Microarray RNA-‐Seq
SAME PLATFORM, DIFFERENT LABORATORIES
−10
−5
0
5
10
0 5 10 15 20Log2 ERCC Spike Amount(attomol nt/µg total RNA)
Log2
Nor
mal
ized
ERC
C C
ount
s
SampleHBRRUHRR
Ratio4:11:11:1.51:2
Ratio4:1
1:1.51:2
AUC0.9350.8490.870
Detected222323
Spiked232323
0.00
0.25
0.50
0.75
1.00
0.00 0.25 0.50 0.75 1.00FPR
TPR
Ratio4:11:1.51:2
●
●
●●
●
●
●
●
● ●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
log(rm)
Weighted Mean
0.08596
(+/−) Weighted Standard Error
0.09636
−4
−3
−2
−1
0
1
2
3
4
−10 −5 0 5 10Log2 Average of Normalized Counts
Log2
Rat
io o
f Nor
mal
ized
Cou
nts
Ratio4:11:11:1.51:2
●●
●
●●
●●●●
●●
●●
●
●●● ●
●●● ●
●
●●
● ●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
● ●
●●
●
●
●●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
1e−12
1e−10
1e−08
1e−051e−041e−031e−021e−011e+00
1e+01 1e+03 1e+05Average Counts
DE
Test
P−v
alue
s
Ratio
●
●
●
●
4:1
1:1
1:1.5
1:2
Ratio4:1
1:1.51:2
LODR Estimate2613047
90% CI Lower Bound9.84326
90% CI Upper Bound4016061
Lab 8
−10
−5
0
5
10
0 5 10 15 20Log2 ERCC Spike Amount(attomol nt/µg total RNA)
Log2
Nor
mal
ized
ERC
C C
ount
s
SampleHBRRUHRR
Ratio4:11:11:1.51:2
Ratio4:1
1:1.51:2
AUC0.8220.6480.842
Detected232322
Spiked232323
0.00
0.25
0.50
0.75
1.00
0.00 0.25 0.50 0.75 1.00FPR
TPR
Ratio4:11:1.51:2
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●●
●
●
●●
●
log(rm)
Weighted Mean
0.2441
(+/−) Weighted Standard Error
0.265
−4
−3
−2
−1
0
1
2
3
4
−10 −5 0 5 10Log2 Average of Normalized Counts
Log2
Rat
io o
f Nor
mal
ized
Cou
nts
Ratio4:11:11:1.51:2
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ● ●●●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●●
● ●
●●
●
●
●
●● ●
●
●
●
●●
●
●
●●
● ●●
●
●
●
●
●
1e−12
1e−10
1e−08
1e−05
1e−04
1e−03
1e−02
1e−01
1e+00
1e+01 1e+03 1e+05Average Counts
DE
Test
P−v
alue
s
Ratio
●
●
●
●
4:1
1:1
1:1.5
1:2
Ratio4:1
1:1.51:2
LODR Estimate29Inf65
90% CI Lower Bound12NA20
90% CI Upper Bound39NA230
Lab 7
INTERLABORATORY STUDY RATIO PERFORMANCE MEASURES
Interlab study – FDA-‐led SEQC Project and the ABRF RNA-‐Seq study -‐ Single-‐round, three different measurement processes
���������
�
�
�AUC Diagnos/c performance
LODR Limit of Detec/on of Ra/os
mRNA FracOon RaOo Bias
RaOo Variability
Acknowledgements
• NIST – Marc Salit – Jenny McDaniel – Maghias Roesslein – Steve Lund – P. Scog Pine – JusOn Zook – David Duewer – Margaret Kline
• Members of the ERCC • FDA SEQC Study ParOcipants and co-‐authors
For informaOon on the erccdashboard sojware and ERCC 2.0 contact:
