How proficient are you?

How proficient are you?

Sylvia Janetzki, M.D.ZellNet Consulting, Inc.

Cancer Immunotherapy [email protected]

How proficient are youreally?

Sylvia Janetzki, M.D.ZellNet Consulting, Inc

Cancer Immunotherapy [email protected]

Proficiency:

Mastery of a specific skill demonstrated by consistently superior performance, measured against established or popular standards.

Business dictionary

http://www.businessdictionary.com/definition/skill.html

http://www.businessdictionary.com/definition/performance.html

External QA in immune monitoring

CLSI (former NCCLS) document I/LA26-A, Vol. 24, No. 29 “Performance of Single Cell Immune response assays; approved guidelines”

“Each new test must be validated before beingimplemented for patient testing… Validation of a test procedure includes the external quality assessment through a proficiency testing program.” (p47)

What is a proficiency panel?

• Same samples sent to each lab

• Each lab tests samples under own SOP

• Each lab reports measured values back

• Those values are compared to

A predefined reference value, or

The results reported by all panelists

• Labs receive a performance report

Every immune monitor”er” can get proficiency tested and should get proficiency tested because

1. participation in proficiency testing is a valuable continuous improvement tool;

2. it gives direct feedback about your performance and can provide reassurance;

3. it builds credibility and confidence with sponsoring and regulatory agencies;

4. it helps accelerate vaccine development through accurate immune monitoring;

5. it is an ethical responsibility to the clinical trial volunteer to create accurate and reliable data.

100

200

300

Min=8

Max=281

Median=109

CV=55

Mea

n sp

ot n

umbe

r per

wel

lElispot

1

2

3

Mea

n %

Cyt

okin

e+ C

D4+

lym

phoc

ytes

Max=3.29

Min=0.16

Median=1.93

ICS

CV=47

Mea

n %

CD

8+ M

ultim

er B

indi

ng

0.30

0.60

1.5

Min=0

Max=1.53

Median=0.16

CV=135

HLA-peptide multimer staining

37 labs, mean of6 replicates per lab



“moderate” response “strong” response “low” response

0.15

0.45

1.35

Because:

Janetzki et al. Immunity 2009

My assay is the best

I only believe my own

data

I am well Known in my field.Hence – myassays mustbe good.

There is a lack of a Gold Standard (test) for assays assessing responses on single cell level

And because:

How do we know what works best?

There is no gold standard for most commonly used immune monitoring assays.

Crucial protocol variables can be identified that influence assay outcome in either direction.

The check-up on performance can be achieved via inter-laboratory testing projects = proficiency panels*.

* The larger the panel, the more powerful the results (panel design essential)

Harmonization Guidelines

Phases of assay “evolution”

Development

Qualification

Optimization

Standardization

Validation

ImplementationRe-validation

External QA=

Proficiency Panel

Harmonization guidelines

“Gold Standard”

2005: 1st Elispot panel (36 labs)2006: 2nd Elispot panel (29 labs)2007: 3rd Elispot panel (35 labs) 1st Multimer panel (29 labs) 1st ICS panel (28 labs) 1st CFSE panel (21 labs)2009: 4th Elispot panel (41 labs)

2nd Multimer panel (20 labs) 2nd ICS panel (31 labs)2010: ICS Gating panel (110 labs)2011-12: 5th Elispot panel (40 labs) 3rd Multimer panel (35 labs) 1st – 3rd Luminex panel (26 labs)2013: 1st Elispot Plate Reading panel (86 labs)

2nd ICS Gating Panel

20 countries

The CIC Proficiency Panel Program

• The first panel was run under highly standardized conditions with standardized reagents (BD lyoplates).

• A high degree of variability and failure of many labs to detect responses was observed.

• Most of the variation was due to:– Number of cells counted (labs that acquired >100,000

cells had better results)– Instrument setup and compensation strategies (many

labs did not accurately compensate their samples)– Gating strategies

• 1st ICS Panel Findings: 28 labs participated

1st ICS Panel: Overall Variability Example

This was confirmed by other independent panels:

CIP/CIMT 2011 (Welters et al. 2011) NIAID 2010 (Jaimes et al. 2010)

Overview of ICS Gating Panel Design

• All participants analyzed the same set of nine FCS files. • No experiments were performed. • Setting up the study this way allows for quantification

of the variability caused by gating since all other experimental sources of variation are removed.

• The panel had two phases:• Phase I: Labs evaluated the FCS files using their

lab-specific protocol/gating strategy.• Phase II: Labs re-analyzed the same FCS files using

a gating strategy that was drafted by consensus from the organizers of the panel.

Lisa McNeil& ICSSteeringCommittee

FCS Files

• Nine FCS files (all labs received the SAME files)

• 3 donors (Donor 1, 2 and 3) that have been stimulated with CEF, CMV peptide pool and media (unstimulated).

• The cells have been stained with 5 antibodies:– Dump channel (CD14 and CD19 Pacific Blue)– CD3-FITC– CD4-PE-Cy7– CD8-Alexa 700 – IL-2/IFN-APC

• The participants were requested to evaluate the % of CD4+ cytokine+ cells and CD8+ cytokine+ cells and to provide gating

strategy and answer a survey.

• CIC/CRI – Jianda Yuan, Leah Price (Statistician), Sylvia Janetzki (Coordinator)

• CHAVI /EQAPOL- Janet Staats (Reviewer)• CVC/CRI - Kunle Odunsi• CIMT - Cedrik Britten (Reviewer)• DAIDS - Maria Jaimes (Reviewer)• FOCIS - Holden Maecker• IAVI - Dilbinder Gill• IDS - Ivana Durinovic-Bello, Jerill Thorpe (Reviewer)• iSBTc/SITC - Lisa Butterfield• ITN - Jason Liu

This was a huge effort – but well worth it

110 labs participated

Summary of Results

• 110 Labs participated and there were 110 different approaches to gating.

• Variability was dramatically reduced when all labs used the same gating strategy (Phase II).

• Proximity of the cytokine gate to the negative population impacted true positive and false positive response detection.

Variability is decreased if all labs use the same gating strategy.

Lab

CEF

Unstimulated

CMV pp65

CEF

Unstimulated

CMV pp65

Donor 1, CD8

Consensus

CEF

Unstimulated

CMV pp65

Lab

Consensus

Placement of Cytokine Gate

APC-A

Cou

nt

-102 102 103 104 1050

284

568

851

1135

M1

Proximity Adequate Too far HistogramToo close

CD4 Cytokine Gate: Gate

Proximity (% of labs)

5 or 6 responses

AND No false positives

(n=37)

5 or 6 responses

AND False

positive (n=44)

< 5 responses

(n=28)

< 4 responses

(n=13)

Adequate Proximity 54% 30% 32% 23%

Too Far 8% 64% 7% 0%

Too Close 38% 2% 57% 69%

21

CD4+ CD8+ T cells: Double Positive cells

CD4+ CD8+ DP cellsare excluded from the CD4 and CD8 gates

CD4+ CD8+ DP cells are includedin both CD4 and CD8 gates

DP Gate (% of labs)

5 or 6 responses


(n=37)

5 or 6 responses

AND False

positive (n=44)

< 5 responses

(n=28)

< 4 responses

(n=13)

% of labs that excluded DP

cells 65% 73% 46% 62%

22

Lymphocyte Gate

Lymphocyte Gate (% of labs)

5 or 6 responses


(n=37)

5 or 6 responses

AND False

positive (n=44)

< 5 responses

(n=28)

< 4 responses

(n=13)

Included all Lymphocytes, No debris or

RBC 49% 43% 43% 23%

Excluded Lymphocytes 19% 30% 14% 31%

Too Large 8% 20% 21% 23%

Included RBC and debris 19% 5% 18% 15%

Inclusion of Dim Cells

Dim cells (% of labs)

5 or 6 responses


(n=37)

5 or 6 responses

AND False

positive (n=44)

< 5 responses

(n=28)

< 4 responses

(n=13)

Included CD3 Dim cells 46% 55% 25% 23%



Examples of Dims not included in the CD4/CD8 Gate

Uniformity of Gates

0 102 103 104 105

0

102

103

104

105

Donor 2_DMSO_D05.fcsÉCD8+CD4-

<Alexa Fluor 700-A>: CD8

<AP

C-A

>: IF

N-g

/IL-2

0.024

0 102 103 104 105

0

102

103

104

105

Donor 2_CEF_D03.fcsÉCD8+CD4-

<Alexa Fluor 700-A>: CD8

<AP

C-A

>: IF

N-g

/IL-2

0.67Changing gates within a donor

Uniform gates (% of labs)

5 or 6 responses

ANDNo false positives

(n=37)

5 or 6 responses

ANDFalse

positive(n=44)

< 5 responses

(n=28)

< 4 responses

(n=13)

Did not use uniform gates within a donor

3% 16% 11% 0%

Sometimes or never used

uniform gates across donors

17% 39% 18% 15%

Biexponential Scaling

CD8 gate not drawn all the

way to the axis, missing ½ of the

cells!

Biexponential scaling not correctly applied.

UngatedDonor 1_CEF_B03.fcsEvent Count: 169945

0 102 103 104 105<Pacific Blue-A>: Dump gate

0

50K

100K

150K

200K

250K

SSC-

A

85.9

dumpFirstDonor 1_CEF_B03.fcsEvent Count: 145945

0 50K 100K 150K 200K 250KFSC-A

0

50K

100K

150K

200K

250K

SSC

-A

66.2

lymphocytesDonor 1_CEF_B03.fcsEvent Count: 96631

0 102 103 104 105<PE-Cy7-A>: CD4

0102

103

104

105

<FIT

C-A

>: C

D3

46.7

CD3CD4Donor 1_CEF_B03.fcsEvent Count: 45137

0 102 103 104 105<PE-Cy7-A>: CD4

0102

103

104

105

<APC

-A>:

IFN-

g/IL

-2 CD3CD8Donor 1_CEF_B03.fcsEvent Count: 35708

0 102 103 104 105<Alexa Fluor 700-A>: CD8

0102

103

104

105

<APC

-A>:

IFN

-g/IL

-2

0 102 103 104 105<Alexa Fluor 700-A>: CD8

0102

103

104

105

<FIT

C-A

>: C

D3

37

lymphocytesDonor 1_CEF_B03.fcsEvent Count: 96631

0 102 103 104 105

0

102

103

104

105

Donor 1_DMSO ii_B06.fcsÉPacific Blue- CD3+

<PE-Cy7-A>: CD4

<Ale

xa F

luor

700

-A>:

CD

8

52.4

39.2

Table 2. Summary of Current Gating Recommendations for Assay Harmonization.

Recommendation

Cytokine gate Cytokine gate should not include negative cells and should include all of the cytokine positive cells (both high and low positive cells).

Double Positive cells Gate on CD4 and CD8 to allow clear identification of the double positive cells. Include as a separate population or with either CD4 or CD8 cells but not with both.

Dim cells Draw gates around the main populations of CD3, CD4 and CD8 that incude dim events.

Lymphocyte gate Lymphocyte gate should be large enough to include all lymphocytes. It should not include RBC or debris.

Uniform gates Uniform gates should be used for all samples within a donor.

Biexponential scaling Proper adjustment of biexponential scaling is important to clearly and completely visualize all populations.

Phase I Phase II

Your Lab Response Overall Results Your Lab Response Overall Results

CD+ Reagent Donor Media

n 25th Pctl

75th Pctl

Median

25th Pctl

75th Pctl

CD4+Unstimulated 1

Below Average 0.00 0.03 0.01 0.07

Above Average 0.05 0.02 0.02 0.04

2Below

Average 0.00 0.03 0.01 0.05 Average 0.02 0.02 0.02 0.03 3 Average 0.01 0.03 0.01 0.06 Average 0.01 0.02 0.01 0.03

CEF 1Below

Average 0.00 0.05 0.02 0.10 Average 0.05 0.04 0.03 0.05

2Below

Average 0.01 0.03 0.02 0.07 Average 0.02 0.02 0.02 0.03

3Below

Average 0.01 0.05 0.02 0.07 Average 0.03 0.04 0.03 0.04

CMVpp65 1Below

Average 0.01 0.35 0.22 0.47Below

Average 0.04 0.23 0.19 0.30

2Below

Average 0.01 0.03 0.01 0.07 Average 0.03 0.02 0.02 0.03

3Below

Average 0.01 0.07 0.04 0.10 Average 0.04 0.05 0.04 0.06

CD8+Unstimulated 1 Average 0.02 0.04 0.01 0.11

Above Average 0.05 0.03 0.02 0.04

2 Average 0.04 0.05 0.03 0.10 Average 0.03 0.03 0.03 0.05 3 Average 0.06 0.09 0.05 0.16 Average 0.08 0.06 0.05 0.09

CEF 1Below

Average 0.09 0.20 0.12 0.33Below

Average 0.05 0.17 0.13 0.23

2Below

Average 0.30 0.80 0.70 0.88Below

Average 0.31 0.73 0.69 0.76

3 Average 0.20 0.28 0.20 0.37Below

Average 0.21 0.26 0.22 0.29

CMVpp65 1Below

Average 0.06 0.24 0.15 0.39Below

Average 0.04 0.16 0.14 0.20 2 Average 0.04 0.05 0.02 0.10 Average 0.03 0.03 0.02 0.04

3 Average 0.13 0.20 0.13 0.27Below

Average 0.11 0.18 0.15 0.21

Your performance report

Phase I Review Parameter Reviewed Reviewer AssessmentAll gates uniform across all donors? Yes, always uniform.All gates uniform within each donor? Yes

Biexponential scaling correctly applied?No, populations are pushed against the axis.

Dump Negative Gate Gate includes dump positive cells.Lymphocyte Gate Gate too large, includes non-lymphocyte cells.CD3 Gate CD3 gate not too tight, includes dim cells.

CD4+ CD8+ (Double Positive) CellsDP cells are included with both CD4 and CD8 cells.

CD4 Gate CD4 Gate not too tight, dims are included.CD8 Gate CD8 Gate not too tight, dims are included.CD4+ Cytokine+ Gate: Negative Cells Gate does not include cytokine negative cells.CD4+ Cytokine+ Gate: Positive Cells Gate includes all cytokine positive cells.

CD4+ Cytokine+ Gate: Gate Proximity Gate does not include negative cells & includes all cytokine positive cells (high and low positives).

CD8+ Cytokine+ Gate: Negative Cells Gate does not include cytokine negative cells.CD8+ Cytokine+ Gate: Positive Cells Gate includes all cytokine positive cells.

CD8+ Cytokine+ Gate: Gate Proximity Gate does not include negative cells & includes all cytokine positive cells (high and low positives).

Reviewer Comments – Reviewer IDifficult to determine gating strategy from powerpoint slides.

Reviewer Comments – Reviewer II No comment provided.

Your performance report

Elispot Panel 1

Assay example: all participants testing same donor for same response

4/36 labs outliers = missed to detect > 50% of responses17/36 labs missed to detect low responder

Variability, outliers, sub-optimal performance

36 labs used own protocol, same PBMC, same antigen

Initial Elispot Harmonization Guidelines

A. Establish lab Elispot SOP for: A1. Counting method for apoptotic cells in order to determine adequate cell dilution for plating A2. Overnight resting of cells prior to platingB. Use only pretested serum with optimal signal:noise ratioC. Establish SOP for plate reading, including: C1. Human auditing during reading process C2. Adequate adjustment for technical artifactsD. Only let well trained personnel conduct assay

Janetzki et al., CII. 2008 Mar;57(3):303-15

Impact of repeated proficiency panels and ongoing harmonization (panel 1-4)

Percentage of labs that missed weak responder dropped dramatically through Panels 1, 2, and 4.

Adapted from van der Burg et al, Science Translational Medicine 2011; V3, Issue 108

Series1

0%

10%

20%

30%

40%

50%

47%

14%

7%

Serum pretested 66% (25)

No serum used 26% (10)

Serum not pretested 8% (3)

> 8 hours 66% (25)

< 8 hours 21% (8)

No resting 13% (5)

Increasing harmonization across labs (panel 5)

• All labs follow at least 2 harmonization guidelines• 68% of labs follow 4-6 harmonization guidelines

0 50 100 150 2000

20

40

60

80

100

120

140

160

180

200

Spots per 2x105 cells

CV

Overall Variability decreased by 20%(for the first time ever)

EPP4EPP5

Num

ber o

f det

ecte

d re

spon

ses

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 27 28 29 30

Lab ID

0

1

2

3

45

6

7

8

9

66% of labs were not able to detect 8 or 9 responses

Multimer Panel – Same Experience

Cancer Immunol Immunother. 2009 Oct;58(10):1701-13.

Cedrik Britten & Pedro Romero

Panel 1:

Initial Harmonization Guidelines – Multimer Staining

Britten et al. 2009 CII

Multimer Panel - MPP2 – Main Findings 1

1. Significantly higher response detection rate (e.g. 88% for all CMV responses [which were low responses])

DonorDump

Channel

Dead Cell Staining N Median

1 No No 6 0.04

No Yes 14 0.02

Yes No 6 0.04

Yes Yes 14 0.01

2 No No 6 0.06

No Yes 14 0.03

Yes No 6 0.05

Yes Yes 14 0.02

2. Dump channel& dead cellstainingdecrease noisereported

Attig et al. JTM 2011

Multimer Panel - MPP2 – Main Findings 2

3. Confirmation of harmonization guidelines

VariableNumber of Replicates

Number of Responses

% of Responses

Number of <100,000 292 146 50%CD8-

positive 100,000-150,000 164 118 72%

cells >150,000 104 94 90%Number of 2 28 10 36%

Colors 3 224 126 56%Used 4 196 142 72%

5 98 68 69%6 14 12 86%

The 14 labs that participated in both Multimer Panel 2 and Panel 3 were the top performers in Panel 3

Mul

timer

(PE)

CD8

0.301

3.97e-30.0377

99.7

D3 D2

Mul

timer

(PE)

CD8

Mul

timer

(APC

)

CD8

D4

Mul

timer

(PE)

CD8

Mul

timer

(PE)

Mul

timer

(PE)

CD8CD8

0.301

3.97e-30.0377

99.7

D3 D2

Mul

timer

(PE)

CD8

Mul

timer

(PE)

Mul

timer

(PE)

CD8CD8

Mul

timer

(APC

)

CD8

D4

Mul

timer

(APC

)

CD8

Mul

timer

(APC

)M

ultim

er (A

PC)

CD8CD8

D4

Repeated Panel participation: Road to successLabs Detected Total Number

of LabsNumberof Labs

Frequenciesof Labs

11 – 12 Responses 33 15 45.5%10 – 6 Responses 33 9 27.3%<6 Responses 33 8 24.2%

SAMPLE ASSAY ACQUISITION & ANALYSIS RESULTS

o Anticoagulanto Processingo Storageo Qualityo etc.

o Antibodieso Fluorochromeso Lyoplateso Incubation timeso Washing stepso etc.

o Machine setupo Event countso Compensationo Transformationo Gatingo etc.

o Example of Raw data

o Statisticso Response

Determinationo etc.

CELLULAR ASSAYS - SOURCES OF VARIATION

ALL THESE COMPONENTS HAVE BEEN SHOWN TO IMPACT ON ASSAY PERFORMANCE

LAB ENVIRONMENT

o Operators and Assays, SOPs, Qualification, Validation, Certification, etc.

Panel activities

SAMPLE ASSAY ACQUISITION & ANALYSIS RESULTS

CELLULAR ASSAYS - SOURCES OF VARIATION

LAB ENVIRONMENT

MANY PUBLICATIONS LACK CRITICAL INFORMATION ON THESE COMPONENTS

THIS PRECLUDES COMPARABILITY OF RESULTS GENERATED ACROSS LABS

MIATA – Initiation

42

Guidelines Version 0.0

• Published at website• Announced in

Immunity (2009, Vol. 31, p527)

Kick-off by a core team and public announcement of Version 0in October 2009

MIATA – Project Overview

43

• Public Consultation Period

• Expert Public Workshop• Three Webinars

Version 0.0• Announced in Immunity

(2009)• Published on MIATA website

Version 1.0• Published on MIATA website

(2010)

Version 2.0• Published in Immunity

(2012) • Published with additional

tools on MIATA website

• Public Consultation Period

• Expert Public workshop• Discussion with experts• Discussion with groups • Discussion with editors

Level of Acceptance

The Sample The Assay Data Acquisition Results Lab Environment

Minimal Information About:

Manuscript: Materials and Methods

Hall of Fame

Transparency

Critical protocol variables

Improved interpretation of results

MIATA stamp

Increased visibility of published work

Increased citation rate

44

Relaunch of Homepage – HALL OF FAME

HOW to implement MIATA

45

FIRST STEP SECOND STEP

visit miataproject.org

Use the checklist

Structure M&M part according to MIATA

STILL UNCERTAIN ?

THREE STEPTS TOWARDS IMPLEMENTATION

46

FIRST STEP SECOND STEP THIRD STEP


Use the checklist


Check example reports

STILL UNCERTAIN ?

THREE STEPTS TOWARDS IMPLEMENTATION

47

FIRST STEP SECOND STEP THIRD STEP


Use the checklist


Check example reports

FOURTH STEP

Ask Cedrik & Sylvia

WHERE to implement MIATA

48

• You can implement MIATA in every journal

• Three journals offcially recommend use if MIATA (optional choice)

o Cancer Immunotherapy Immunology

o OncoImmunology

o Cancer Immunity

• Journals may label your mansucripts with the MIATA logo on the cover page

STILL NOT CONVINCED ?

WHY to implement MIATA

49

• Accepted manuscripts that adhere to MIATA will be listed at the homepage

• Show that you control the variables critically impacting on quality of results

• This may attract more readers to your study

• Peers may acknowledge that you transparently report your experiments

• This may lead to more citations on the long-run

STILL NOT CONVINCED ?

MIATA – PRIZE FOR FIRST PUBLICATIONS !

50

CIMT is donating a prize to the first 6 manuscripts for MIATA-adherent studies

o Acceptance in a peer reviewed journal

o PRIZE goes to one member of the author list

o Study needs to adhere to MIATA

o Study needs to mention MIATA

o 500,- Euro plus free registration for upcoming CIMT meeting

TRY MIATA !!!

How proficient are you?

Documents

Transcript of How proficient are you?