1
Shadya George, MLS(ASCP)CM
GMP Quality Manager Cell Therapy Manufacturing
2
Why are We Here?
*
3
Purpose & Use of Quality Indicators
Critical Areas to Monitor
Indicator Types & Desired Characteristics
Design of Balanced Monitoring Program
Metrology (Measurement) Selection
SPC Analysis – Statistical Process Control
Ground Up Challenges & Strategies
4
Provide Evidence-Based Quality Assurance, Risk Prediction, & Corrective Action
5
Predict Acceptability of Clinical Outcomes
Treatment/Collection Decisions
Predict Risk of Adverse Reactions
Development of Process Understanding
Process Improvement/Optimization (DOE)
Alerts for Needed Corrective Action
Regulatory Compliance
6
Key Process Areas for Optimization
• Potency
• Purity
• Identity
• Efficacy
• Cell Dose Enumeration
Technical Manipulations: Cell Loss/Recovery
Clinical Outcomes: Engraftment
Regulatory Based Priorities:
• Historic Weak Compliance Areas
Critical Areas of Risk: FMEA
7
Team Brainstorming Process Failure Risk Study:
List Possible Process Failures Assign Risk Score
Severity
Likely/Frequency of Occurrence
Detectable
8
Balance a Mix of Types of Indicators based on
your Process and System Risks
9
10
# Name Categor
y Level Type Dimension
Department Responsibility
Definition Formula Units Data
Location Collection Frequency
Report Type
16 DMSO
Exposure Time Internal Process
Activity Process Safety Manufacturing Time from addition of DMSO
freeze media to cells until step advancement for freeze
protocol
Time of freeze initiation - Time of DMSO addition min
Product Batch
Record
Quarterly Run Chart
17 Cell Viability Post Thaw
Internal Process
Activity Process Effective Manufacturing % of living cells as determined
by validated & controlled release testing assay
(# viable cells/total cells)100%
Average product type viability, SD, CV% Annually
% Product Batch
Record
Quarterly
Run Chart
Bar Graph with Annual Averages, SD, CV%
14
Adverse Product Infusion
Reactions
Internal Process
Activity Outco
me Safety Clinical
Graded seriousness of reactions from 0 for no
reactions to 5 for complications resulting in patient death.
Charted for each infusion on graph quarterly, calculate
average, standard deviation and %CV annually
Average: Sum of all infusion reaction grades/total # infusions performed within calendar year SD: %CV: (SD/Average)100%
Grade/Infusion
Annual Grade Average
Annual Grade
SD
Annual Grade %CV
Product Infusion
Sheet Quarterly
Run Chart,
Scatter Plot linear Correlation against unit characteristics
Annually
Patient Mortality
Internal Process
System Outco
me Safety Clinical
Tracking % patient
deaths for any cause
within 1 year of treatment
(# Patient Deaths/Total
Patients Treated)100%
% Clinical
Outcome Report
Annual Bar
Graph
Manufacturing Cost
Finance Activit
y Struct
ure Efficient Manufacturing
Average Cost to manufacture 1 unit, to include supplies, labor,
release testing
Sum of (manufacturing supplies used, testing services, testing
supplies, office supplies, proportion of equipment & facility certification costs, &
labor)/#Units manufactured. Performed for each individual
product type.
$/unit Finance
Department
Annual Run
Chart
11
Can We Monitor Too Much?
*What type of processes do you
perform?
Autologous HPC
Allogeneic HPC
Clinical Trial Cell Manufacturing
All of the Above
12
*How many indicators do you track
on average for each manufacturing
processes?
1-4
5-10
11-20
>20
13
*How many indicators total are
tracked and reported quarterly
within your facility?
1-9
10-24
25-50
>50
14
15
Benefit vs Time & Cost Considerations
Risk Loss Focus on Critical Areas - CLUTTER
Frequency Decisions? How Often?
Should Our Indicators of Focus Change?
Effective Quality Improvement
Requires Prioritizing!!
*How frequently do you revise your
monitoring system (adding/omitting
indicators)? Quarterly
Annually
Rarely (> 5 years)
Never done it 16
17
Feasible Valid Understandable Specific Sensitive Reliable: Repeatability, Robustness, Reproducibility Mainz, J. (2003). Defining and Classifying Clinical Indicators for Quality Improvement. International Journal for Quality in Health Care, 15(6), 523-530.
18
How can my indicator be
valid if I am not sure my
measurement is?
19
20
21
Balanced Quality Cost Budget
Intended Use Risk Assessment
COST
CAPABILITY FMEA
22
How Much Error can I Tolerate? EXAMINE WHAT YOU TOLERATE…
vs
23
Total Error Tolerated?
Accuracy—true/traceable to a standard? Do I always need this?
Bias/Systematic Error (SE)-why do different machines give
me different results?
Linearity – Define Limit Reportable range of results?
Specificity-measuring the quality I intended despite potential
interferences?
24
RANDOM VARIATION Error Tolerated?
Precision: Random Error (RE) Repeatability/Robustness –
Sensitivity-Noise to Signal Ratio See through the static? Resolve small differences in quality despite assay
imprecision?
25
Compare Tolerance of Error with
Measurement Capabilities
26
Is my process in a state of statistical control? - Does it matter?
Quality Control Charts
27
Control Charts – Levey Jennings
Defining Limits?? Mean, SD, %CV
Trends & Patterns
Shifts/Drifts of Quality
Reporting & Visualization Data
Shewart, Demming, Westgaurd (lab), etc….
28
Common Cause vs Special Cause Variation
DEMMING SAYS: DON’T TAMPER
29
Measurement Methodology of Clinical Trial Cells Data is limited –experience and publication minimal
Limited understanding of the process, what variables are really
affecting my quality?
How do you define quality in this new product? Who defines it?
Lacking manufacturer controls, manufacturer validated test kits,
established standards, clinical reagents, proficiency surveys, etc.
30
Clinical Trial Cell Challenges… Is cell count/viability/dose always the most meaningful
indicator to predict therapeutic outcomes?
Do we always know what the true target cell marker is?
New indicators may be needed in order to measure the
most meaningful quality attributes
New Measurement Systems may need developed
Need Controls that meaningfully monitor variation in
assays-(i.e. frozen cell bank controls)
31
Standardization Needs: Community peer
comparison initiatives?
Collaboration with Manufacturers &
Standardization Organizations for Support
32
Develop Balanced Facility Quality Dashboard
Review Plan Optimize for Improvement Focus
Use SPC Understand Measurement Capabilities
Balance Investment in Capabilities according to Risk- Determined Tolerances
Seek Inter-laboratory Comparison Initiatives
Petition Manufacturer Development Support
33
Questions
35
Post Thaw Viability Scenario….
Facility began post thaw 7-AAD viability by Flow Cytometry for HPC, Apheresis products after a limited validation of 10 testing events (due to cost and low volume), all performed by one tech.
Upon implementation, noticed methods averaged 80% Viability post thaw as seen in validation, but rare outliers 50% post thaw viability with no manufacturing cause discovered during investigation. Engraftment outcomes were as expected.
Has this ever happened to you???
36
Indicator Purpose:
Monitor stability of freeze/thaw manufacturing
process to detect clinically significant differences
in quality and functionality of cells.
WARNING FLAG for problems,
Evidence of acceptability/improvement during
process changes
37
Having assay standardized to a gold
standard is not as important ….
The question is not….
Is the Viability of cells
as they are IV infused into the patient
really 50% or 80%?
38
BUT is my assay capable to indicate
when the clinical quality of my cells
increases or decreases…
or is there too much “NOISE to hear”?
39
To Be Meaningful Assay Must….
Exhibit a Large Signal
(sensitive to real changes in quality)
Have Small “Noise”
(comparably small amount of random error uncertainty and imprecision in the result)
Desired Tolerance: +/- 10% Viability
Discovered
Assay NOT ROBUST—
Minor technique difference in tech preparation
vast differences in results
Poor Repeatability due to cell clumping
No meaningful QC available
Investigation annual process data: CV 40%
40
IMPROVEMENT ACTION:
NOISE --Random Error
pipette measurement capability &
standardized rate diluent delivery electronic
positive displacement technology
Optimized “Thaw Diluent”: clumping
Coordinated 1 hour analysis time limit
Increased detail SOP
Performed training
41
Next years processing data:
CV% Process decrease from 40 to 17%
Repeats gave near identical results
between operators.
Indicator now had value for intended use.
42
43
*Do we need to transfuse
PLTs or NOT?
Cancer center following engraftment of patients post
autologous HPC, Apheresis transplant. Patient CBCs
tested on small office Horiba Micros60 analyzer, but
sometimes sent to hospital analyzed on LH750.
Transplant nurses noticed significant discrepancies
occasionally (i.e. > 30 PLTS difference) between
machines.
QC was acceptable both instruments.
Manufacturer says they are both functioning within
spec??? 44
45
Discrepant Results Monitoring patient
engraftment PLTs Transfusion Decisions?
Tolerance Limits
CLIA allows a tolerance limit of 25%
total error for PLT counts on hematology analyzers for
acceptable PT testing criteria as printed in the Federal Register Feb. 28, 1992; 57(40):
7002-186. Theses ranges are commonly used to define Analytical Quality Requirements.
46
Tolerance Limits
However, for critical medical decisions,
biological cut-offs for 13% Total Allowable Error
have been defined in the literature for PLTs
(or +/- 2.5 at a true PLT value of 20). See Desirable Specification for Total Error, Imprecision, and Bias, derived from intra- and
inter-individual biologic variation and Current databases on biologic variation: pros, cons and
progress. Scand J Clin Lab Invest 1999; 59: 491-500. Ricos C, Alvarez V, et al.
Also,Clinical Decision Levels for laboratory Tests, Second Edition (Oradell NJ; Meical
Economics Books, 1987; Statland BE.)
47
TRUE CAPABILITY ANALYZER HIDDEN:
Both Manufacturers State: Linearity down to 0.0 PLTs
PT testing in US and standard manufacturer QC does
NOT monitor precision/accuracy at critical PLT decision
points: 0-20 PLTs!
Most standard linearity kits provide limited testing at this
level and CLIA acceptability range is wide.
48
UK NEQAS (United Kingdom national external Quality Assessment Scheme for Haematology)
General Haematology Study presented by Barbara De la Salle , 2009.
Thrombocytopenic Platelet Counting
Low Platelet Counting in EQA- the UK Experience
Discrepancies observed in automated Counting EQA: What do they mean?
49
Using 13% TE cutoff…After Service!!!
50
51
# 0 1 2 3 4 5 7 8 9 10 112 9.3 20 23.1 37.6 95.7 134.1 191.2 227.8 622.8 1471 3730
3 10 18.5 24.3 37.2 95.5 139 190.1 232.9 605.7 1447 3731
4 9.7 19.4 23 38.4 95.1 134 201.9 229.2 615.6 1483 3506
5 9.3 19.7 22.9 38.8 97.3 136.5 193.2 237 610 1489 3726
Mean 9.58 19.40 23.33 38.00 95.90 135.90 194.10 231.73 613.53 1472.50 3673.25SD 0.34 0.648 0.66 0.73 0.97 2.37 5.36 4.12 7.39 18.57 111.52
CV% 3.55% 3.34% 2.81% 1.92% 1.01% 1.74% 2.76% 1.78% 1.21% 1.26% 3.04%
Expected
Results 9.705 19.41 23.1725 38.82 97.05 135.87 194.1 231.725 625.6575 1506.2125 3475.875
Range 7.2 - 12.2 16.91 - 21.91 20.67 - 25.67 33.82 - 43.82
Absolute Bias -0.13 -0.01 0.15 -0.82 -1.15 0.03 0.00 0.00 -12.13 -33.71 197.38% Bias -1.34% -0.05% 0.66% -2.11% -1.18% 0.02% 0.00% 0.00% -1.94% -2.24% 5.68%
Absolute
Accuracy
Limits 2.50 2.50 2.50 5.00 13.00 18.21 26.01 31.05 83.84 201.83 465.77
% Accuracy
Limits 25.76% 12.88% 10.79% 10.00% 13.40% 13.40% 13.40% 13.40% 13.40% 13.40% 13.40%
SD Limit @ 0
Bias 0.83 0.83 0.83 1.67 4.33 6.07 8.67 10.35 27.95 67.28 155.26
Acceptable? Passed Passed Passed Passed Passed Passed
Reference
Point:
Precision
Passed
Reference
Point:
Precision
Passed
Passed Passed Passed
Testing Performed by: Judy Charlton, MT, Hematology Specialist Date: 7/21/2011
Calculations & Graphical Analysis Performed by: Shadya George, MLS (ASCP)CM
Date: 7/25/2011
PLT Statistics Study: LH 750 AK06092
Total Error as %
-30%
-20%
-10%
0%
10%
20%
30%
0 500 1000 1500 2000 2500 3000 3500
Expected Result
% E
rro
r
% Bias
Upper Limit
Low er Limit
52
PLTs Ultra Low: Total Error as %
-40%
-30%
-20%
-10%
0%
10%
20%
30%
40%
0 10 20 30 40 50 60 70 80 90 100
Expected Result
% E
rro
r
% Bias
Upper Limit
Lower Limit
53
54
• Ceased monitoring engraftment on Horiba Micros60
• Routine purchase of ultra-low PLT linearity kit for LH750
• Within 2 years increased instrument capability,
Replaced Hospital LH750 with DXH800
Replaced office Horiba Micros60 with Sysmex XS-1000i
Will the real HCT please
stand up! 55
*Transplant Center has been using the LH750 to
analyze HCT of cell therapy HPC, Apheresis
product for use in optimization of Apheresis
collection. The hospital replaced LH750 with the
advanced DXH800. Shortly after, Cancer Center
installed a new Sysmex XS-1000i to for better
precision of very low PLT count monitoring. The
Processing facility began a validation study for
comparison, but having problems with analyzer
agreement on HCT. Which one is “right”? 56
57
Beckman Type Technology: Calculated HCT
HCT = MCV x RBCs MCHC=HGB x 100
10 HCT
Technology Review:
58
Beckman Type Technology: Calculated HCT
Technology Disadvantages:
WBCs are counted in same chamber as RBCs.
High WBC /low RBC common in cell therapy
products interfere with all RBC indices
HCT is significantly falsely elevated in WBC
counts > 100 x103 cell/uL
59
Beckman Type Technology: Calculated HCT Technology Disadvantages: Example Typical Interferences
60
CALCULATED CORRECTION POSSIBLE LH750 & Similar Technologies:
Assuming 100% WBC interferences, acceptable correction can be
obtained mathematically. Improvement in use of corrected HCT
for optimization of collection, with adjustment in ranges.
• Use patient peripheral MCV, • Subtract WBC from RBC count • Manually Recalculate HCT
61
Beckman Type Technology: Calculated HCT
DXH800 technology:
Auto-Correction of Interferences
Advances in algorithms developed for
abnormal peripheral bloods; less than
satisfactory for cell therapy products.
62
DXH800 technology: Disadvantages: Manual calculated corrections impossible
due to differing degrees of failed “auto-correction”
technology. Significant unpredictable error, erratic results,
limited usefulness for collection optimization
63
Manual Spun Hematocrit
Significant Interferences:
WBCs appeared trapped in the RBC fraction,
poor separation –
--falsely elevating the apparent HCT value.
Additional limitations: poor resolution, difficulty
reading due to large buffy coat, and trapped
plasma
64
Sysmex-Direct Measured HCT
Based on the cumulative pulse heights of all
the RBCs counted as proportional to cell
volume. No calculation interferences
65
HCT Counts: Technology Comparison
LH750 #1 LH750 #2DXH800 #1
Manual Mode
DXH800 #1
Cassette
(2mL)
DX800#2
Manual
Mode
DX800#2
Cassette
(2mL)
Patient
Peripheral
Concurrent
MCV
LH750
Average
DXH800
Average
6 Ralph Tallent #2 11/9/2011 2.21 2.18 1.78 1.78 1.78 1.78 89.2 2.19 1.78
5 Ralph Tallent #3 11/9/2011 2.46 2.24 1.87 1.87 1.87 1.87 89.2 2.35 1.87
1 Donna Moore #2 11/7/2011 3.01 3.09 2.92 2.73 94.3 3.05 2.83
13 Wanda Miller #2 10/12/2011 3.35 3.10 2.99 106.8 3.35 3.04
2 Donna Moore #3 11/7/2011 3.48 3.33 2.92 2.83 94.3 3.41 2.88
14 Wanda Miller #3 10/12/2011 3.48 2.78 2.88 106.8 3.48 2.83
8 William Wells #3 11/14/2011 3.88 1.92 1.92 91.3 3.88 1.92
15 Wanda Miller #2 10/13/2011 4.27 3.80 3.80 105.5 4.27 3.80
3 Ralph Tallent #2 11/8/2011 4.52 4.24 3.89 3.80 4.15 88.4 4.52 4.02
17 Wanda Miller #2 10/14/2011 4.52 5.48 5.48 105.4 4.52 5.48
4 Ralph Tallent #3 11/8/2011 4.75 4.15 4.42 4.51 4.51 88.4 4.75 4.40
16 Wanda Miller #3 10/13/2011 5.25 5.24 4.43 4.64 105.5 5.25 4.54
7 William Wells #2 11/14/2011 5.50 2.47 2.56 91.3 5.50 2.51
18 Wanda Miller #3 10/14/2011 8.43 8.01 8.01 105.4 8.43 8.01
Average 4.22 3.22 3.27 3.92 3.04 3.62 4.21 3.57
Difference
r
LH750 vs DXH800
Calculated Corrected HCT : LH750 corrected using corrected RBC (Reported RBC - WBC) and patient concurrent MCV. DXH800 corrected
using patient concurrent MCV only (Instrument provides RBC auto correction in presence of high WBCs)
0.65
0.8517
DatePatient
Note: Wells sample contains Giant Plts flag which could interfere with the RBC count. DXH800 may be better at correcting
for this error, hence the two points that do not agree well. Without the well's points superior agreement.
Conclusion: DXH800 runs just slightly lower corrected HCT on average, clinically insignificant. Corrections equivalent.
#
66
HCT counts: Technology Comparison
HCT Comparison
y = 0.9172x - 0.2971
R2 = 0.7254
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
0.00 2.00 4.00 6.00 8.00 10.00
LH750
DX
H800
67
WHICH ANALYZER IS REPORTING
THE “RIGHT” HCT?
Purpose: optimization of Apheresis collection to
limit granulocyte contamination and increase
efficiency collection.
68
Again…
Having assay standardized to a gold
standard is not as important ….
The question is not
Is the HCT really 3, 2, 9 or 12%?
69
BUT is my assay capable or SENSITIVE enough
for my intended use?
Increase and decrease proportionately to
the actual product HCT?
Specific admidst fluctuating interferences?
Can it be used to direct in apheresis
settings to improve collection?
Outcome:
*LH750: Nurses report acceptable indicator use in setting
collection parameters using corrected values. However,
machine is gone.
*DXH800: Nurses report frustration and lack of
consistency in applying corrections based on product HCT.
Corrected and uncorrected HCT reported as not helpful.
*Recommended switching to HCT analysis using the
Sysmex XS-1000i, and re-setting new acceptable
expected ranges using statistical analysis.
70
Stability Program
You prepare a comprehensive Stability Program. Your
program includes the cryopreservation of 6 vials to
be tested at: 30, 60, 90 days and 1, 3 years. One of
the products included in program was collected on
9/22/2015. You start acquiring data and have the
following results:
The first 4 specimens yielded the following results:
71
Pre
Freeze TNCx1010 (Vol x WBC) =
7.10
7AAD (%
Alive)=
99.49
CFUx104/
kg=_____
Not
Done
CD34 %
=1.03
CD34x106/kg=
10.23
Post
Thaw WBC
(x106/ml)
TNCx1010
(Vol x WBC)
Total 7AAD
(% Alive)
CFU
(Growth) CD34%
CD34
Only
%Viabilit
y
Viable CD34 Dose
(x106/kg) (%
Recovery)
Tech Initials
30 days 276.0 7.73 52.48 Yes
No 1.54 68 7.81(43%) LC/MM`
60 days 290.0 8.12 49.55 Yes
No 2.74 92.7 13.78(76%) LC/RR
90 days 282.0 7.90 50.62 Yes
No 3.07 96.5 15.34(84%) LC/LCR
1 year 276.0 7.73 56.70 Yes
No 2.82 81.9 15.45(85) JM/YY
72
1. Is the difference in results due to different staff, different technique?
2. Should I check the gating technique used for all samples?
3. Should I trust these results? What this is really telling me?
1. Is the difference in results due to different staff, different
technique?
*We decided to assign a team that does most of the post-thaw together. We
stated to get better results, more “credible”
2. Should I check the gating technique used for all samples?
*When we went back to check gating, we noticed we were all over in our
technique. With time, more cells (grans) will die and gating became more
challenging. We decided to “standardize” gating by using similar viability
gating for all vials. Now we checked results from previous thawing and
compare gating.
73
74
ORIGINAL GATING RE- GATED
75
ORIGINAL GATING RE- GATED
Pre
Freeze TNCx1010 (Vol x WBC) =
7.10
7AAD (%
Alive)=
99.49
CFUx104/
kg=_____
Not
Done
CD34 %
=1.03
CD34x106/kg=
10.23
Post
Thaw WBC
(x106/ml)
TNCx1010
(Vol x WBC)
Total 7AAD
(% Alive)
CFU
(Growth) CD34%
CD34
Only
%Viabili
ty
Viable CD34
Dose (x106/kg)
(% Recovery)
Tech Initials
30 days 296.0 6.22 90.1 Yes
No 0.93 93.4 5.74(103%)
60 days 300.0 6.30 88.8 Yes
No 1.01 96.1 6.23(117%)
90 days 288.5 6.06 90.6 Yes
No 0.92 96.4 5.56(104.7)
76
With standardization we started to see “better” results. One thing we noticed:
1. We continue to recover >100% of cells. Why?
2. Have you experience a change in methodology, ex. new cell counter with different technology,
and results for WBC from post-thaw are very different from counts made with previous instrument:
a. Did you stop the stability testing for the rest of the vials?
Top Related