Assay validation, performance and quality control · Assay validation, performance and quality...

Assay validation, performance and quality control

Computational Aspects of High-Throughput Screening planning and analysis EMBL Heidelberg, Germany Monday 14 October - Wednesday 16 October 2013

David Murray October 2013

A few words about me …

• A biochemist by training • Currently an Associate Principal Scientist in the AstraZeneca

global HTS unit based in the UK • 5 years in HTS and 10 years in HTS output profiling • Experience across biochemical assays and cell based assay

including human primary cells • Interest in exploiting statistics, QC data and data analysis

techniques to extract the most information from assays

!2


High Throughput Screening in AstraZeneca

• A screen of >50, 000 compounds/wells • Can be either a biochemical or cell based assay • Run in 1536-well plates where possible • A screen is typically between 250K and >>1 million wells • A screen takes from a few days to weeks depending on complexity

!3

HTS is a significant investment of time and money and remains a key activity to generate chemical equity for AZ and in the wider industry


Basic HTS Process

!4

Assay transfer

Optimisation on

automation platform

Validation

Review

Primary Screen

XC50s

Confirmation test

Results Database

Confirmation testing depends on hit rate in primary screen

Drug discovery project


Some terminology

• Controls - Neutral controls

• Controls in an assay at steady state • e.g. In an inhibitor assay represent maximal activity in absence of

an inhibitor - Scale reference control

• The activity in presence of a supra-maximal concentration of a compound

• Or no added enzyme

!5


Key to HTS success – the primary screen

• In most cases a primary screen consists of each compound being tested just once at a single concentration

• It is vital that the assay is reliable enough such that this single test is a true representation of the activity of the compound

• We also need to be sure that each plate in the screen is valid and that data is comparable across plates and experimental runs

!6

Why is this important? Consider an assay where if you tested a compound twice you got the results of 60% on day 1 and 35% on day 2. Cut-off in the assay is 50% On day 1 the compound would be a hit and on day 2 it would not be selected

QC of single shot assays


Single shot assays

• Compounds tested once at a single concentration

• Compounds typically have their activity normalised to 2 controls, the neutral and scale reference

- This allows us to compare data across plates and experiments on a common scale

• The separation of the values of these controls dictates how reliable the assay is at detecting active compounds in the presence of typical errors of measurement

!8

Neutral

Scale reference

0%

100%

50%


Standard methods for QC of HTS assay data

Mean Neutral Control

Mean Scale Reference

This looks like a good assay?

S/B = 8

Assay S/B alone does not define the quality of an assay and alone can be misleading



S/B = 8

…but not anymore !!

+/- 3SD

+/- 3SD

S/B = mean signal / mean background

!9


A good assay has ….. •Clear separation between the tails of the distributions of the controls •A large signal window or separation band (>3-fold)

What would be a better approach ……. take into account the standard deviation around the data

We can transform this principle into a simple number

Separation band Data variability band

Data variability band3 SD

3 SD



!10


Z’/Z factor – a screening window coefficient

NeutralScale

Reference

Z’ value Structure of Assay Related to screening

1 SD=0 or window ➡ ️∞ An ideal assay

1>Z’>=0.5 Separation band is large An excellent assay

0.5>Z’>0 Separation band is small A marginal assay

0 No separation band – variation bands touch

A “yes/no” assay

<0 No separation band – variation bands overlap

Single point screening impossible

Zhang et al. Journal Biomolecular Screening 4(2) 1999

!11


Z’/Z factor – a screening window coefficient

•Point removal only “fixes” the Z’ factor and is very subjective •Quite often the screener chases a Z’ value

•Mean and standard deviation used in the calculation are not robust to outliers due to plate position effects etc.

•Extreme values (outliers) in either the neutral or scale reference controls can adversely affect the Z’-factor leading to a poor estimate of the Z’

NeutralScale

Reference

!12


Robust Statistics

• Normal statistical measures work very well with normally distributed data with few or no extreme outliers

• HTS data contains outliers and the whole purpose of a screen is to find these outliers - In the presence of outliers mean and standard deviation can

become inaccurate estimates of the centre and spread of the data

• To ensure we get good statistical estimates we employ robust measures - Robust measures are resistant to outliers and so provide

good statistical estimates from HTS data

!13


Robust Statistics

Robust statistics are more resistant to the presence of outliers than classical statistics based on the normal distribution

2,3,2,4,3,4,3 Mean = 3

2,3,2,4,3,11,3 Mean = 4

2⬇ ️, 2⬆ ️, 3➡ ️

5⬇ ️, 1⬆ ️, 1➡ ️

Mean is no longer a representative measure of the central tendency of the data

2,2,3,3,3,4,4 Median = 3

2,2,3,3,3,4,11 Median = 3

2,2,3,3,3,4,106 Median = 3

The median is robust towards the outlier

(mean = 142,859)

!14


Averages can be misleading …

• The average number of legs per person in Sweden is slightly less than 2

A silly but "true" example ( from Hans Rosling)

From BBC TV’s “The Joy of Stats”

!15

- It is 1.999 legs per person

- This is due to some people having 1 or no legs and no one having 3 legs

- The median and the mode is 2


Robust Z’/Z factor – a better screening window coefficient

•No need for point removal – the data is what it is •No chasing a Z’ value

•Median and robust standard deviation used in the calculation are robust to outliers due to plate position effects etc.

•Extreme values (outliers) in either the neutral or scale reference controls will not lead to a poor estimate of the Z’

NeutralScale

Reference

!16

Assay Validation


Assay Validation

• A series of experiments and subsequent data analysis to ensure that the assay is reliably detecting compounds having an effect on the biological target using the equipment and reagents to be used in the primary screen

• Typically in a secondary screening scenario we have known inhibitors we can use to determine if the assay is reliably detecting active compounds

• In HTS often we have no known inhibitors

What does this mean?

!18

In validating an assay we must employ some replication to assess the reliability of the assay.

As a minimum 2 experiments across 2 days


Validation

• 2 approaches can be taken • Construct a validation set that is representative of the diversity in your

screening collection - For large scale HTS this is typically around 10, 000 compounds

• Select at random a number of plates from your collection - Screen these plates at least twice across 2 days

• Assess the level of reproducibility across the repeated data sets • False discovery rate (active once out of 2 replicates) • Median of the differences between the 2 replicate values

!19

We need some active compounds to assess validation across the activity range Add in some known inhibitors or construct a set with known frequent hitters

Assessing validation data


Why may a compound show different activity day to day?

Typical sources of errors

!21

• Errors adding compound • Errors adding assay reagents • Reader errors • Edge effects • Patterns on plates in general • Positioning of controls

Edge effects assessed in heat map view


Look at the distribution of the data – is it as expected?

!22

Centred at or near 0

Long tail containing active compounds

• In screening the data is typically not normally distributed

• The distribution has long tails and is not symmetrical

• This means that the normal assumptions such as 3*SD = 99.7% of the data do not hold true

• The long tails mean that average and standard deviation are often not valid measures


How to compare data

!23

Typically we use XY plots

• Shows us that high values associate with high values, low with low

• This plot produces no statistical measures

• Beware of correlation coefficients as a measure of agreement

• If I added 50 to one data set the correlation coefficient would remain unchanged as the data still correlates

• However it is now not in agreement

There is a better way …


Bland Altman Plots

!24

This plots the difference between pairs of measurements vs. the means of the pairs to assess agreement between the data sets

Average difference

95% confidence interval of the differences

95% confidence interval is useful to judge reproducibility of the assay assuming average difference is on or close to 0

Bland & Altman Lancet 1986 307 - 310

False +ve or -ve


Summary

• Does the distribution of the data look as expected? • Do I see any systematic plate patterns in the data? • When comparing the replicates do they:

- Show a mean difference between replicates of close or on zero between experiments

- Are the confidence intervals of the average difference acceptable? - Remember to look at both active and inactive compounds separately

as well as together in the Bland Altman plots

!25

Many other measures and tests can be employed but the key ones above are a good place to start as you build a validation workflow

Quality Control during a primary screen


Quality control during an HTS Primary Screen

!27

The following should be tracked during a screen across each plate screened; rZ’ – plates should be rejected if below a threshold Neutral and scale reference controls should not vary significantly across plates Plot median of compound wells and control wells over time to look for trends and drifts Incorporate a compound control run at a concentration to give 50% effect and monitor over time

If drift is large then consider smaller plate batches


Monitor between experimental runs

• Monitor rZ’ and control signals on individual plates - neutral, scale reference and 50% activity control if used

• Also monitor mean and median of the above across experiments to look for trends over time

• Can use Manhattan analysis to detect changes but usually this is not necessary

!28

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Media

n rZ' p

er ru

n

Run

Overall median

across all runs

These runs look suspect and should be reviewed

For example:


Looking across large data sets

!29


Box Plots• John Tukey developed the box plot to display a 5 number

summary of a set of data. Additionally the box plot is the only chart that displays outliers and so is a very useful display of HTS data

!30

3Q+1.5xIQR

Q1-1.5xIQR

John Tukey has provided a precise definition for two types of outliers: 1. Outliers (or extreme outliers) are either 3×IQR or more above the third quartile or 3×IQR or more below the first quartile. 2. Suspected outliers are slightly more central versions of outliers: either 1.5×IQR or more above the third quartile or 1.5×IQR or more below the first quartile.

3Q+3xIQR

Key statistical measures

Outliers


Rapidly assess across your plates

!31

Plate ID

% E

ffect

Very rapidly spot plates showing a markedly different distribution of data

Assessing performance of concentration response assays


Assessing performance of Concentration Response (CR) assays

• Typically in HTS the compounds are only run once in an IC/EC50 screen - However, the compounds will be run in other labs and so a measure of

reproducibility is important • Similar QC steps should be applied as for a primary screen

- Are scale and neutral controls consistent? - Is there a drift in assay signal over time?

• rZ’ is not as useful a measure in CR assays - You can get meaningful data from a CR assay with a rZ’ of 0 - However, rZ’ can be used as an indication of consistent assay

performance (see slide 29)

!33


QC in Concentration Response Assays

• We need to look at the variability of repeated IC50/EC50 determinations

• 2 ways to achieve this: - “Borrow” information from standard compounds that are

included in every assay • Good for SAR assays

- Repeat a number of compounds (typically 30 or more) and assess variability from this data • Easier to achieve for HTS data

!34

We can therefore assess how close together repeated IC/EC50 determinations are giving rise to a measure called Minimum Significant Difference (or Minimum Discriminatory Distance)


A word about reporting IC/EC50s

• IC/EC50s reported as molar values are on a proportional scale - This means errors are not symmetrical

• 10nM +/- 30nM would mean IC50 ranges from -20nM to 40nM on a linear scale. Evidently this is nonsense.

• Can only report fold differences, in this case +/- 4-fold (2.5nM – 40nM) • If we report log10 IC/EC50s we put the values on a linear scale

- If we report –log10 IC/EC50 we get pIC50 and pEC50 - As compounds get more potent their pIC/EC50 gets larger - Errors are now symmetrical

• pIC50 8.0 +/- 0.6 does make sense

!35


Determining Minimum Significant Difference (MSD)

• Carry out an experiment to determine IC/EC50 of 30 or more compounds twice in independent experiments

- Try and select compounds spanning the potency range in the assay • Calculate the pIC/EC50 for all compounds • Calculate the log potency differences for each compound • 2*SD of log potency differences = MSD

!36

We also need to plot the 2 replicates in a Bland Altman plot to further determine how the assay is performing

Eastwood et al. Journal Biomolecular Science 11(3) 2006

Average pIC50

pIC

50 D

iffer

ence

Average difference on or close to 0

Limits of agreement are not too large in context of the assay. These relate to the individual potency differences

Looking at the replicate data


Assessing the assay• The Bland Altman plot shows that the average difference in pIC50 is 0.03 which is very close to 0

so there is no significant bias from one experiment to the other • There are no trends in the data and variability is reasonably uniform across the range of potency • The limits of agreement of 0.26 and 0.19 are acceptable • The MSD calculates at 0.23

!38

• What does MSD tell us? • It tells us the largest potency difference that can be considered random

• We use our judgement as to whether this value is acceptable • As a rule we find that most biochemical assays have an MSD of 0.2 – 0.3 and most cell

assays have an MSD of 0.4-0.5 • To be sure that the pIC50 for one compound is truly different to that of another compound the

2 pIC50s must differ by >0.23

You need to use your judgement as to what is acceptable for an assay

David Murray October 2013!39

Remember, statistics can guide you but …

Statistics are no substitute for judgment.

David Murray October 2013!40

Confidentiality Notice This file is private and may contain confidential and proprietary information. If you have received this file in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorized use or disclosure of the contents of this file is not permitted and may be unlawful. AstraZeneca PLC, 2 Kingdom Street, London, W2 6BD, UK, T: +44(0)20 7604 8000, F: +44 (0)20 7604 8151, www.astrazeneca.com


Some basic robust concepts

Median or robust mean is the robust equivalent of the average

MAD = median absolute deviation which is a robust measure of how close a set of data is to the median

The robust standard deviation is MAD * 1.483

The Interquartile Range is another robust measure of how close a set of data is to the median

!41

Assay validation, performance and quality control · Assay validation, performance and quality...

Documents

Transcript of Assay validation, performance and quality control · Assay validation, performance and quality...