Post on 11-Jan-2016
description
A new exact test to globally assess a population PK and/or PD model
C. Laffont & D. Concordet
UMR 181 Physiopathologie et Toxicologie ExpérimentalesINRA, ENVT
ECOLENATIONALEVETERINAIRET O U L O U S E
• Weighted residuals
– WRES NONMEM
– CWRES Hooker et al. (2007) Pharm Res 24:2187
– PWRES Monolix
Classical metrics for global model evaluation
Weighted residuals: difficulty in interpretation
Simulated data analysed with the correct model !Simulated data analysed with the correct model !
CWRES WRES
Karlsson & Savic, 2007,Clin Pharmacol Ther 82:17
Diagnosing Model Diagnostics
Weighted residuals (WR) calculation:
)()( 2/1iiii YEYYVarWR
• Calculated from the vector of observations Yi in subject i and denoted here WRi
• Simulation (PWRES)
• FO/FOCE approximation of the model (WRES,CWRES)
• Simulation (PWRES)
• FO/FOCE approximation of the model (WRES,CWRES)
Expected distribution if the model is correct ?
I)N(0,~WRi
For linear Gaussian models, Yi distribution is Gaussian so:
For nonlinear models, WRi distribution is unknown !!!
Recently proposed metric: NPDE
• Simulation-based approach
– compares at each time j the distribution of WRij with their predictive distribution according to the model
Brendel et al. (2006) Pharm Res 23: 2036
Expected distribution if the model is correct ?
NPDE are assumed to be independent and Gaussian:
I)N(0,~NPDE
Independence issue rightly discussed by the authors:
when NPDE are dependent, they are jointly not Gaussian
New metric : GUD (Global Uniform Distance)
The purpose of this work was to propose:
• an exact test for global model evaluation
• an easy diagnostic graph with no subjective interpretation
Reject model with 5% risk= black line (data) outside the ring = black line (data) inside the ring
Do not reject model
GUD calculation & testing
• Step 1Step 1
– As for calculation of NPDE, we compute for each subject i the vector of WRi
)(ˆ)(ˆ 2/1iiii YEYYarVWR
M (=2000) simulations (unbiased)
2/1)( iYVar is the Cholesky decomposition of the full variance matrix of Yi
WRi are decorrelated within subject i
Decorrelation does not imply independence !!!
Decorrelation= independence
Decorrelation= independence
Decorrelation independence
Decorrelation independence
• Only true for linear Gaussian models
iiiii tY 21
• Case of nonlinear models
1 compt i.v. model
We project WRi vector on R random vectors eir taken from a uniform distribution on the unit sphere
WRi vector (observations)
Unit sphere
r = 1… R
• Step 2:Step 2:– To handle data dependency, we use a recent random
projection method (See Cuesta-Albertos et al. (2007) for an application)
vector ei1
ProjectionProjectioni1i1
GUD calculation & testing
Unit sphere
WRi vector (observations)
vector ei2ProjectionProjectioni2i2
GUD calculation & testing
• Step 2:Step 2:– To handle data dependency, we use a recent random
projection method (See Cuesta-Albertos et al. (2007) for an application)
r = 1… R
We project WRi vector on R random vectors eir taken from a uniform distribution on the unit sphere
Each subject i
)(, WRiiri feWRProjection on R (=100) random directions
...
)(WRif
Subject i
)(1 WRf
Subject 1
GUD calculation & testing
cdf cdf
)(2 WRf
Subject 2
Random pdf independent between subjectsRandom pdf independent between subjects
Mixture of projection distributions
GUD calculation & testing
• Step 3:Step 3:
– Compare this global cdf obtained for the sample to its distribution under H0 (i.e. correct model)
cdf cdf
SampleSample Simulations under HSimulations under H00
95% prediction region
95% prediction region
For each replicate, we compute the maximal absolute distance
from mean cdf curve (GUD)
GUDGUD
= Global Uniform Distance= Global Uniform Distance
00
K cdf curves
Calculation of 95% prediction region under H0
K =(5000) replicates of the study design
Simulations under H0 with tested model
Simulations under H0 with tested model
mean cdfmean cdf
95%5%
5% of curves that 5% of curves that are the most distant are the most distant
from mean cdffrom mean cdf
5% of curves that 5% of curves that are the most distant are the most distant
from mean cdffrom mean cdf
Calculation of 95% prediction region under H0
mean cdfmean cdf
Uniform region containing 95% of curves
Uniform region containing 95% of curves
= Global Uniform Distance= Global Uniform Distance
95% prediction region
For each replicate, we compute the maximal absolute distance
from mean cdf curve (GUD)
K cdf curves
K =(5000) replicates of the study design
Simulations under H0 with tested model
Simulations under H0 with tested model
GUDGUD0
95%
0
GUDGUD
GUD test for your sample
Sample cdf
Sample
P valueP value
Do not reject modelDo not reject modelSimulations under H0Simulations under H0 True modelTrue model
= Global Uniform Distance= Global Uniform Distance
5%
95% prediction region
0
5%
P valueP value
Sample
Wrong modelWrong model
Sample cdf
GUD test for your sample
Reject modelReject model
GUDGUD
= Global Uniform Distance= Global Uniform Distance
95% prediction region
Sample
QQ ring diagnostic plot
Reject modelReject model
Sample
Do not reject model
Do not reject model
PK model PK/PD model Linear model
1 compt i.v. bolus Sigmoidal Emax model (γ=4)
Exponential IIV on CL, V
Proportional res. error
Exponential IIV on Emax, EC50
Additive res. error
Additive IIV on 1, 2
Additive res. error
Performances under H0: GUD vs. other metrics
iiiii tY 21
• Simulations under H0 to evaluate the level of the tests 100 subjects i.i.d with 4 obs./ subject 5000 replications of study design
• Weighted residuals & NPDE calculated in C++ Kolmogorov-Smirnov (KS) test to test for a N(0,1)
PK model
PK/PD model
Linear model
KS testfor N(0,1)
WRES
CWRES
PWRES
NPDE
100
57
66
4.0
100
13
62
7.5
4.9
4.9
4.5
4.6
GUD test GUD 5.1 5.0 4.9
Level of the tests under H0
Type I error (%) - nominal level = 5%
95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz)
p value
The p-value should follow a uniform distribution under H0 !
Exp
ecte
d v
alu
e fr
om
u
nif
orm
dis
trib
uti
on
05.0)05.0(:Ex valuepProb
PP plotPP plot
Level of the tests under H0
GUD testGUD test
PK model PK/PD model
p value p value
Exp
ecte
d v
alu
e fr
om
u
nif
orm
dis
trib
uti
on
95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz)
Good whateverGood whatever the level !the level !
Good whateverGood whatever the level !the level !
Level of the tests under H0
NPDE: KS test for N(0,1) NPDE: KS test for N(0,1)
PK model
95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz)
p value p value
Exp
ecte
d v
alu
e fr
om
u
nif
orm
dis
trib
uti
on
or or of of type I errortype I error or or of of type I errortype I error
PK/PD model
Level of the tests under H0
Conclusion• Poor performances of weighed residuals
• NPDE show much better performances but do not deal with the issue of data dependency within subjects
– Possible increase or decrease of type I error depending on model
• New test and graph based on GUD metric
– Encouraging results
– More work needed to evaluate this test under more complex conditions (different sampling times per subject, real-case data…)