1

The analysis of longitudinal studies in rheumatology:

Is it done correctly?

Emmanuel LesaffreEmmanuel LesaffreDepartment of Biostatistics, Erasmus MC, Rotterdam, the Netherlands

L-Biostat, K.U.Leuven, Leuven, Belgium

6th Annual Clinical Research Methodology 6th Annual Clinical Research Methodology CourseCourse

Friday, December 16, 2011

In collaboration withIn collaboration withKarolina Sikorska, Maurits de Rotte, Jolanda Luime, Mieke Hazes

2

Conclusions

1. Many longitudinal studies in rheumatology are not using an appropriate statistical methodology.

2. Repeated significance tests and summary statistics methods are inappropriate in practice and can not show the evolution of an outcome over time.

3. Classical repeated measurement methods are inappropriate in practice.

4. The mixed effects model and the GEE approach can make use of all available cases independently whether they have data on all time points studied.

5. The mixed effects model and the GEE approach are sparsely used in longitudinal studies rheumatology.

3

Description JIA study

44

Juvenile Idiopathic Arthritis (JIA) study

• Description

– Longitudinal study: 5 time points (0, 3, 6, 9, 12 months)

– Recruitment at University Medical Centre Utrecht (UMCU) and Wilhelmina Children’s Hospital, the Netherlands

– Children diagnosed with JIA according to the ILAR criteria and started methotrexate (MTX) therapy between 1990 and 2006

– All patients and their parents gave their informed consent and approved by the medical ethics committee of the UMCU

– Treatment was tight-controlled using a standardized report form on disease activity every 3 months

– Information on MTX usage, disease activity, route of administration, dosing of MTX, reasons for ending MTX treatment, concomitant therapy and laboratory parameters were collected

55

JIA studyJIA study • Description

– 183 JIA patients of our test cohort were genotyped for ABCB1 3435C>T.

– 2 groups: genotype CC homozygous (53 patients, 28%) T-carrier (129 patients, 72%)

– Continuous response = erythrocyte sedimentation rate (ESR) was obtained on all 5 visits for 106/183 patients (58%): CC homozygous (31 patients, 29%), T-carrier (75 patients, 71%) = complete cases

– ESR was log-transformed because of a skewed distribution.

– Binary response = ACR30 response, obtained for all 183 patients on all visits. Of the 183 patients, 46 (25%) responded at 3 months, 85 (46%) at 6 months, 110 (60%) at 9 months and 113 (62%) at 12 months.

6

Classical analyses for ESR

77

JIA studyJIA study • Classical analysis (ESR)

88

Classical Classical approachesapproachesfor comparing 2 groups over timefor comparing 2 groups over time

1. Repeated significance tests (t-tests, Wilcoxon tests, etc)

2. Summary Statistic Method: summarizing whole curve by e.g. AUC and compare AUC between 2 groups with t-test

3. Repeated measurements ANOVA

4. MANOVA

9

1. Repeated significance tests1. Repeated significance tests

1010

1. Repeated significance tests1. Repeated significance tests JIA study (ESR)

• Here unpaired t-tests giving P= 0.066, 0.12, 0.62, 0.33, 0.44

• Nowhere significant no real issue here

• Suppose P= 0.066, 0.03, 0.62, 0.33, 0.44 … what to conclude?

• Or P= 0.066, 0.12, 0.62, 0.33, 0.03 … what to conclude?

• If no adjustment of P-value, then problem of multiple testing

1111

1. Repeated significance tests1. Repeated significance tests

• Popular approach in medical research

• But

– Inefficient: only patients are included that are present at visit

– Likely to be biased: missing data process can bias analysis

– No good insight in individual evolution: link between subsequent observations is lost

– Turns longitudinal study into several cross-sectional studies

• Not suitable for contemporary rheumatology studies

12

JIA studyJIA study

CC homozygous T-carrier

• All individual profiles (ESR)

13

JIA studyJIA study

• 3 individual profiles (ESR)

1414

2. Summary Statistic Method2. Summary Statistic Method JIA study (ESR)

• Compute Area-under-the-Curve (AUC) for each profile

• On complete cases only:

CC-Homozygous – T-carrier= -1.33, P = 0.54, 95% CI=(-5.61; 2.96)

• Complete the incomplete cases by LOCF:

CC-Homozygous – T-carrier= -3.78, P = 0.04, 95% CI=(-7.33; -0.21)

• In addition one could compare

– Averages of the profiles

– ….

15

JIA study

• 3 individual profiles (ESR): incomplete profiles completed by Last-Observation-Carried-Forward (LOCF)

1616

2. Summary Statistic Method2. Summary Statistic Method

• Not frequently used in medical research

• Recommended for balanced data

– Balanced: all patients are examined at all (regular) visits

– Difficult for unbalanced data, i.e. when there are missing observations and patients can come at all times

– LOCF is a popular approach to make data more balanced

• LOCF

– Imputes unrealistic values

– Underestimates variability of the data

• Not suitable for contemporary rheumatology studies

1717

3. Repeated Measurements 3. Repeated Measurements ANOVAANOVA

• One of only 2 approaches available for analysis of repeated measurements 50 years ago

• Requires balanced data

– Patients with missing values are excluded

– Time points must be regular, patients with irregular time points are excluded

– Restrictive assumptions on correlation between subsequent responses: correlations are equal (corrections available but not sufficient)

– Splits treatment effect up into time, group and group*time effect

• Not suitable for contemporary rheumatology studies

18

3. Repeated Measurements 3. Repeated Measurements

ANOVAANOVA JIA study (ESR)

• On complete cases only:

– CC-Homozygous – T-carrier=

• Time effect: P < 0.001

• Genotype effect: P=0.47

• Time*genotype effect: P=0.02

• Complete the incomplete cases by LOCF:

– CC-Homozygous – T-carrier=

• Time effect: P < 0.001

• Genotype effect: P=0.04

• Time*genotype effect: P=0.34

19

3. Repeated Measurements 3. Repeated Measurements

ANOVAANOVA JIA study (ESR)

• Assumption equal correlation??

TIME (months) 0 3 6 9 12

0 1 0.71 0.51 0.46 0.42

3 0.71 1 0.61 0.47 0.48

6 0.51 0.61 1 0.70 0.64

9 0.46 0.47 0.70 1 0.69

12 0.42 0.48 0.64 0.69 1

2020

4. MANOVA4. MANOVA

• Other of only 2 approaches available for analysis of repeated measurements 50 years ago

• Requires balanced data

– Patients with missing values are excluded

– Time points must be regular, patients with irregular time points are excluded

– No structure allowed on correlations large studies are required

– Splits treatment effect up into time, group and group*time effect

– Output more difficult to understand

• Not suitable for contemporary rheumatology studies

21

4. MANOVA4. MANOVA JIA study (ESR)

• On complete cases only:

– CC-Homozygous – T-carrier=

• Time effect: P < 0.001

• Genotype effect: P=0.47

• Time*genotype effect: P=0.03

• Complete the incomplete cases by LOCF:

– CC-Homozygous – T-carrier=

• Time effect: P < 0.001

• Genotype effect: P=0.04

• Time*genotype effect: P=0.36

22

Modern analyses for ESR

2323

““Modern” Modern” approachesapproachesfor comparing 2 groups over timefor comparing 2 groups over time

1. Types of missing data processes

2. Approaches to deal with missing data

3. Mixed effects models

4. GEE models

24

1. Types of missing data processes1. Types of missing data processes

• What are missing data?

• Data that are not observed

• How are missing data generated?

• Patients/clinicians forget to fill item(s) in questionnaire

• Patients refuse to fill item(s) in questionnaire

• Lost or damaged biological sample

• Patients miss a visit because …

• Patients decide not to return to clinician anymore

• Patient died

• …

Different reasons why data are missingDifferent reasons why data are missing

25

Impact of missing data

• What is believed

• Loss of efficiency

• In reality

• Loss of efficiency

• Biased results (often)

Classical statistical methods need to be adClassical statistical methods need to be adaaptedpted

• Approach

• CLEVER explicit or implicit imputation of missing data

• ButBut: can NEVERNEVER replace true data

26

Terminology

• Monotone missing

• Also called dropouts

1 2 3 4

1

2

n

• Non-monotone missing

• Also called intermittent missingness1 2 3 4

1

2

n

27

More terminology

• Missing completely at random

• Missing at random

• Missing not at random – informative missing

• Terminology might be confusing – Rubin (1975)

28

Missing completely at random (MCAR)

• Probability of missingness is independent of all responses

• Examples

• A random selection of teeth in mouth are taken in the study

• A blood tube is dropped

• Patient died in a car accident, but careful: patient could have experienced a sleep attack when taking a dopamine agonist

• Then

• Simple mean of response is unbiased estimate of true mean

• Classical statistical techniques (repeated t-tests, Summary statistics method, repeated measurements ANOVA, MANOVA) can be used

• Impact: loss of efficiency

29

Missing at random (MAR)• Probability of missingness depends on observed

responses

• Examples• Study design specifies that if blood pressure is not lowered

patient will be removed from anti-hypertensive trial

• Multi-stage screening: data are missing at subsequent stages due to result at initial stage (negative test)

• Then

• Simple mean of response is biased estimate of true mean

• Classical statistical techniques canNOT be used

• Statistical tests to distinguish versus MCAR exist

• Impact: loss of efficiency + bias, but likelihood analysis can correct for bias due to missingness

30

Missing not at random (MNAR)

• Probability of missingness depends on observed responses & unobserved responses

• Examples

• Patient shows a flare up in the disease unobserved in the study + patient decides to leave the study

• Then

• Simple mean of response is biased estimate of true mean

• Classical statistical techniques canNOT be used

• No test to distinguish versus MAR

• There is no satisfactory analysis, ONLY sensitivity no satisfactory analysis, ONLY sensitivity analysisanalysis

• Impact: loss of efficiency + bias, only a sensitivity analysis can shed light on the problem

31

Bias of simple mean

COMPLETE 2.07 4.41 6.87 9.03 11.57 13.30

MCAR 2.14 4.07 6.50 8.53 10.95 12.49

MAR 2.07 4.41 7.41 10.3010.30 17.1417.14 20.2220.22

32

2. Approaches to deal with missing data

• Prevention and planning

• Analytical remedies

• Complete case analysis

• Available case analysis

• Imputation techniques

• Likelihood-based analyses

• Weighted analyses

• Sensitivity analyses

Most Most appropriateappropriate statistical solutions are statistical solutions are COMPLICATEDCOMPLICATED

33

2. Approaches to deal with missing data

• Complete case analysis (MCAR)

– Default analysis in many packages, only ok for MCAR

– If not MCAR: substantial bias can be the result

• Available case analysis (MCAR)

– Use for each variable/time point all observations available

• Single value imputation (MCAR, MAR??)

– Mean value imputation

– Hot decking

– LOCF

methods are based on unrealistic models & underestimate variance

34

Example: LOCF

LOCF

time

35

2. Approaches to deal with missing data

• Multiple imputation (MAR)

– Explicit imputation of missing data

– Incorporate random mechanism

– Generate M different completed imputed data sets

– Combine M means and M variances 1 overall mean & variance

method is based on statistical model for imputation

36

2. Approaches to deal with missing data

• Likelihood-based models (MAR)

– Implicit Implicit imputation of missing values

– Model-based:

• Linear & generalized linear regression

• Linear & generalized linear mixed models

• …

method is based on statistical model for response

• GEE models (MCAR) and weighted GEE models (MAR)

• Bayesian models (MAR)

37

2. Approaches to deal with missing data

• MNAR models

– Model also missing data mechanismModel also missing data mechanism

– Complex modelling

– Never completely satisfactory sensitivity analysissensitivity analysis

– BUT: BUT: if time points are close to each then MNAR close to MAR

38

3. Mixed effects models

• Two examples

RANDOM INTERCEPT MODEL

TIME

RE

SP

ON

SE

6 7 8 9 10

80

10

01

20

14

0

b04

eps_43

RANDOM INTERCEPT + SLOPE MODEL

TIME

RE

SP

ON

SE

6 7 8 9 10

10

01

50

20

02

50

39

3. Mixed effects models

• Assumption in mixed effects models

4040

3. Mixed effects models3. Mixed effects models • Not enough used YET in medical research

• Recommended for contemporary rheumatology studies

– Allows unbalanced data

– None of the patients are deleted from the study

– Irregular time points are allowed

– No explicit imputation of responses

– Time evolution, effect of covariates and correlation structure: all can be flexible and modelled

• BUT

– One needs statisticians for the job

– Luckily: they are OFTEN cheap

4141

3. Mixed effects models3. Mixed effects models

• Linear mixed effects models

– Response = continuous

– Normality assumptions

• Generalized linear mixed effects models

– Response = continuous or discrete

– Normality assumptions

4242

3. Mixed effects models3. Mixed effects modelsJIA study (ESR) (ESR)

• On ALL cases:

– CC-Homozygous – T-carrier=

• Time effect: P < 0.001

• Genotype effect: P=0.03

• Time*genotype effect: P=0.09

• + ESTIMATES and 95% CIs

43

3. Mixed effects models3. Mixed effects modelsJIA study (ESR) (ESR)

44

4. GEE models4. GEE models

• Can cope only with MCAR

• Makes only few assumptions about distribution of data

• Most popular for discrete responses

• Can also be used for continuous response

45

4. GEE models4. GEE modelsJIA study (ACR30)

• Binary response

• No missing data in the study

• Possible analyses

– Chi-square tests (Fisher’s exact tests) at each time point

– Generalized mixed effects model

– GEE model

• Same comments as before

• Here all analyses gave non-significant difference between genotype groups

46

Literature review

4747

Literature Literature reviewreview • Literature search (criteria) :

– PubMed

– January 1- December 31, 2008

– Annals of the Rheumatic Diseases & Arthritis and Rheumatism

– Patients with arthritis were followed up in time over > 3 measurements

– Response that could vary over time

• Information collected:– Research aim– Primary outcome, secondary outcomes– Used technique for handling with missing data – Statistical analysis techniques used

• Eligible studies were ranked according to degree they correctly analyzed the longitudinal research question

4848

Literature Literature reviewreview Study characteristics

• 203 longitudinal studies in ARD or in AR

• 156 out of these studies dealt with arthritis– 11 studies with JIA patients– 58 studies described a RCT– 98 studies described a longitudinal cohort study

• 110 studies were included (excluded: 30 studies not longitudinal, 1 study was not first published in 2008, 16 studies did not include patients with arthritis)

Statistical techniques used

• 110 studies were ranked

• 17 made optimal use of modern statistical methods

49

Literature Literature reviewreview

RankingDescription Number of

studies

Suitable for repeated measurement analysis

110

1 Repeated character of the data not taken into account

78

2 Use of Bonferroni correction for multiple testing 7

2 Use of Repeated Measurements ANOVA or MANOVA

8

3 Appropriate repeated measurement techniques used on all suitable research questions

17

50

Conclusions

51

Conclusions

1. Many of the longitudinal studies in rheumatology are not using an appropriate statistical methodology.

2. Repeated significance tests and summary statistics methods are inappropriate in practice and can not show the evolution of an outcome over time.

3. Classical repeated measurement methods are inappropriate in practice.

4. The mixed effects model and the GEE approach can make use of all cases independently whether they have data on all time points studied.

5. The mixed effects model and the GEE approach are sparsely used in longitudinal studies rheumatology.

52

Thank you for your attention