Part 6: Review of Recent Practices · with instructional slide sets, readings, and guided...

David M. Murray, Ph.D.Associate Director for PreventionDirector, Office of Disease PreventionNational Institutes of Health

A free, 7-part, self-paced, online course from NIH with instructional slide sets, readings, and guided activities

Pragmatic and Group-Randomized Trials in Public Health and MedicinePart 6: Review of Recent Practices

Target Audience

Faculty, post-doctoral fellows, and graduate students interested in learning more about the design and analysis of group-randomized trials.Program directors, program officers, and scientific review

officers at the NIH interested in learning more about the design and analysis of group-randomized trials.Participants should be familiar with the design and analysis of

individually randomized trials (RCTs).Participants should be familiar with the concepts of internal and

statistical validity, their threats, and their defenses.Participants should be familiar with linear regression, analysis of

variance and covariance, and logistic regression.

135Pragmatic and Group-Randomized Trials Part 6: Review of Recent Practices

Learning Objectives

And the end of the course, participants will be able toDiscuss the distinguishing features of group-randomized trials

(GRTs), individually randomized group-treatment trials (IRGTs), and individually randomized trials (RCTs).Discuss their appropriate uses in public health and medicine.For GRTs and IRGTsDiscuss the major threats to internal validity and their defenses.Discuss the major threats to statistical validity and their defenses.Discuss the strengths and weaknesses of design alternatives.Discuss the strengths and weaknesses of analytic alternatives. Perform sample size calculations for a simple GRT.Discuss the advantages and disadvantages of alternatives to

GRTs for the evaluation of multi-level interventions.


Organization of the Course

Part 1: Introduction and Overview Part 2: Designing the Trial Part 3: Analysis Approaches Part 4: Power and Sample Size Part 5: Examples Part 6: Review of Recent PracticesPart 7: Alternative Designs and References


A Review of Recent Practices in GRTs

Murray DM, Pals SP, Blitstein JL, Alfano CM, Lehman J. Design and analysis of group-randomized trials in cancer: a review of current practices. Journal of the National Cancer Institute. 2008;100(7):483-91.

Pragmatic and Group-Randomized Trials Part 6: Review of Recent Practices 138

A Review of Recent Practices in GRTsPrevious Reviews of the GRT Literature

The first review was published by Donner et al. in 1990.Only 19% took the ICC into account in the sample size

calculations.Only 50% took the ICC into account in the analysis.

A review by Simpson et al. in 1995 reported little progress.Only 19% took the ICC into account in the sample size

calculations.Only 57% took the ICC into account in the analysis.

Donner A, Brown KS, Brasher P. A methodologic review of non-therapeutic intervention trials employing cluster randomization, 1979-1989. International Journal of Epidemiology. 1990;19(4):795-800.

Simpson JM, Klar N, Donner A. Accounting for cluster randomization: a review of Primary Prevention Trials, 1990 through 1993. American Journal of Public Health. 1995;85(10):1378-83.


A Review of Recent Practices in GRTsPrevious Reviews of the GRT Literature

A review by Varnell et al. in 2004 reported no progress, though the standards were higher than in previous reviews.Only 15% took the ICC into account in the sample size

calculations.Only 54% always took the ICC into account in the analysis.

We were interested in whether the situation had improved.

Varnell SP, Murray DM, Janega JB, Blitstein JL. Design and analysis of group-randomized trials: a review of recent practices. American Journal of Public Health. 2004;94(3):393-9. PMC1448264.


A Review of Recent Practices in GRTsProcedures

Systematic review of cancer-related studies published 2002-2006.Medline and PubMed search.Studies had as their primary outcome cancer risk factors, cancer

morbidity, or cancer mortality.Studies used randomization to assign identifiable social groups to

study conditions, with observations taken on members of those groups to assess the impact of an intervention.Where the paper referred to an earlier "design paper", we also

reviewed that paper.Each reviewer independently assessed the article on items

related to design, sample size estimation, and analysis.The reviewers discussed each paper as a group and any

disagreements were resolved in discussion.


A Review of Recent Practices in GRTsFindings

92 possible group-randomized trials in 45 journals.75 articles from 41 journals that met the inclusion criteria.20 background "design" papers.20% in the Preventive Medicine7% in American Journal of Public HealthNo more than 4% in any other single journal15.0 GRT papers per year (2002-06) vs 11.6 per year (1998-

2002) in Varnell et al. and 5.3 per year (1990-93) in Simpson et al.



Table 1. Analytic methods frequently used in group-randomized trials and the conditions under which their use is appropriate

Method Appropriate application in group-randomized trials

Mixed-model methods

Repeated measures

ANOVA/ANCOVA*

Random coefficient approach

Generalized estimating equations

With correction for limited df

With no correction

Outcomes for which there are one or two time points, variation in means or rates at the condition level is assessed against variation in means or rates at the group level, and df are based on the number of groups

Outcomes for which there are three or more time points, variation in slopes and intercepts at the condition level is assessed against variation in slopes and intercepts at the group level, and df are based on the number of groups

Trials where there are fewer than 40 df available for the test of the intervention effect

Trials where there are 40 or more df available for the test of the intervention effect

* ANOVA = analysis of variance; ANCOVA = analysis of covariance.


Table 1. Analytic methods frequently used in group-randomized trials and the conditions under which their use is appropriate

Method Appropriate application in group-randomized trials

Two-stage methods

Post hoc correction based on external estimates of intraclass correlation

Analysis at individual level, ignoring group-level intraclass correlation

Analysis at subgroup level, ignoring group-level intraclass correlation

Calculation of group means or other summary statistic in the first stage and analysis of variation in those summary statistics at the condition level in the second stage, df based on the number of groups

To data for which there are valid external estimates of intraclass correlation

None for group-randomized trials

None for group-randomized trials

* ANOVA = analysis of variance; ANCOVA = analysis of covariance.


Table 3. Characteristics of 75 articles that reported the results of group-randomized trials from cancer research in selected peer-reviewed journals during the period 2002 2006, inclusive *

CharacteristicNo.%

No. of study conditions

26688

356.7

445.3

Design

Cohort5067

Cross-sectional1520

Combination of cohort and cross-sectional 1013

Matching or stratification in design

Matching only1520

Stratification only2635

Matching and stratification4 5

Randomization without matching or stratification 3040

* Percentages within subsections of may not add to the subsection total because the categories were not always mutually exclusive.



CharacteristicNo.%

Type of group

Churches45.3

Communities, neighborhoods, or community groups 1419

Housing projects or apartment buildings34.0

Physicians or provider groups2837

Schools or colleges1520

Worksites1115

No. of groups per condition

145.3

2 52128

6 81013

9 12912

13 251723

>251216

Not reported22.7




CharacteristicNo.%

No. of members per group

1001723

>1 type of member56.7

Not reported22.7

No. of time points

14154

22229

3 945.4

No. of time points varies within study79.3

Unknown or continuous11.3




CharacteristicNo.%

Focus of study

Primary prevention3952

Secondary prevention2432

Tertiary prevention56.7

Combination56.7

None of the above22.7

Target population

Individuals with no personal history of the target cancer 1419

Cancer survivors during primary treatment34.0

Cancer survivors after primary treatment11.3

Unknown or mixed cancer survivorship4965

Caregivers34.0

Combination56.7




CharacteristicNo.%

Primary outcome variables

Alcohol use22.7

Delivery of health services79.3

Dietary variables1115

Incidence of cancer22.7

Knowledge of cancer or attitudes regarding cancer 912

Mortality from cancer22.7

Physical activity79.3

Screening2432

Sun protection68.0

Tobacco use1520

Other34.9



Table 4 . Distribution of analytic methods in 75 articles that reported the results of group-randomized trials from cancer research published in selected peer-reviewed journals during the period 2002 2006 inclusive *

Criteria No.%

Articles reporting only appropriate methods 3445

Mixed-model methods with adjustment for baseline or other 1520

covariates

Mixed-model repeated measures with two time points 912

Random coefficient model with more than two time points 11.3

Generalized estimating equations with 40 groups 811

Two-stage analysis (analysis of group means or other summary 45.3

statistics)

* Percentages within subsections of may not add to the subsection total



Criteria No.%

Articles reporting some appropriate and some inappropriate methods 68.0

Appropriate methods

Mixed-model methods with adjustment for baseline or other 45.3

covariates

Mixed-model repeated measures with two time points 11.3

Random coefficient model with more than two time points 00.0

Generalized estimating equations with 40 groups 11.3

Two-stage analysis (analysis of group means or other 00.0

summary statistics)

* Percentages within subsections of may not add to the subsection total



Criteria No.%

Articles reporting some appropriate and some inappropriate methods 68.0

Inappropriate methods

Analysis at an individual level, ignoring group-level 56.7

intraclass correlation

Analysis at a subgroup level, ignoring group-level 00.0


Generalized estimating equations or other asymptotically 00.0

robust method with



Criteria No.%

Articles reporting only inappropriate methods 2635

Analysis at an individual level, ignoring group-level 1723


Analysis at a subgroup level, ignoring group-level 45.3


Analysis with group as a fixed effect 00.0

Mixed-model repeated measures with more than two time points 22.7

Generalized estimating equations with

A Review of Recent Practices in IRGTs

Pals SP, Murray DM, Alfano CM, Shadish WR, Hannan PJ, Baker WL. Individually randomized group treatment trials: a critical appraisal of frequently used design and analytic approaches. American Journal of Public Health. 2008;98(8):1418-24. PMC2446464


A Review of Recent Practices in IRGTsProcedures

There were no prior systematic reviews of IRGT trials.We manually searched six journals for the period 2002-06.American Journal of Public HealthPreventive MedicineHealth PsychologyObesity ResearchAddictive BehaviorsAIDS and Behavior

Procedures parallel to those used for the GRT reviewCriteria for sample size and analysis methods parallel to

those used for the GRT review34 eligible articles



Sheet1

TABLE 2- Characteristics of the Studies Described in 34 Articles Reviewed 2002-2006

Study characteristicsNumber of articles%

Journal

American Journal of Public Health411.8

Preventive Medicine617.6

Health Psychology823.5

Obesity720.6

Addictive Behaviors720.6

AIDS and Behavior25.9

Year of publication

2002514.7

2003617.6

2004617.6

2005514.7

20061235.3

Number of study conditionsa

Two2367.6

Three823.5

Four38.8

Number of group treatment conditionsb

One1132.3

Two1750

Three411.8

Four25.9

Baseline sample size


Sheet1




Two2367.6

Three823.5

Four38.8


One1132.3

Two1750

Three411.8

Four25.9



Sheet1



Target population

Adults or adolescents with mental health issues38.8

Overweight or obese children25.9

Overweight or obese adults926.5

Adults with cardiovascular risk factors other than weight38.8

Cancer patients25.9

College or University students25.9

HIV-positive adults38.8

Smokers or substance abusers720.6

Other38.8

Primary Outcome Variablec

Weight, BMI, Body Fat percentage or Dietary Variables1338.2

Physical activity/ physical fitness variables514.7

Smoking or substance use variables720.6

Mental health variables617.6

Sex behavior variables617.6

Treatment retention25.9

Medication adherence25.9

Other variables720.6

Journal




Obesity720.6



Year of publication

2002514.7

2003617.6

2004617.6

2005514.7

20061235.3


Two2367.6

Three823.5

Four38.8


One1132.3

Two1750

Three411.8

Four25.9



Sheet1



Primary Outcome Variablec

Weight, BMI, Body Fat percentage or Dietary Variables1338.2

Physical activity/ physical fitness variables514.7

Smoking or substance use variables720.6

Mental health variables617.6

Sex behavior variables617.6

Treatment retention25.9

Medication adherence25.9

Other variables720.6

Journal




Obesity720.6



Year of publication

2002514.7

2003617.6

2004617.6

2005514.7

20061235.3


Two2367.6

Three823.5

Four38.8


One1132.3

Two1750

Three411.8

Four25.9



Sheet1

TABLE 3- Results of the Review of Sample Size Calculations and Analytic Methods in

34 Articles Reviewed, 2002-2006


Sample size calculations

Authors reported sample size calculations at individual level617.6

Authors stated power calculations performed, but no detail12.9

No mention of sample size calculation2573.5

Authors claimed sample size accounted for ICC, but no detail12.9

Other12.9

Any significant results reported

Yes2779.4

No720.6

Analytic approachesa

Analysis at an individual level, ignoring group entirely3294.1

Mixed-model approach with baseline as covariate25.9

Structural equation modeling12.9

Appropriateness of analytic methods

All analytic methods appropriate12.9

No analytic methods appropriate3294.1

Not enough information12.9

Sheet2

Sheet3


Sheet1

TABLE 3- Results of the Review of Sample Size Calculations and Analytic Methods in

34 Articles Reviewed, 2002-2006


Analytic approachesa

Analysis at an individual level, ignoring group entirely3294.1

Mixed-model approach with baseline as covariate25.9

Structural equation modeling12.9

Appropriateness of analytic methods

All analytic methods appropriate12.9

No analytic methods appropriate3294.1

Not enough information12.9

Sample size calculations

Authors reported sample size calculations at individual level617.6

Authors stated power calculations performed, but no detail12.9

No mention of sample size calculation2573.5

Authors claimed sample size accounted for ICC, but no detail12.9

Other12.9

Any significant results reported

Yes2779.4

No720.6

Sheet2

Sheet3

Summary

Our results for GRTs were similar to those in earlier reviews. 45% of the articles reported only analyses judged to be

appropriate. 35% reported only analyses deemed inappropriate. 8% reported a mix of appropriate and inappropriate analyses.

Crespi et al. (2011) published a more recent review, but limited to cancer screening studies. 1995-99 55% 2000-02 82% 2003-06 92% 2007-10 55%

Both suggest there is much room for improvement. Crespi CM, Maxwell AE, Wu S. Cluster randomized trials of cancer screening

interventions: are appropriate statistical methods being used? Contemporary Clinical Trials. 2011;32(4):477-84. PMC3104062.


Summary

Warnings have appeared in the literature for at least 30 years regarding the development of intraclass correlation in IRGTs.Even so, the literature on the design and analysis of IRGTs is

limited.The use of inappropriate design and analytic methods is

pervasive for IRGTs.The picture is similar to what GRTs looked like in the mid

1970s.Hopefully, the pattern will improve with time.


Pragmatic and Group-Randomized Trials in Public Health and Medicine

Visit https://prevention.nih.gov/grt to: Provide feedback on this series Download the slides, references, and suggested activities View this module again View the next module in this series:

Part 7: Alternative Designs

Send questions to:[email protected]

164

Pragmatic and Group-Randomized Trials in Public Health and MedicinePart 6: Review of Recent Practices Target AudienceLearning ObjectivesOrganization of the CourseA Review of Recent Practices in GRTsA Review of Recent Practices in GRTsPrevious Reviews of the GRT LiteratureA Review of Recent Practices in GRTsPrevious Reviews of the GRT LiteratureA Review of Recent Practices in GRTsProceduresA Review of Recent Practices in GRTsFindingsSlide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20A Review of Recent Practices in IRGTsA Review of Recent Practices in IRGTsProceduresSlide Number 23Slide Number 24Slide Number 25Slide Number 26Slide Number 27Slide Number 28SummarySummarySlide Number 31

Part 6: Review of Recent Practices · with instructional slide sets, readings, and guided...

Documents

Transcript of Part 6: Review of Recent Practices · with instructional slide sets, readings, and guided...