Study Eligibility CriteriaPrepared for:

The Agency for Healthcare Research and Quality (AHRQ)

Training Modules for Systematic Reviews Methods Guide

www.ahrq.gov

Systematic Review Process Overview

To understand the role of study eligibility criteria (inclusion/exclusion) in framing a systematic review

To know when and how to set study eligibility criteria

To understand the effect of study eligibility criteria on interpretation of a review

Learning Objectives

Determine which studies will be included in the analysis

Function the same in systematic reviews as in primary research

Should reflect the analytic framework and key questions

Are powerful tools for widening or narrowing the scope of a review

Provide information to determine whether reviews can be compared or combined

Study Eligibility Criteria

Population type (adult, community-dwelling females)

Intervention (screening tool to identify depression risk)

Country (United States only) Setting (hospital-based studies only) Study population size (N > 200) Study design (randomized controlled trials)

Some Sample Criteria

Can be as broad as “all related studies” Can be helpful for exploring “what is known” May result in too much literature to feasibly

review Could pull in disparate literature that cannot

be compared

Using Broad Criteria

Can be helpful in culling homogenous literature

Can reduce size of the literature to a manageable scope

Could be appropriate for a narrow research question

May reduce applicability of the review and sometimes risks introducing bias

Using Narrow Criteria

Choosing how broad or how narrow to set criteria requires a balance of obtaining adequate information to answer a given question without obscuring the results with irrelevant literature.

Review teams should work together to find this balance.

The overarching goal is to minimize bias related to which studies are selected.

Refining Criteria

Distortion of the estimate of effect that comes from how studies are selected for inclusion

Affects the applicability, or “external validity,” of the review itself

Bias in This Context

Inappropriate eligibilty criteria may limit applicability of the review. Example: the use of studies of twin pregnancies in a review

of preterm labor management for low-risk women

Eligibility criteria may result in the inclusion of more of a certain study type that either overestimates or underestimates effectiveness.

Examples of Bias in This Context

Review study goals Assess analytic framework and key

questions Set criteria before beginning abstract review

Selecting Criteria

Population — condition, disease severity and stage, comorbidities, patient demographics

Intervention — dosage, frequency, method of administration

Comparator — placebo, usual care, or active control

Outcome — health outcomes, morbidity, mortality, quality of life

Timing — Duration of followup Setting — Primary, specialty, inpatient,

cointerventions

Tie Criteria to PICOTS

What study designs should be included? Include studies in foreign languages? Include studies conducted in other countries

but published in English? Include “grey” or “fugitive” literature? Include year of publication?

Other Considerations for Setting Criteria

Limit to randomized controlled trials? Include observational studies?

If so, what specific types?

What is the value of a case series? How is the definition of case series

operationalized?

Types of Studies To Be Included

There are four main types of observational studies: Cohorts (with comparisons) Case controls Case series Registries/databases

Observational Studies (I)

Well-conducted observational studies can address issues of applicability and the need for longer-term outcomes if they: include more representative patient populations, have relevant comparators, and report more meaningful clinical outcomes over longer time

frames.

Observational studies may be a better source of information about harms.

Observational Studies (II)

Positive findings may be more likely to be published in high-profile English-language journals.

Therefore, to include only English-language journals may overestimate the positive effect of an intervention.

Empirically, the bias associated with limiting a review to English-language reports has been shown to be small.

Non-English–Language Study Reports

Gregoire G, et al. J Clin Epidemiol 1995;48:159-63; Moher D, et al. J Clin Epidemiol 2000;53:964-72.

Material that may be difficult to identify and retrieve: Government reports and regulatory sites (Center for Drug Evaluation and

Research/U.S. Food and Drug Administration)

Pharmacoepidemiologic databases, including postmarketing surveillance

Book chapters

Conference proceedings

Published dissertations

Scientific information packets (Agency for Healthcare Research and Quality)

Rationale for a search: Because journals may publish positive or statistically significant results,

finding grey literature of unpublished nonsignificant or null results may indicate the presence of publication bias.

Grey or “Fugitive “Literature

Literature searches typically have date parameters (i.e., earliest or latest date of publication).

Date parameters are particularly important when there has been a change in policy, practice, or formulation that makes older studies less applicable.

Year of Publication

The rest of this module contains a number of questions and some sample criteria.

Using your newly acquired knowledge of selection criteria and the guidelines described in the previous slides, work through the thought exercises and examples that follow.

Using the Criteria

What would you do if you were asked to review the literature on transition support for adolescents with autism spectrum disorder who are entering adulthood? Before seeing the key questions, consider the categories of criteria that you will want to apply.

Exercise 1

What is the relevant population? What is the intervention of interest? To what exposure is the intervention being

compared? What outcomes are relevant? What short-term or long-term outcomes

should be considered? In what setting would the results be

applicable?

Exercise 1: Basic PICOTS Questions

PICOTS = population, intervention, comparators, outcomes, timing, and setting

Population Population What constitutes an adolescent? What constitutes a diagnosis of autism spectrum disorder (ASD)?

InterventiInterventionon

How is transition support defined?

ComparatoComparator r

Do we compare to no transition support or directly compare types of support?

OutcomeOutcomeWhat are the goals for adolescents with ASD as they transition to adulthood?

Timing Timing How quickly should the outcomes be apparent?

SettingSettingIs transition support provided in multiple settings, such as schools, clinics, and the community?

Exercise 1: PICOTS

A study that included “individuals over 10 years of age”?

A paper about an intervention for individuals with a range of developmental disabilities, not only autism spectrum disorder (ASD)?

A study of children with Asperger’s syndrome but not other forms of ASD?

Exercise 1: What Would You Do With . . .

There are no “right” answers to these questions.

The team should consider how selection criteria may bias the review or affect applicability.

The team should clearly define how eligibility criteria are operationalized and record these decisions throughout the review.

Exercise 1: No “Right” Answers

What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth?

Exercise 2: Selecting Criteria for a Narrow Research Question

Efficacy: randomized controlled trials only Low risk: no previous preterm birth Yield: 11 studies

Exercise 2: Implications of a Question That Requires Narrow Criteria

What is the best management strategy for overactive bladder (OAB) among women? consideration:

OAB is a fairly difficult condition to define. The study of OAB is often combined with other

types of urological conditions, such as stress incontinence or prostate issues, when studied.

OAB is treated pharmacologically, behaviorally, and surgically and with complementary and alternative interventions — each area includes very different types of studies.

Exercise 3:Selecting Criteria for a Broad Research Question

The research team required a minimum sample size N ≥ 50 in each study at study start.

Implications: Total excluded for N < 50: 79

Excluded for N < 20: 36 Excluded for N 20 to 29: 23 Excluded for N 30 to 39: 8 Excluded for N 40 to 49: 12

Exercise 3: Impact of Study-Size Criteria on Excluded Studies

Studies had to include at least 75 percent women. This decision was based on expert opinion and the size and

scope of the literature.

Based on this criterion, 40 studies were excluded.

Another 27 studies would have been excluded had the review been limited to studies of only women.

Exercise 3:Impact of Gender on Study Selection Criteria

Case series with N ≥ 2 were included. Only 3 of 169 studies were randomized

controlled trials; 122 were case series. Because of the relative newness of this area

of research, it was important to capture data even from studies without comparison groups.

Example 1: A Maternal-Fetal Surgery Technical Brief

A systematic review on outcomes of cesarean delivery on maternal request.

Conducted for the National Institutes of Health-Office of Medical Applications of Research (NIH-OMAR) State-of-the-Science Conference

Example 2:Systematic Review of Cesarean Delivery

No evidence on outcomes of cesarean delivery on maternal request (CDMR) versus other modes of delivery

Urgent need for actionable evidence Need to recognize and account for

confounders 

Example 2: The Challenge in the Systematic Review on Cesarean Delivery

Expand search to include proxies Weight rungs of evidence to account for

confounding Highest rung: trials of breech delivery, but only for maternal

outcomes Lower rung: planned cesarean versus planned vaginal

delivery Lowest rung: comparisons of maternal and neonatal

outcomes from actual modes of delivery

Example 2: The Solution in the Systematic Review on Cesarean Delivery

Eligibility criteria: Specify the scope of the literature to be included Should be tied to the analytic framework, key questions,

and PICOTS Can reduce bias and support applicability, but run the risk

of introducing bias if not chosen appropriately. Must be applied consistently among individuals on the

review team Need to be recorded throughout the project and in the

report for transparency and replicability

Key Messages

PICOTS = population, intervention, comparator, outcome, timing, and setting

Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309:1286-91.

Gregoire G, Derderain F, Le L. Selecting the language of the publication included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995;48:159-63.

McAuley L, Pham B, Tugwell P, et al. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000;356:1228-31.

McPheeters ML, Briss P, Teutsch SJ, et al. Systematic reviews in public health. In: Brownson RC and Petitti DB, eds. Applied epidemiology. New York: Oxford University Press; 2006. p. 99-124.

Moher D, Pham B, Klassen TP, et al. What contribution do languages other than English make on the results of meta-analyses? J Clin Epidemiol 2000; 53:964-72.

References

This presentation was prepared by Melissa L. McPheeters, Ph.D., M.P.H., a member of the Vanderbilt University Evidence-based Practice Center.

The information in this module is currently not included in Version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (available at: http://www.effectivehealthcare. ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf).

Author