SPECIAL ARTICLE

Randomized and nonrandomized studies:Complementary or competing?

Nikolaos Pandis,a Yu-Kang Tu,b Padhraig S. Fleming,c and Argy Polychronopouloud

Bern, Switzerland, Taipei, Taiwan, London, United Kingdom, and Athens, Greece

aVisitipedicSwitzbAssocine,cSenioMedicdAssoSchooAll autentiaAddre49100Subm0889-Copyrhttp:/

Both randomized and nonrandomized studies are integral to orthodontic research and practice because theypermit evaluation of relationships between exposures and outcomes, allowing the efficacy, effectiveness, andsafety of interventions to be assessed. These designs allow clinical decisions to be informed. Nonrandomizeddesigns include nonrandomized clinical trials, cohort studies, case-control studies, cross-sectional studies, caseseries, and ecological studies. There is debate surrounding the optimal research design; however, both random-ized and nonrandomized designs are important to build a broad, informative evidence base. The designs aretherefore complementary, with unique advantages and limitations. The applicability of either approach hingeson the clinical question posed, the feasibility of studying it, and ethical considerations. (Am J OrthodDentofacial Orthop 2014;146:633-40)

Clinical research involves investigation of the causeof a disease, evaluation of associations betweencause and effect, and assessment of the preventive

or therapeutic value by isolating and recording potentialassociations. It is imperative that research findings areinterpreted and used optimally.1 Best conduct, use,and interpretation of research require an understandingof research methodology and study design comple-mented by adequate and transparent reporting.2,3

Clinical research can be broadly classified into non-randomized and randomized studies. The nonrandom-ized category, also known as observational studies,includes mainly nonexperimental studies: nonrandom-ized clinical trials, cohort studies, case-control studies,cross-sectional studies, case series, and ecological studies(Fig 1). Randomized studies are usually known in biomed-ical research as randomized controlled trials (RCTs). Thepurpose of randomization is the creation of groups that

ng assistant professor, Department of Orthodontics and Dentofacial Ortho-s, School of Dental Medicine, Medical Faculty, University of Bern, Bern,erland.ciate professor, Graduate Institute of Epidemiology and Preventive Medi-College of Public Health, National Taiwan University, Taipei, Taiwan.r clinical lecturer and honorary consultant, Barts and the London School ofine and Dentistry, Queen Mary University, London, United Kingdom.ciate professor, Department of Preventive and Community Dentistry,l of Dentistry, University of Athens, Athens, Greece.thors have completed and submitted the ICMJE Form for Disclosure of Po-l Conflicts of Interest, and none were reported.ss correspondence to: Nikolaos Pandis, 29 P. Zafiropoulou Str, Corfu, Greece; e-mail, npandis@yahoo.com.itted, June 2014; revised and accepted, August 2014.5406/$36.00ight � 2014 by the American Association of Orthodontists./dx.doi.org/10.1016/j.ajodo.2014.08.002

differ only randomly at the time of allocation of the inter-vention. Thus, the key difference between randomizedand nonrandomized studies is that in the former, theinvestigator allocates the interventions to participantsrandomly: eg, by throwing dice or coins, or by using com-puter software to generate an unpredictable sequence.4

There has been considerable debate as to whichdesign is more appropriate to answer clinically importantquestions.5-7 RCTs are considered the gold standard forassessing the efficacy and safety of the intervention ofinterest. However, it is not always possible or ethical toconduct an RCT. When RCTs are not feasible orunethical, nonrandomized studies, such as cohort orcase-control studies, may be undertaken. In general,nonrandomized studies are more prone to systematicand confounding biases than are RCTs; consequently,it is also more difficult to make causal inferences con-cerning the effect of an intervention.4,8

Observational studies are used extensively to describethe distribution of disease and exposure in populations.They are also useful for hypothesis generation andtesting; however, hypotheses may be assessed moreadequately, when feasible, with RCTs.8 On the otherhand, RCTs, although highly controlled and often lessbiased because they are conducted in highly selectedsettings, may yield less generalizable results that lackrelevance to other populations and settings. RCTs, there-fore, tend to have high internal validity but low externalvalidity.9,10 For example, if we consider the assessment ofClass II correction, Class IImalocclusion is amultifactorialproblem expressed with many variants includingmaxillary protrusion, mandibular retrognathia, or a

633

RCTs

Randomized Non-randomized

Clinical studies

Parallel, Factorial, Cross-over, Split-mouth, Non-inferiority, Cluster randomized, Adaptive

trials

Clinical trial (non-randomized, before-after, historic control)

Cohort

Case control

Case series

Cross-sectional

Ecological

Fig 1. Types of clinical studies based on Cochrane collaboration classification.4

Table I. Examples of randomized and nonrandomized (observational) clinical studies

Study type Treatment Outcome SourceRandomized

Randomized clinical trial Lingual retainer failuresChemical vs light cured bonding

Failure Pandis et al, 201312

Randomized clinical trial Comparison of the Twin-blockvs the Dynamax appliance

Class II correction Thiruvenkatachari et al, 201013

NonrandomizedCohort Self-ligating vs conventional brackets

Wire size and materialArch-wire ligation time Turnbull and Birnie, 200714

Case control Self-ligating vs conventional brackets Treatment efficiency Harradine, 200115

634 Pandis et al

combination. A successful intervention for Class IIcorrection demonstrated in an RCT with a highlyselected group of participants may not apply to thewider Class II population. Consequently, we can saythat the findings of RCTs are less generalizable.11 Exam-ples of published randomized and nonrandomized(observational) studies12-15 are shown in Table I.

NONRANDOMIZED STUDIES

For the purposes of this review, the discussion will belimited to cohort and case-control designs, since theseare most commonly used when an RCT is inappropriate.

Cohort studies

Cohort studies are also called follow-up, longitudinal,or incidence studies. The subjects are followed over timeto monitor their health outcomes. Participants withdifferent levels of exposures to risk factors or differentcharacteristics at baseline are then compared to estimatedifferences in the rate of developing certain health out-comes later in life. These groups are defined as study

November 2014 � Vol 146 � Issue 5 American

cohorts. All participants must be at risk of developingthe outcome. The participants are followed for a setperiod of observation (usually a long period), and allnew cases of the outcome of interest are identified. Com-parisons of outcome experiences are made within thestudy cohorts. Ideally, the exposure factor would consti-tute the only difference between the populations undercomparison, although in reality people with differentlevels of exposures also differ in other characteristics.16

Cohort studies that are based on information con-cerning the exposure and the outcome collected frompreexisting sources in the past are called retrospectivecohort studies. This design is common in orthodonticswhen information on exposures and outcomes from pa-tient records is retrieved and associations explored.However, the usefulness of such a study depends onthe thoroughness of the certification of the outcomein the records for the time period under consideration.Moreover, information on confounding factors maynot be available because there was no planning for thestudy during completion of the files. Cohort designshave key advantages and disadvantages (Table II).16

Journal of Orthodontics and Dentofacial Orthopedics

Table II. Advantages and disadvantages of cohort, case-control, and randomized controlled designs

Type of study Advantages DisadvantagesCohort study Direct information on the sequence of events; therefore,

the temporal relationship between exposure anddisease can be more easily clarified

Prospective cohort studies can be expensive and time-consuming,whereas retrospective cohort studies require the availability ofadequate records

Multiple effects of an exposure can be examined Losses to follow-up, incomplete information on exposures andoutcomes, and confounding (retrospective designs)

Direct measurement of incidence rates or risks and theirdifferences and ratios is possible

Associations between exposures and rare outcomes require a largecohort and a long period of observation

Case-controlstudy

Less expensive and easier to conduct compared withcohort and randomized studies

Temporal relationship between exposure and disease may be difficultto establish

The method of choice for investigating rare diseases Assessment of rare exposures is usually difficult unless the study islarge or the exposure is common among those with the disease

Multiple risk factors can be studied simultaneously Incidence rates of disease in exposed and unexposed subjects cannotbe estimated in most instances.

Efficient in terms of sample size Prone to selection and recall biasesRandomized

controlled trialIntervention studies often provide the strongest support

for a cause-effect relationshipMany research questions cannot be tested in trials

Confounding is minimized, thus facilitating faircomparisons

Ethical, feasibility, and cost issues

External validity

Table III. Systematic bias encountered in nonrandom-ized studies

Systematic bias DescriptionSelection bias Comparison groups differ due to selection error in

terms of important baseline characteristics thatare important outcome predictors.

Information biasRecall bias Recollection of information on exposure is not

accurate and can be influenced by experiencingor not experiencing the outcome.

Detection bias Recording the exposure or the outcome isinfluenced by knowledge of the outcomeor the exposure, respectively.

Pandis et al 635

Case-control studies

This design aims to achieve the same goal as thecohort study more efficiently by using sampling. Simul-taneously, cases and controls free from that particularcondition are chosen as a representative sample of thepopulation. Ideally, the control group represents theexposure distribution in the source population that pro-duced the cases. Subsequently, information concerningexposure is collected for both the cases and the controls.Data are analyzed to determine whether exposure pat-terns differ between the cases and the controls.16

The selection of subjects is based on disease status.Sources of cases can include private practices, clinics,health registries, or screening programs. Controls canbe selected by using random sampling from the generalpopulation, practices, or clinics, or they might be rela-tives and friends of the cases. The specific advantagesand disadvantages of case-control studies are outlinedin Table II.16

Bias and confounding in nonrandomized studies

It is often difficult to establish a causal relationshipbetween the exposure and the outcome based on obser-vational designs because of potential biases and con-founding. Bias is the systematic error in the selectionof participants, the collection of data, and the ascertain-ment of exposure and outcome variables, leading toincorrect associations. Methods of reducing potentialbias should be carefully considered and planned at thestudy design stage, and the risk of bias should be ac-counted for when interpreting the results of suchstudies. Bias or systematic error differs from random

American Journal of Orthodontics and Dentofacial Orthoped

error, which is related to the variability in the sampledpopulation and is expressed by uncertainty (confidenceintervals) around the estimate. Increasing the samplesize increases the precision of the estimate by reducingthe random error but not the systematic error.17 Selec-tion and information bias are the 2 most prevalent typesof systematic bias encountered in observational studies(Table III).

Confounding bias caused by factors that causallyaffect the exposure and outcome (known as con-founders) can blur the associations between exposuresand outcomes (Fig 2).18 Let us assume that we are inter-ested in evaluating the association between oral hygienestatus and dental caries in 2 groups of young childrenwith good and less-good hygiene. If all other parameterswere equal, as expected in a randomized design, wecould be confident that any differences in dental cariesbetween treatment groups would be the result of differ-ences in the levels of hygiene. Now, imagine that another

ics November 2014 � Vol 146 � Issue 5

Fig 2. Diagrammatic presentation of confounding.

Table IV. Subcategories of randomized controlledtrials

Subcategory CharacteristicsSingle site Treatment provided by a single centerMultisite Treatment provided by multiple coordinated centersPreventive Prophylactic agent is provided to prevent outcome

occurrenceTherapeutic Therapy is provided to improve survival or a conditionParallel Everyone in a group receives the same treatment that

is different from the treatment given to other groupCross-over Each treatment is given at different times to each

subject. A variation is the split-mouth design.Factorial Each group gets more than 1 treatmentNoninferiority To establish noninferiority of a new therapeutic

modality that may be more cost-effective or easierto deliver

Clustered Interventions are assigned to clusters (schools,classrooms) and not to individuals

Adaptive Allows prespecifying prospective modification of at least1 aspect of the trial based on the accumulating data

636 Pandis et al

risk factor, dietary sugar, is associated with caries (theoutcome). Those subjects with substandard hygieneconsumed more sugar than those with good hygienebecause the former like to eat sweets. If we accept thatsugar consumption promotes caries and that the con-sumption of sugar is higher among those with substand-ard hygiene, it is likely that the effect of hygiene oncaries may be exaggerated because of another factorthat would be counteracting caries prevention in thosewith substandard hygiene. This factor is called aconfounder because it complicates the association be-tween exposure (brushing) and outcome (caries). Sugarconsumption is considered a potential confounder, sinceit is associated with the exposure—hygiene level—and isan independent predictor of the outcome, and it is notan effect of the exposure (the third condition for aparameter to be considered a potential confounder).

RANDOMIZED STUDIES

RCTs are investigations similar to cohort studies inwhich the researcher randomly assigns the exposure orintervention to the study participants. The aim of clinicaltrials is to investigate the effectiveness and safety of anintervention. RCTs are widely regarded as the optimalapproach in assessing a new treatment. In orthodontics,studies exploring the efficacy of a device could be gener-ally characterized as clinical trials. Clinical trialsinvolving skill-dependent interventions such as devices(device trials) resemble clinical trials for drugs (drug tri-als); however, devices have less need for extensive

November 2014 � Vol 146 � Issue 5 American

developmental testing compared with drugs in humans,but not less rigorous testing overall. All RCTs, like devicetrials, share common core design features such asexplicit inclusion and exclusion criteria, use of controls,randomization, masking where feasible, and theintention-to-treat principle. Some of those characteris-tics also apply to nonrandomized clinical studies.RCTs can be subcategorized in many ways, dependingon their characteristics and purposes (Table IV).9,10

The key advantages and limitations of randomizedstudies are outlined in Table II, and the differences

Journal of Orthodontics and Dentofacial Orthopedics

Table V. Differences between nonrandomized (observational) studies and randomized controlled trials

Characteristic Nonrandomized studies Randomized controlled trialsConfounding bias Randomization is not possible Randomization is possible

Known confounders can be controlled Known and unknown confounders can be controlledUnknown confounders are difficult to control

Outcome assessment Blinding is possible Blinding is possibleExposure No limit 1 or 2 interventionsParticipant selection Broad range of patients can be included Strict inclusion and exclusion criteria applyCost Comparatively low HighApplicability Pragmatic Less pragmatic

Effectiveness Efficacy/effectivenessHarms HarmsEtiologyPrognostic and diagnostic models

Pandis et al 637

between randomized and nonrandomized studies areoutlined in Table V.

DISCUSSION

The increasing emphasis on RCTs in assessing the ef-fect and safety of interventions started when theMedicalResearch Council in the United Kingdom completed thefirst modern randomized trial in 1948, which confirmedthe value of streptomycin in treating pulmonary tubercu-losis.19 This RCT heralded the shift from observation toexperimentation stemming from the desire to eliminatefactors that may confound associations between expo-sures or interventions and outcomes. The breakthroughcame with the implementation of randomization, whichallows “fair comparisons” by making groups similar forall known and unknown parameters with the exceptionof the intervention. Randomized studies require strictprotocols, and this has proven to be both a benefit anda constraint. The beneficial aspect relates to the fair com-parison, which is more likely to give unbiased results, andthe constraint is founded on the heightened demands interms of resources, time, and cost.

In randomized designs, a number of ethical questionsarise such as the following. Is it ethical to randomize? Isit ethical to assign a treatment other than the best to pa-tients? Is it possible to obtain informed consent? Underwhat circumstances can the trial be stopped early for ef-ficacy, harm, or futility? Issues related to individual vscollective benefits are contentious where random alloca-tion to a treatment is concerned.20 A trial is ethicallyjustified when there is equipoise: an indication ofgenuine uncertainty in the scientific community aboutthe effects of the therapy of interest.21

RCTs are often conducted by using selected groups ofpatients in accordance with strict protocols; thereby,their findings may have limited relevance to real-life sit-uations. In the “sterile” environment of an RCT, selectionof participants, allocation, and follow-up, althoughideal to assess efficacy (effect under ideal conditions),

American Journal of Orthodontics and Dentofacial Orthoped

can yield results applying only to a highly selected group.On the contrary, observational studies include popula-tions with diverse characteristics and thus can give indi-cations of how an intervention works in everyday clinicalpractice; they may therefore have greater external valid-ity.9,10 However, external validity can vary depending onthe intervention, the examined outcome, and theinfluence of the settings on the results. For example,studies assessing effective interventions such asorthodontic appliances for alignment in restrictedsettings are likely to be externally valid, regardless ofthe study design.

A conceptual approach to assessing the external val-idity of a trial could consider the following.22 Is the studypopulation different from the population to which wewish to apply the findings? Are the target populationcharacteristics likely to influence the results? Are the re-sults generalizable to the target population not meetingall eligibility criteria?

Issues concerning external validity can be overcomeby conducting larger studies with less stringent inclusionand exclusion criteria. There is a move toward more effi-cient and pragmatic studies without threats to validity,which can be implemented by applying simpler interven-tions and involving diverse settings and multiple opera-tors, thus mimicking real-life scenarios to enhancegeneralizability.23 Use of the existing architecture torun high-quality, pragmatic trials within the current de-livery pathway to answer important questions is likely tobe facilitated by the dental practice-based research net-works.24 This initiative aims to answer important ques-tions facing general dental practitioners by conductingrandomized and nonrandomized studies and by creatinga platform for data sharing.25

The Pragmatic Explanatory Continuum Index Sum-mary (PRECIS) was introduced with the objective of bet-ter understanding the influence of study design on theexternal validity of the findings.26 PRECIS is a “wheel”(Fig 3) with 10 domains corresponding to trial-design

ics November 2014 � Vol 146 � Issue 5

Practitioner expertise (experimental)

Flexibility of experimental intervention

Eligibility criteria

Primary analysis

Participant compliance

Practitioner compliance

Follow-up intensity

Outcomes

Flexibility of comparison interventions

Practitioner expertise(comparison)

E

Fig 3. The PRECIS tool wheel, adapted from Thorpe et al.22

638 Pandis et al

decisions. Explanatory trials (assessing efficacy) producewheels nearer the center, whereas pragmatic trials pro-duce wheels farther from the center.

The debate surrounding the merits of randomizedand observational studies has also been fueled by re-ported discrepancies in findings between those 2 de-signs.27-38 To our knowledge, no studies evaluating theagreement between nonrandomized and randomizeddesigns in orthodontics have been published. A well-known example of this discrepancy, however, concernsthe effectiveness of self-ligating appliances when theinitial observational data showed significant improve-ments in terms of efficiency that were later refuted inrandomized designs.15,39,40 In medicine, there wasgreat controversy surrounding the use of hormonereplacement therapy; observational studies showed aprotective effect for cardiovascular events, whereasrandomized studies alluded to a harmful effect. Recentreassessment and reanalysis of the data concluded thatthe results of the 2 designs on the effect of hormonereplacement therapy were less divergent.32-37 Whenlarge effects or strong associations exist betweenexposures and outcomes, observational studies canshow them, since the noise and confounding are notsufficiently strong to obscure the effect. In fact, whenthe “signal to noise ratio” is particularly strong, theremight be no need for a randomized study.41 For example,there is no need to conduct a study to prove that ortho-dontic brackets and nickel-titanium wires are capable ofalignment. However, when the differences in the effectsof interventions are small but clinically important, a“fair” design is more likely to detect them; this mightapply when we wish to test the efficacy of one orthodon-tic appliance against another in a randomized design.However, if differences are small, the effects may have

November 2014 � Vol 146 � Issue 5 American

limited clinical relevance. A further issue is that of imple-mentation fidelity, which refers to the extent of agree-ment between the intended intervention and thedelivered intervention. Implementation fidelity is animportant source of variation42 and may bias the studyresults, leading to conclusions of ineffectiveness.43,44

Outcomes such as rare adverse events are more likelyto be identified when large observational studies areused, since limited sample sizes and shorter follow-uptimes in RCTs may be insufficient to expose them. Forexample, the incidence of severe root resorption associ-ated with orthodontic treatment, a rare adverse event, isless likely to be accurately determined in a small ran-domized study. Another example is assessment ofadverse events after the placement of lingual retainers.45

Because retainers are intended to remain in place indef-initely, a large observational approach is appropriate.Vandenbroucke7 argued that confounding by indica-tion, a key problem inherent in nonrandomized studies,is not an issue when assessing harms, since the investi-gator is agnostic in terms of the prognosis of adverseevents and therefore less likely to apply biased assign-ments. In this scenario, he suggests that nonrandomizedand randomized designs may have a similar value.Clearly, therefore, the debate and the hierarchy shouldtranscend terminology and should focus on appraisalof threats to validity, irrespective of the general design.There are excellent observational studies that can bemore trustworthy than randomized studies, and viceversa. Nevertheless, biases are more easily controlled inrandomized designs. If known confounders exist,adjusted analyses may mitigate them; however, failureto adjust for them represents a serious threat to validity.In large randomized trials, simple analyses are often suf-ficient because of the “fair comparison” principle,

Journal of Orthodontics and Dentofacial Orthopedics

Pandis et al 639

whereas in observational studies, residual confoundingstemming from unknown and thus unadjusted-for con-founders can be problematic. In orthodontics, wherestudies are usually small, conducting observationalstudies without any consideration for potential con-founders in the analysis is likely to produce invalid re-sults unless the “signal to noise ratio” is large. Forlarge data sets, adjusted and more advanced statisticalmethods can be used, such as propensity scoring, inverseprobability of treatment weighting, and the G-computa-tion approach; although helpful, they are not apanacea.46 Moreover, if statistics could overcome alllimitations of observational studies, then RCTs wouldbe unnecessary; unfortunately, this is not the case.

Clearly, both observational and randomized studiescan make important contributions to evidence-based or-thodontics. This is exemplified by recent efforts by theCochrane collaboration to devise a new assessmenttool for nonrandomized studies that will be includedin Cochrane reviews.4 The proposed tool is based onthe preexisting risk of bias tool for randomized studieswith appropriate modifications. A key concept relatesto the judgment of how close the nonrandomized studyapproximates the “ideal” randomized design in terms ofvalidity.

The GRADE (Grading of Recommendations Assess-ment, Development and Evaluation) approach for theassessment of the evidence and formulation of clinicalrecommendations, endorsed by many societies and insti-tutions including the Cochrane collaboration, also incor-porates observational studies when grading theevidence.47 Randomized studies start at the highest levelof evidence and can be downgraded based on 5 criteria:study limitations (similar to the Cochrane risk of biastool), inconsistency, indirectness, imprecision, and pub-lication bias. Observational studies start from a lowerlevel but can be upgraded based on the size of the effect,dose response effect, and confounding.48

It has been suggested that study designs are not hi-erarchical but flat, yielding similar results if the rightquestion is considered. Randomized and nonrandom-ized designs have advantages and limitations; both de-signs contribute to the evidence and should be viewednot as competing but rather as a continuum, withboth informing clinical decisions. This information canstimulate more valid, pragmatic, and efficient clinicalstudies and facilitate better, more informed clinicalchoices.

We believe that the following quote fittingly summa-rizes the synergy between observational and randomizedstudies: “Experiment, observation, and mathematics,individually and collectively, have a crucial role inproviding the evidential basis for modern therapeutics.

American Journal of Orthodontics and Dentofacial Orthoped

Arguments about the relative importance of each arean unnecessary distraction. Hierarchies of evidenceshould be replaced by accepting—indeed embracing—adiversity of approaches. This is not a plea to abandonRCTs and replace them with observational studies.Rather, it is a plea to investigators to continue to developand improve their methods; to decision makers, to avoidadopting entrenched positions about the nature of evi-dence; and for both, to accept that the interpretationof evidence requires judgment.”49

CONCLUSIONS

Both nonrandomized and randomized studies arevaluable in informing clinical practice. Randomizedstudies allow us to determine whether an interventioncan have the intended effect, and large observationalstudies can help to pinpoint effects in everyday practiceby considering efficacy, safety, and patient compliance.Distinct differences between the scope of nonrandom-ized and randomized studies ensure that these designsare complementary rather than competing.

ACKNOWLEDGMENTS

The authors would like to dedicate this article to thememory of the late Vincent G. Kokich, former editor ofAJO-DO, who commissioned this paper.

REFERENCES

1. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S,et al. Reducing waste from incomplete or unusable reports ofbiomedical research. Lancet 2014;383:267-76.

2. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC,Devereaux PJ, et al. CONSORT 2010 explanation and elaboration:updated guidelines for reporting parallel group randomised trials.BMJ 2010;340:c869.

3. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC,Vandenbroucke JP, STROBE Initiative. The strengthening the re-porting of observational studies in epidemiology (STROBE) state-ment: guidelines for reporting observational studies. J ClinEpidemiol 2008;61:344-9.

4. Reeves BC, Deeks JJ, Higgins JPT, Wells GA. Including non-randomized studies. In: Higgins JPT, Green S, editors. Cochranehandbook for systematic reviews of interventions (version 5.1.0,updated March 2011). The Cochrane Collaboration; 2011. Avail-able at: www.cochrane-handbook.org. Accessed March 25, 2014.

5. Vandenbroucke JP. Why do the results of randomised and obser-vational studies differ? BMJ 2011;343:d7020.

6. Vandenbroucke JP. The HRT controversy: observational studiesand RCTs fall in line. Lancet 2009;373:1233-5.

7. Vandenbroucke JP. When are observational studies as credible asrandomised trials? Lancet 2004;363:1728-31.

8. Hennekens CH, Buring JE. Epidemiology in medicine. 1st ed. Bos-ton: Little, Brown; 1987. p. 101-212.

9. Rothwell PM. Treating individuals 1—external validity of rando-mised controlled trials: “to whom do the results of this trialapply?”. Lancet 2005;365:82-93.

ics November 2014 � Vol 146 � Issue 5

640 Pandis et al

10. Treweek S, Zwarenstein M. Making trials matter: pragmatic andexplanatory trials and the problem of applicability. Trials 2009;10:37.

11. Wiltshire W. Class II growth modification in perspective. Semin Or-thod 2006;12:2-3.

12. Pandis N, Fleming PS, Kloukos D, Polychronopoulou A, Katsaros C,Eliades T. Survival of bonded lingual retainers with chemical orphoto polymerization over a 2-year period: a single-center, ran-domized controlled clinical trial. Am J Orthod Dentofacial Orthop2013;144:169-75.

13. Thiruvenkatachari B, Sandler J,Murray A,Walsh T, O'Brien K. Com-parison of Twin-block and Dynamax appliances for the treatmentof Class II malocclusion in adolescents: a randomized controlledtrial. Am J Orthod Dentofacial Orthop 2010;138:144.e1-9.

14. Turnbull NR, Birnie DJ. Treatment efficiency of conventional vsself-ligating brackets: effects of archwire, size, and material. AmJ Orthod Dentofacial Orthop 2007;131:395-9.

15. Harradine NW. Self-ligating brackets and treatment efficiency.Clin Orthod Res 2001;4:220-7.

16. Rothman KJ. Epidemiology: an introduction. New York: OxfordUniversity Press; 2002. p. 57-93.

17. Gordis L. Epidemiology. 4th ed. Philadelphia: Saunders Elsevier;2009. p. 247-63.

18. Koepsell TD, Weiss NS. Confounding and its control. Epidemiolog-ical methods: studying the occurrence of illness. New York: OxfordUniversity Press; 2003. p. 247-80.

19. Medical Research Council. Streptomycin treatment of pulmonarytuberculosis: a Medical Research Council investigation. BMJ1948;2:769-82.

20. Heilig CM, Weijer C. A critical history of individual and collectiveethics in the lineage of Lellouch and Schwartz. Clin Trials 2005;2:244-53.

21. Freedman B. Equipoise and the ethics of clinical research. N Engl JMed 1987;317:141-5.

22. Dekkers OM, von Elm E, Algra A, Romijn JA, Vandenbroucke JP.How to assess the external validity of therapeutic trials: a concep-tual approach. Int J Epidemiol 2010;39:89-94.

23. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasingthe value of clinical research for decision making in clinical andhealth policy. JAMA 2003;290:1624-32.

24. Luce BR, Kramer JM, Goodman SN, Connor JT, Tunis S, Whicher D,et al. Rethinking randomized clinical trials for comparative effec-tiveness research: the need for transformational change. AnnIntern Med 2009;151:206-9.

25. The Northwest Practice-based Research Collaborative in Evidence-based Dentistry (NW PRECEDENT). Available at: https://workbench.axioresearch.com/NWPrecedent/index.htm. AccessedApril 26, 2014.

26. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD,Altman DG, et al. A pragmatic-explanatory continuum indicatorsummary (PRECIS): a tool to help trial designers. J Clin Epidemiol2009;62:464-75.

27. Sacks H, Chalmers TC, Smith H Jr. Randomized versus historicalcontrols for clinical trials. Am J Med 1982;72:233-40.

28. Concato J, Shah N, Horwitz RI. Randomized, controlled trials,observational studies, and the hierarchy of research designs. NEngl J Med 2000;342:1887-92.

29. Benson K, Hartz AJ. A comparison of observational studies andrandomized, controlled trials. N Engl J Med 2000;342:1878-86.

30. Pocock SJ, Elbourne DR. Randomized trials or observational trib-ulations? N Engl J Med 2000;342:1907-9.

31. Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI,TektonidouMG, et al. Comparison of evidence of treatment effects

November 2014 � Vol 146 � Issue 5 American

in randomized and nonrandomized studies. JAMA 2001;286:821-30.

32. Prentice RL, Langer R, Stefanick ML, Howard BV, Pettinger M,Anderson G, et al., for the Women's Health Initiative Investigators.Combined postmenopausal hormone therapy and cardiovasculardisease: toward resolving the discrepancy between observationalstudies and the Women's Health Initiative clinical trial. Am J Epi-demiol 2005;162:404-14.

33. Reeves GK, Beral V, Green J, Gathani T, Bull D, for the MillionWomen Study Collaborators. Hormonal therapy for menopauseand breast-cancer risk by histological type: a cohort study andmeta-analysis. Lancet Oncol 2006;7:910-8.

34. Hern�an MA, Alonso A, Logan R, Grodstein F, Michels KB,Willett WC, et al. Observational studies analyzed like randomizedexperiments: an application to postmenopausal hormone therapyand coronary heart disease. Epidemiology 2008;19:766-79.

35. Prentice RL, Chlebowski RT, Stefanick ML, Manson JE,Pettinger M, Hendrix SL, et al. Estrogen plus progestin therapyand breast cancer in recently postmenopausal women. Am J Epi-demiol 2008;167:1207-16.

36. Prentice RL, Chlebowski RT, Stefanick ML, Manson JE, Langer RD,Pettinger M, et al. Conjugated equine estrogens and breast cancerrisk in the Women's Health Initiative clinical trial and observationalstudy. Am J Epidemiol 2008;167:1407-15.

37. Prentice RL, Langer RD, Stefanick ML, Howard BV, Pettinger M,Anderson GL, et al., for the Women's Health Initiative Investiga-tors. Combined analysis ofWomen's Health Initiative observationaland clinical trial data on postmenopausal hormone treatment andcardiovascular disease. Am J Epidemiol 2006;163:589-99.

38. Danaei G, Tavakkoli M, Hern�an MA. Bias in observational studiesof prevalent users: lessons for comparative effectiveness researchfrom ameta-analysis of statins. Am J Epidemiol 2012;175:250-62.

39. Eberting JJ, Straja SR, Tuncay OC. Treatment time, outcome, andpatient satisfaction comparisons of Damon and conventionalbrackets. Clin Orthod Res 2001;4:228-34.

40. Pandis N, Fleming PS, Spineli LM, Salanti G. Initial orthodonticalignment effectiveness with self-ligating and conventional appli-ances: a network meta-analysis in practice. Am J Orthod Dentofa-cial Orthop 2014;145(Suppl):S152-63.

41. GlasziouP,Chalmers I, RawlinsM,McCullochP.Whenare randomisedtrials unnecessary? Picking signal fromnoise. BMJ 2007;334:349-51.

42. CarrollC, PattersonM,WoodS,BoothA,Rick J,BalainS. A conceptualframework for implementation fidelity. Implement Sci 2007;2:40.

43. Thomas RE, Baker P, Lorenzetti D. Family-based programmes forpreventing smoking by children and adolescents. Cochrane Data-base Syst Rev 2007;CD004493.

44. Noel P. The impact of therapeutic case management on participa-tion in adolescent substance abuse treatment. Am J Drug AlcoholAbuse 2006;32:311-27.

45. Pazera P, Fudalej P, Katsaros C. Severe complication of a bondedmandibular lingual retainer. Am J Orthod Dentofacial Orthop2012;142:406-9.

46. Castillo RC, Scharfstein DO, MacKenzie EJ. Observational studiesin the era of randomized trials: finding the balance. J Bone JointSurg Am 2012;94(Suppl 1):112-7.

47. Guyatt GH, Oxman AD, Sultan S, Glasziou P, Akl EA, Alonso-Coello P, et al., GRADE Working Group. GRADE guidelines: 9. Rat-ing up the quality of evidence. J Clin Epidemiol 2011;64:1311-6.

48. Balshem H, Helfand M, Sch€unemann HJ, Oxman AD, Kunz R,Brozek J, et al. GRADE guidelines: 3. Rating the quality of evi-dence. J Clin Epidemiol 2011;64:401-6.

49. Rawlins M. De testimonio: on the evidence for decisions about theuse of therapeutic interventions. Lancet 2008;372:2152-61.

Journal of Orthodontics and Dentofacial Orthopedics