12 Evidence Based Period Ontology Systematic Reviews

8/3/2019 12 Evidence Based Period Ontology Systematic Reviews

1/17

Evidence-based periodontology,systematic reviews and research

qualityIA N N E E D L E M A N, DAVI D R. M O L E S & HE L E N WO R T H I N G T O N

Periodontology has a rich background of research

and scholarship. A simple MEDLINE search of Peri-

odontal Diseases OR Periodontitis alone from 1966

to 2003 brings up more than 45,000 hits. Therefore,efficient use of this wealth of research data needs to

be a part of periodontal practice. Evidence-based

periodontology aims to facilitate such an approach,

accelerating the introduction of the best research into

patient care.

This chapter will review the concepts of evidence-

based periodontology, introduce the systematic

review as a research tool and examine how evidence-

based periodontology can both inform on and benefit

healthcare in periodontology. Finally, we will exam-

ine the strengths and limitations of different research

designs and their appraisal. We hope that the infor-

mation in this chapter will provide a basic under-

standing of the concepts that will be relevant to

reading and enjoying the other chapters in this vol-

ume of Periodontology 2000.

What is evidence-basedperiodontology?

Evidence-based periodontology is the application of

evidence-based health care to periodontology. Auseful definition of evidence-based health care has

been proposed by Muir Gray: An approach to

decision making in which the clinician uses the

best evidence available, in consultation with the

patient, to decide upon the option which suits that

patient best (8). Therefore, evidence-based period-

ontology is a tool to support decision making and

integrating the best evidence available with clinical

practice.

The highest quality evidence will be used if it exists,

but if it does not, lower levels of evidence will be

considered. Lower levels of evidence usually means

research designs more prone to bias and thereforewith less reliable data. However, the nature, strengths

and weaknesses of the evidence will be made clear to

the reader. In addition, wherever possible, the data

presentation supplies more clinically relevant infor-

mation, including the probability of achieving a cer-

tain effect such as a benefit, and considering possible

adverse effects.

What evidence-based

periodontology is notEvidence-based periodontology is not simply sys-

tematic reviews of randomized controlled trials,

although this can be an important aspect. Evidence-

based periodontology is an approach to patient-care

and nothing more. The expectations that are some-

times laid on it can be inappropriate. It cannot pro-

vide answers if research data do not exist (other than

using expert opinion) and it cannot substitute for

highly developed clinical skills. Therefore, it can

never be cookbook healthcare or use statistics in

isolation to drive clinical care. Instead it is the com-prehensive integration of appropriate research evi-

dence, patient preference and clinical expertise

(Fig. 1).

This can be illustrated with data from a recent

systematic review on periodontal plastic surgery for

root surface coverage in localized Miller Class I and II

defects (25). The data from the systematic review

demonstrated that connective tissue grafts were sig-

nificantly better than guided tissue regeneration in

12

Periodontology 2000, Vol. 37, 2005, 1228

Printed in Denmark. All rights reserved

Copyright Blackwell Munksgaard 2005

PERIODONTOLOGY 2000


2/17

reducing recession (mean difference 0.43 mm,

95%CI [0.62,0.23], chi square for heterogeneity 7.8

(df 5) P 0.17). This indicates that the pooled

difference between six studies included in the review

is 0.43 mm, with a 95% confidence interval from 0.62

to 0.23. The chi-square test indicates that there is no

evidence of any heterogeneity between the studies

(they could theoretically all be measuring the samedifference). So, does this mean that only connective

tissue grafts should be used in the treatment of

localized recession defects? Clearly, this would not be

appropriate. The data show that both GTR and con-

nective tissue grafts can work. For the selected out-

come, which was recession reduction, connective

tissue grafts produce 0.43 mm greater effect; the

result is both reasonably precise (judged by the

confidence interval) and the studies from which

the data were taken were similar (no evidence of

heterogeneity).

However, recession reduction might not be theonly outcome of interest. The two surgical proce-

dures are very different. One requires the harvesting

of a soft tissue graft from the palate and the other

does not. There are no data available examining

patient preferences, but it is likely that some indi-

viduals will prefer a procedure that does not involve

two surgical sites, even if it does not reduce

recession to the same extent. It is also possible that

aesthetics are different following the two procedures

and this might inform on the decision. However,

surprisingly, no data are available on patient views

on aesthetics comparing the two procedures.

Therefore, this evidence-based approach to man-

agement of recession has produced the best avail-

able evidence, shown how precise this estimate

actually is, and highlighted the limitations of the

evidence, in this case the lack of data on some

outcomes that are relevant to the decision makingprocess.

Clinical relevance

One of the barriers to the application of research

findings in clinical practice is the way that results are

often presented. Typically, a mean value will be

published, based on a statistical analysis comparing

experimental groups. Such a value in conjunction

with its associated 95% confidence interval is useful

to determine whether there is a statistically signifi-

cant difference between groups and will often be a

requirement of a study designed for regulatory

approval. However, this type of analysis is not

designed to provide information about the probabil-

ity of achieving a certain outcome were the reader to

apply it in practice. Such an outcome could include

achieving a health benefit or preventing further dis-

ease. For instance, in a meta-analysis from a sys-

tematic review on guided tissue regeneration (GTR)

for periodontal infrabony defects, the additional

benefit of using GTR over access flap surgery was a1.1 mm gain in clinical attachment (21, 22). This

should, however, not be interpreted as the additional

benefit to be expected every time that GTR is used

instead of access flap surgery.

One approach to analysing and presenting data in a

more clinically useful format is to calculate the

number needed to treat (NNT). This is the number of

patients that would need to be treated to achieve a

stated benefit (NNTb) or to avoid a stated harm

(NNTh). It is derived from a dichotomous outcome

such as the proportion of sites achieving at least

2 mm gain in attachment. For the GTR meta-analysis,and using this benefit, the NNTb is eight. In other

words, for every eight patients treated with GTR, you

can expect one to have at least 2 mm more gain in

clinical attachment than if you had used an access

flap (95% confidence interval [4,33]). For detailed

guidance regarding the use and calculation of

the NNT the reader is recommended to the elec-

tronic journal Bandolier: http://www.jr2.ox.ac.uk/

bandolier/booth/painpag/NNTstuff/numeric.htm.

Cliniciansskills

Best evidenceavailable

Patientpreferencesand views

Evidence-basedperiodontology

Fig. 1. How evidence-based periodontology fits into

healthcare. Reproduced with permission from Clarkson,

J, Harrison, JE, Ismail, AI, Needleman, IG, Worthington, H,eds. Evidence Based Dentistry for Effective Practice.

London: Martin Dunitz, 2003 (20).

13

Evidence-based periodontology


3/17

Evidence-based periodontology vs.traditional periodontology

High quality research and the use of evidence are

fundamental to both evidence-based periodontology

and traditional periodontology. The differences

between these approaches emanate from how

research informs clinical practice. Evidence-basedperiodontology uses a more transparent approach to

acknowledge both the strengths and the limitations

of the evidence. An appreciation of the level of

uncertainty or imprecision of the data is essential in

order to offer choices to the patient regarding

treatment options. Evidence-based periodontology

also attempts to gather all available data and to

minimize bias in summarizing the data. These

aspects are key to decision making and are high-

lighted in Table 1.

Furthermore, evidence-based periodontology

acknowledges explicitly the type or level of research

on which conclusions are drawn. The research hier-

archy is discussed in more detail later in this chapter.

However, one aspect that influences the reliability of

the data is the control of bias. Bias is a collective term

for factors that systematically distort the results of

research away from the truth. Different research

designs offer different possibilities for the control of

bias and therefore vary in their reliability and will be

discussed further below.

The components of evidence-basedperiodontology

An overview of the components is given in Fig. 2.

Evidence-based periodontology starts with therecognition of a knowledge gap. From the know-

ledge gap comes a focussed question that leads on

to a search for relevant information. Once the rele-

vant information is located, the validity of the

research needs to be considered in two broad areas.

Firstly, is the science good (internal validity)? Inter-

nal validity focuses on the methodology of research.

Secondly, can the findings be generalized outside of

the study (external validity)? External validity might

be affected by the way treatment was performed. For

instance, if the time spent on treatment was exten-

sive it might not be practical to provide this therapy

outside of a research study. Another example could

relate to the use of many specific inclusion criteria

in a trial which could make it difficult to generalize

the findings to a wider group of patients. The

question the reader should ask is whether their types

of patients are so different from the study that it is

Table 1. Comparison of evidence-based periodon-

tology vs. traditional periodontologyEvidence-based

periodontology

Traditional

periodontology

Similarities

High value of clinical

skills and experience

Fundamental importance

of integrating evidence

with patient values

Differences

Uses best evidence

available

Systematic appraisal

of quality of evidence

More objective, more

transparent and less

biased process

Greater acceptance

of levels of uncertainty

Unclear basis

of evidence

Unclear or absent

appraisal of quality

of evidence

More subjective, more

opaque and more

biased process

Greater tendency to

black and white

conclusions

Reproduced with permission from Clarkson, J, Harrison, JE, Ismail, AI,Needleman, IG, Worthington, H, eds. Evidence Based Dentistry for EffectivePractice. London: Martin Dunitz, 2003 (20).

Reject

if invalid

or poor

Evaluate the effects

Integrate into practice

Evaluate the evidence

Search for evidence

Develop into a focussed question

Recognize clinical knowledge gap

Fig. 2. The steps of evidence-based periodontology.

Reproduced with permission from Clarkson, J, Harrison,

JE, Ismail, AI, Needleman, IG, Worthington, H, eds. Evi-

dence Based Dentistry for Effective Practice. London:

Martin Dunitz, 2003 (20).

14

Needleman et al.


4/17

reasonable to expect differences in outcomes. After

locating and appraising the research, the results

then need to be applied clinically, or at least inclu-

ded in a range of options. Finally, the results in

clinical practice need to be evaluated to reveal

whether the adopted technique achieved the

expected outcome.

The example of gingival recession mentioned ear-

lier can be used to illustrate this approach. Theuncertainty might relate to whether to change from

using connective tissue grafts for recession defects to

guided tissue regeneration and can be translated into

a focussed question. Here the patient or problem

group could be refined more closely to localized

recession defects and perhaps Miller Class I or II, as

we might reasonably expect these lesions to respond

differently from more advanced lesions. The inter-

vention is guided tissue regeneration and the com-

parison, connective tissue grafts. The outcomes would

include change in recession or possibly the chance of

achieving complete root coverage. Since the proce-

dure is primarily for aesthetics, a patient-centred

assessment of aesthetics should be an outcome. As

always, there must be a consideration of adverse

effects and these might include pain, postoperative

infection, and severe bleeding postoperatively.

Reassembling this structure into a focussed

question would lead to In patients with localized

Miller Class I or II recession defects, what is the

effect of guided tissue regeneration vs. connective

tissue grafts on change in recession, chance of

complete defect coverage and aesthetics and whatare the adverse effects? For this particular research

question, the randomized controlled trial is best able

to address the change in recession outcome. For the

other outcomes, other research designs might have

been used, such as observational studies. Preferably,

we would like to find a systematic review that will

have completed the searching and study appraisal

for us.

The search quickly identifies a systematic review

(25). The review has a research question that is

appropriate to our question and demonstrates a

statistically superior effect of connective tissue graftscompared with guided tissue regeneration. The

review also acknowledges certain limitations. In

terms of the validity of the meta-analysis, the

reviewers urge caution as publication bias could be

affecting the overall result, but this could not be

tested due to the low number of studies. Publication

bias is discussed later in this chapter. Another limi-

tation was that there were no data on aesthetics or

adverse effects.

Therefore, having reviewed the data, it is clear that

there is good evidence to indicate that connective

tissue grafts have a greater effect on change in

recession than guided tissue regeneration, although

there are several limitations to this evidence. Clinical

recommendation is tempered by the lack of data on

aesthetics and adverse effects and the possible

exaggeration of benefit through publication bias. This

information can then inform on the case presenta-tion to the patient and a choice of options discussed

and agreed. The outcome of treatment can then be

evaluated to see whether the desired endpoint was

achieved and this helps to refine the case presenta-

tion discussion in future.

Systematic reviews

One important element of evidence-based period-

ontology is the systematic review. Systematic reviews

are a research design termed research synthesis.

That is, they use research methodology to pool data

from multiple studies that address a particular

hypothesis. A systematic review can be defined as a

review of a clearly formulated question that attempts

to minimize bias using systematic and explicit

methods to identify, select, critically appraise and

summarize relevant research.

The description of systematic reviews as providing

the highest level of evidence is widespread but also

raises expectations that may or may not be fulfilled. A

realistic understanding of what a systematic reviewcan provide is important for the appropriate use of

this type of evidence (Table 2). More detailed infor-

mation on systematic reviews exist (5, 19), and guides

to conducting them are freely available (1, 13).

As with all research, a systematic review starts from

an hypothesis. This is derived from a focussed ques-

tion which is set to answer a particular area of

uncertainty. For instance, for the systematic review

on smoking and periodontal therapy in the chapter by

Labriola et al. in this volume, the focussed question

was: In patients with chronic periodontitis, what is

the effect of smoking or smoking cessation on theresponse to nonsurgical periodontal therapy in terms

of clinical and patient-centred outcomes?(14). The

question has set the types of patients (individuals

with chronic periodontitis undergoing nonsurgical

therapy), type of exposure (cigarette smoking) and

types of outcomes (clinical and patient-centred) to be

investigated, each aspect being defined in more detail

within the protocol. As this is a prognostic research

question, where exposure (smoking) cannot be

15



5/17

randomized, the cohort study is the research design

of choice to incorporate into this investigation.

These components help in the design of the search

strategy that aims to be comprehensive. Usually,

searching of multiple electronic databases is carried

out together with searching other sources. The most

commonly searched databases include MEDLINE

(strong on English-language studies), EMBASE(strong on other European languages), and CENTRAL

(the Cochrane Collaboration register of trials

records). Searching only electronic databases can

miss important data, as records on the database may

not be appropriately coded. To supplement the

electronic search, other approaches are used.

Typically, this will include checking for publications

in the bibliographies of retrieved studies and review

articles, hand-searching of journals for missed

reports, and contacting researchers, industry and

journals for unpublished data.

The search strategy aims for high sensitivity, i.e. the

greatest chance of finding all relevant studies. The

downside of this approach is low precision, i.e. in

addition to the relevant studies, the search will

identify many irrelevant hits (probably more than

90% of hits from the search will not be relevant). Forexample, in a systematic review on systemic anti-

microbials, the search identified 1,300 hits. Screening

of the title and abstracts (if available) indicated that

158 papers might be relevant. Once the full text of the

studies had been reviewed, 25 trials were judged

relevant and could be included (9). At first sight,

rejecting 1,275/1,300 studies would appear to be

wasting potentially useful data. However, the delib-

erately inclusive search identifies a large number of

Table 2. The potential of systematic reviews

What a high quality systematic review can do

Find and summarize all available studies.

A comprehensive search will identify all relevant studies up to the point of the date of the completion of the search.

This should give the reader greater confidence that bias in selecting studies has been minimized.

Provide an objective assessment of the quality or research and in particular the degree of protection from bias within

the original studies.

Components of methodological quality that have evidence of affecting bias can be evaluated. It may be much harder

to evaluate the impact of otherquality

issues if there is no consensus on how to measure them. An example could be

the quality of the treatment provided.

Estimate research effects across multiple studies with meta-analysis.

Meta-analysis is valid only if studies are similar in their research question and design.

Meta-analysis can estimate uncertainty and precision of the effect.

Meta-analysis may generate hypotheses for differential effects across subgroups of the population tested.

If the effect is consistent across multiple studies (with small differences in design), then it may more readily possible to

generalise the results to clinical practice than the results from a single study.

Overcome limitations of underpowered studies in detecting a true difference if such a true difference really exists.

What a high quality systematic review cannot do

It cannot be used in isolation to dictate clinical practice.

It is a synthesis of available research and must be used in context with clinical judgement and patient preference.

Produce strong conclusions if the research base is weak in quality.

The value of the review will then be to present a comprehensive objective summary of the strength of the data and toidentify the design of research to answer important gaps.

Overcome limitations of narrowly designed clinical research.

If the clinical studies only investigate the effect of an intervention in highly selected individuals, the conclusions

cannot be generalised outside of these conditions. However, the objective communication of any limitations in the

research base will help to set the degree of uncertainty and indicate the priorities for future research.

Exclude relevant studies.

Although the majority of hits from the search will be excluded, this is due to the deliberate strategy of achieving high

sensitivity (likelihood of finding all relevant studies) but low precision (likelihood of only finding relevant studies).

Therefore, it is common to find that more than 90% of the search records are totally irrelevant to the question and

must be excluded. The alternative approach of aiming for high precision also carries a high risk of missing relevant

studies, although it will appear as if few studies are being excluded.

Be a miracle research design.

All research has strengths and limitations/weaknesses. Systematic reviews are no different from other research

designs in this respect.

16

Needleman et al.


6/17

irrelevant papers, including veterinary medicine,

review papers, duplicate reporting of research and

laboratory studies. The systematic review screens the

search findings against prestated criteria. These cri-

teria aim to exclude studies irrelevant to answering

the question, but do not attempt to exclude on the

basis of the quality of the study. Instead, the quality of

relevant studies is critically appraised using objective

criteria that could influence the study outcome.The dimension of quality can be incorporated into

a systematic review in a number of ways. If the

studies are similar enough to be combined in a meta-

analysis, the impact of quality on the overall result

can be estimated (through sensitivity analyses or

meta-regression). If meta-analysis is not possible, the

quality of studies can be summarized in narrative

tables, in particular for those elements designed to

protect against bias. Whilst this may not be as

powerful as the use of meta-analysis, it will highlight

limitations to be placed upon the conclusions. Fol-

lowing pooling of the data with meta-analysis or

qualitative methods, the conclusions from the

investigation can be drawn and related to the data

derived from the review.

The systematic review will not be appropriate for

some questions. For instance, to address the ques-

tion, Which indices have been used to measure gin-

givitis? a descriptive survey will be more appropriate.

However, systematic methods should be adopted for

some aspects, in particular to ensure that the search

is both comprehensive and contemporary. This

might form an important initial stage to answering aquestion such as Which gingivitis indices have been

validated? This is a research question answerable by

a systematic review.

The development of evidence-based periodontology

Evidence-based periodontology is built upon devel-

opments in clinical research design throughout the

18th, 19th and 20th centuries (15, 20, 23, 28).

Evidence-based medicine has only been known for just over a decade and the term was coined by the

clinical epidemiology group at McMaster University

in Canada (4).

The influence of the McMaster group spread far.

One of the earliest to take up the challenge in peri-

odontology (in fact in oral health research overall)

was Alexia Antczak Bouckoms in Boston, USA. Ant-

czak Bouckoms and colleagues challenged the

methods and quality of periodontal clinical research

in the mid 1980s (3) and set up an Oral Health Group

as part of the Cochrane Collaboration in 1994. The

editorial base of the Oral Health group subsequently

moved to Manchester University in 1997 with Bill

Shaw and Helen Worthington as co-ordinating edi-

tors (http://www.cochrane-oral.man.ac.uk/). The

first Cochrane systematic review in periodontology

was published in 2001 and researched the effect of

guided tissue regeneration for infrabony defects (21).Many individuals have been active in the critical

analysis of the periodontal literature. These include

Jan Egelberg, Loma Linda University, Noel Claffey,

Trinity College Dublin, and Gary Greenstein, Uni-

versity of Medicine and Dentistry of New Jersey.

There have been many notable events in evidence-

based periodontology. The 1996 World Workshop in

Periodontology held by the American Academy of

Periodontology included elements of evidence-based

healthcare, supported by Michael Newman at UCLA

(2). The 2002 European Workshop on Periodontology

became the first international workshop to use rig-

orous systematic reviews to inform the consensus.

The workshop was organized by the European Acad-

emy of Periodontology for the European Federation of

Periodontology, under the chairmanship of Professor

Klaus Lang. Sixteen focussed and rigorous systematic

reviews formed the basis of intense consensus dis-

cussions. A similar approach was used subsequently

by the American Academy of Periodontology for the

Contemporary Science Workshop in 2003.

Many other groups are now using similar methods

in healthcare and research. Most recently, the Inter-national Center for Evidence-Based Oral Health was

launched in 2003 (http://www.eastman.ucl.ac.uk/

iceboh) to produce high quality evidence-based

research with an emphasis on, but not limited to,

periodontology and implants and to provide generic

training in systematic reviews and research methods.

Study designs and critical appraisal

Different study designsDifferent clinical research questions require evalua-

tion through different study designs. A study to

determine the effectiveness of surgical therapy com-

pared with nonsurgical debridement deals with the

effectiveness of a treatment option and would be best

answered by a randomized controlled trial (RCT) or,

ideally, a systematic review of RCTs. However, it must

be noted that although RCTs and systematic reviews

of RCTs may well be the gold standard upon which

17



7/17

to base decisions on the effectiveness of interven-

tions, they are not necessarily appropriate, or ethical,

to answer all questions. An RCT would obviously not

be helpful in answering the question posed on the

epidemiological evidence of plaque in the etiology of

periodontitis. For such questions regarding prognosis

or etiology, cohort studies would be more appropri-

ate. Table 3 illustrates the types of study designs

most suitable for different types of research questionsarising in periodontology. The most appropriate

source of information will depend upon the type of

study design being sought.

Critical appraisal: Why, what and how?

Why critically appraise?

Evidence-based periodontology, as its name implies,

is periodontology that is based on evidence, but not

just any so-called evidence. Richards wrote a toolbox

article for the journal Evidence-based Dentistry enti-

tled Not all evidence is created equal (24). We have

already seen in this chapter that the quality of evi-

dence may vary according to study design and that

this has led to the concept that there can be a hier-

archy of evidence. One hierarchy is illustrated in

Table 4 and is specific to studies on therapy, pre-

vention, etiology, and harm. Other suggested levels

for different types of research question can be found

at the Center for Evidence-Based Medicine: (http://

www.cebm.net/levels_of_evidence.asp#levels).

The publication of research in a high-rankingjournal may not be an absolute guarantee of quality.

Within the medical literature there are methodolo-

gical studies which have empirically shown that

quality is not merely a hypothetical concept but also

affects study outcomes. As examples of this, the re-

views of Schulz et al. (26), Moher et al. (17) and Juni

et al. (12) showed that in studies in which there was

inadequate concealment of treatment allocation, the

treatment effects were exaggerated by about 40%

compared to trials of higher quality.

Quality assessment of trials in periodontology and

implantology

Two recent studies have investigated the methodo-

logical quality of RCTs in periodontology and

implantology as assessed by their publications. Both

studies targeted RCTs for investigation due to the

importance of the RCT in providing evidence for

the effect of interventions and also because of the

empiric data indicating the effect of key domains of

methodology on bias.

Montenegro et al. (18) conducted a systematic

review of the quality of RCTs of periodontal ther-

apy, published in Journal of Periodontology, Journal

of Clinical Periodontology or Journal of Periodontal

research over a 3-year period from 1996 to 1998.

From the electronic search, 283 papers were poss-

ibly relevant and 177 studies met the inclusion

criteria of being an RCT, performed on humans and

for which a full text article was available. Screeningand data abstraction were performed independently

and in duplicate to minimise error and bias. The

evaluation was not performed blind to author

affiliation identity of the RCTs as the evidence

suggests that this has a minimal impact on out-

come (16). In view of the empirical data described

above, the quality components chosen were those

demonstrated to be important for protection from

bias: adequacy of method of generation of the

random sequence, adequacy of method of con-

cealing the allocation sequence from the patient

recruitment, examiner blinding (where it was

judged possible to achieve), and handling of losses

and withdrawals.

The results indicated that 29/177 (17%) of RCTs

employed a clearly adequate method of generating

the random number sequence, and that 12/177 (7%)

of studies described adequate allocation conceal-

ment (Fig. 3). Furthermore, where examiner blinding

was possible, 97/177 (55%) of studies reported an

adequate method. Clear accounting for study sub-

jects was present in 100/177 (55%) of reports. Since

the study was conducted on trial reports, it is notclear how much of the inadequacy was due to

incomplete reporting rather than inadequate study

methods. If the data do reflect study conduct, then

bias and exaggeration of the effect of the test inter-

ventions could be a problem with some trials in

periodontology.

Similar results were found when investigating

RCTs of oral implants (6). This study searched for

RCTs up to the end of 1999 in multiple databases.

Seventy-four publications were located and 43 RCTs

were quality assessed as many studies were pre-

sented in multiple publications. Although themethods and criteria were a little different for this

study compared with the quality appraisal of peri-

odontal studies, the results are broadly comparable.

A clearly adequate method of randomization/con-

cealment of allocation was present in 1/43 (2%)

papers. Blinding was described in 12/43 (28%)

studies and the reasons for withdrawals and losses

to follow-up were specified in 33/43 (77%) of

reports.

18

Needleman et al.


8/17

Table3.Studydesignsand

thetypesofquestionstheyaddress

Definitionofstudydesign

Usedfor(examplesgiveninitalics)

Experimentalstudies

Randomized-controlledtrial:parallelgroupdesignagroupof

participants(orotherunitofanalysis,e.g.teeth)israndomizedinto

differenttreatmentgroups.Thesegroupsarefollowedupforthe

outcomesofinterest

Randomized-controlledtrial:split-mouthdesigneachpatientis

his/herowncontrol.Apairofsimilarteeth,orgroupsofteeth

(quadrants),maybeselected

andrandomlyallocatedtodifferent

treatmentgroups.

Non-randomizedcontrolledtrialallocationofparticipantsunder

thecontroloftheinvestigator

,butthemethodfallsshortof

genuinerandomization.

Evaluatingtheeffec

tivenessofanintervention

Randomizedcontrolledtrialcomparingtheeffectivenessofsurgicaltherapy

andnonsurgicaldebridement.

Controlledtrialcom

paringtwomethodsoftreatingperiod

ontalintrabony

defectsusingpairso

fsiteswheretheLHSisalwaysgroupAandtheRHSgroupB.

Observationalstudies

Cohort:alongitudinalstudy,identifyinggroupsofparticipants

accordingtotheirexposure/interventionstatus.Groupsarefollowed

forwardintimetomeasureth

edevelopmentofdifferentoutcomes.

Case-Control:astudythatide

ntifiesgroupsofparticipants

accordingtotheirdisease/outcomestatus.Groupsareinvestigated/

questionedtodeterminetheirexposurestatus

Cross-sectional:astudy(survey)undertakenonadefinedpopulatio

n

atasinglepointintime(snap-shot).Subjectsareobservedonjust

oneoccasionandarenotfollowedup.

Measuringtheincidenceofadisease;lookingatthecaus

esofdisease;

determiningprognosis.

Cohortstudylookin

gattheprogressofperiodontitisovertimeandrelatingthisto

externalfactorssuc

hassmokingorplaque.

Identifyingpotentia

lriskfactorsforadisease;lookingatthepossiblecausesof

disease.

Case-controlstudyl

ookingattheprevalenceofperiodontitisandrelatingthisto

factorssuchasgene

ticmarkers.

Measuringtheprev

alenceofadiseaseorriskfactorinadefinedpopulationata

specifictime.

Across-sectionalstu

dytodeterminethecurrentperiodontaltreatmentneedsin

aspecificpopulation.

19



9/17

Improving the quality of reporting ofclinical research in periodontology

The adequacy of reporting of clinical research is

crucial if the reader is to evaluate the quality and

possible impact of studies. The importance of sev-

eral of the quality issues that we have described has

not been thoroughly appreciated until relatively

recently. Therefore, it is unfair to judge the past

from the standpoint of current knowledge. In

addition, the pressure on page numbers in paper-

based journals can restrict detail. Hopefully this

aspect will be alleviated by initiatives in electronic

publication.

Guidelines are available to help the publication of

clinical research. These guidelines are well accepted

by high impact biomedical journals and offer guid-

ance not only to authors but also to editors and

reviewers. These guidelines include CONSORT(Consolidated Standards of Reporting Trials) for

reporting randomized controlled trials and STARD

(Standards for Reporting of Diagnostic Accuracy) for

reporting studies on diagnostic tests (http://consort-

statement.org/). In addition, three guidelines for

reporting systematic reviews are available: QUOROM

(Quality of Reporting of Meta-analyses) (http://

consort-statement.org/), MOOSE (Meta-analysis Of

Observational Studies in Epidemiology) (27), and

QUADAS (Quality Assessment of studies of Diagnos-

tic Accuracy included in Systematic reviews) (29). For

clarification, it should be remembered that system-atic reviews are termed meta-analyses by some in

North America, whereas the term meta-analysis is

usually reserved only for the statistical combining of

data which may or may not be part of a systematic

review.

The format of these guidelines is similar. Each

presents a checklist of items for incorporation into

the research report. The selection of items is evi-

dence-based as far as possible and otherwise derived

Table 4. Center for Evidence-Based Medicine hier-archy of evidence for studies on therapy, prevention,etiology or harm (http://www.cebm.net/levels_of_evidence.asp#levels)

Level Type of evidence

Ia Systematic review (with homogeneity*) of

randomized controlled trials (RCT).

1b Individual RCT (with narrow confidenceinterval, see notes below).

2a Systematic review (with homogeneity*) of

cohort studies.

2b Individual cohort study (including low quality

RCT; e.g. < 80% follow-up).

2c Outcomes research; Ecological studies.

3a Systematic review (with homogeneity*) of

case-control studies.

3b Individual case-control study.

4 Case-series (and poor quality cohort and

case-control studies)

5 Expert opinion without explicit critical

appraisal, or based on physiology, bench

research or first principles.

Users can add a minus-sign to denote the level of thatfails to provide a conclusive answer because of:

EITHER a single result with a wide Confidence Interval(such that, for example, an absolute risk reduction in anRCT is not statistically significant but whose confidenceintervals fail to exclude clinically important benefit orharm);

OR a Systematic Review with troublesome (and statisti-cally significant) heterogeneity.

Such evidence is inconclusive.*A systematic review that is free of worrisome variations(heterogeneity) in the directions and degrees of resultsbetween individual studies. Not all systematic reviews

with statistically significant heterogeneity need be wor-risome, and not all worrisome heterogeneity need bestatistically significant. As noted above, studies display-ing worrisome heterogeneity should be tagged with a at the end of their designated level.

Poor quality cohort study: one that failed to clearly de-fine comparison groups and/or failed to measure expo-sures and outcomes in the same (preferably blinded),objective way in both exposed and nonexposed individ-uals and/or failed to identify or appropriately controlknown confounders and/or failed to carry out a suffi-ciently long and complete follow-up of patients. Poor

quality case-controlstudy:one that failedto clearly definecomparison groups and/or failed to measure exposuresand outcomes in the same (preferably blinded), objective

way in both cases and controls and/or failed to identify orappropriately control known confounders.

0%

10%

20%

30%

40%

50%

60%

Randomization

method

Allocation

concealment

Examiner blinding Accounting for all

subjects

Fig. 3. Quality of reporting of randomized controlled tri-

als in periodontology (18). Percentage of studies with

adequate method.

20

Needleman et al.


10/17

(a)

PAPERSECTIONAnd topic

Item Description Reportedon

Page #TITLE &

ABSTRACT1 How participants were allocated to interventions

(e.g., "random allocation", "randomized", or"randomly assigned").

INTRODUCTIONBackground

2 Scientific background and explanation of rationale.

METHODSParticipants

3 Eligibility criteria for participants and the settingsand locations where the data were collected.

Interventions 4 Precise details of the interventions intended foreach group and how and when they were actuallyadministered.

Objectives 5 Specific objectives and hypotheses.Outcomes 6 Clearly defined primary and secondary outcome

measures and, when applicable, any methodsused to enhance the quality of measurements(e.g., multiple observations, training of assessors).

Sample size 7 How sample size was determined and, whenapplicable, explanation of any interim analysesand stopping rules.

Randomization --sequencegeneration

8 Method used to generate the random allocationsequence, including details of any restrictions(e.g., blocking, stratification).

Randomization --allocation

concealment

9 Method used to implement the random allocationsequence (e.g., numbered containers or centraltelephone), clarifying whether the sequence was

concealed until interventions were assigned.Randomization --Implementation

10 Who generated the allocation sequence, whoenrolled participants, and who assignedparticipants to their groups.

Blinding(masking)

11 Whether or not participants, those administeringthe interventions, and those assessing theoutcomes were blinded to group assignment.When relevant, how the success of blinding wasevaluated.

Statisticalmethods

12 Statistical methods used to compare groups forprimary outcome(s). Methods for additionalanalyses, such as subgroup analyses andadjusted analyses.

RESULTS

Participant flow

13 Flow of participants through each stage (adiagram is strongly recommended). Specifically,for each group report the numbers of participantsrandomly assigned, receiving intended treatment,completing the study protocol, and analyzed for

the primary outcome. Describe protocol deviationsfrom study as planned, together with reasons.

Recrui tment 14 Dates def ining the periods of recrui tment andfollow-up.

Baseline data 15 Baseline demographic and clinical characteristicsof each group.

Numbersanalyzed

16 Number of participants (denominator) in eachgroup included in each analysis and whether theanalysis was by intention-to-treat. State theresults in absolute numbers when feasible (e.g.,10/20, not 50%).

Outcomes andestimation

17 For each primary and secondary outcome, asummary of results for each group, and theestimated effect size and its precision (e.g. 95%confidence interval).

Ancillaryanalyses

18 Address multiplicity by reporting any otheranalyses performed, including subgroup analyses

and adjusted analyses, indicating those pre-specified and those exploratory.Adverse events 19 All important adverse events or side effects in

each intervention group.DISCUSSIONInterpretation

20 Interpretation of the results, taking into accountstudy hypotheses, sources of potential bias orimprecision and the dangers associated withmultiplicity of analyses and outcomes.

Generalisabil ity 21 Generalisabil ity (external validity) of the trialfindings.

Overall evidence 22 General interpretation of the results in the contextof current evidence.

Fig. 4. a) CONSORT Checklist of items to include when reporting a randomized trial. b) CONSORT Flow chart. Available

from: http://www.consort-statement.org/.

21



11/17

by a Delphi approach to consensus. In addition to the

checklist, a chart is used to illustrate the flow of pa-

tients through the study. The checklist and chart for

CONSORT are illustrated in Fig. 4a, b. The checklist

should accompany the manuscript in its journal

submission but not be part of the final paper. The

intention with the chart, however, is that it should be

published as part of the paper. Whilst the checklist

has numerous items, each can be concisely ad-

dressed and is unlikely to be the main cause forexcessive length of a publication.

At the time of writing, oral health journals that have

adopted CONSORT as editorial policy and their dates

of adoption are: British Dental Journal(1999), Journal

of Orthodontics (2000), International Journal of End-

odontics(2003) and Journal of Dental Research (2004).

The British Dental Journal is the only one that has

adopted QUOROM (2002).

What should be appraised?

Given that some evidence is better than other evi-dence, it seems reasonable to place greater emphasis

on good than on poor quality evidence when mak-

ing clinical decisions. The problem arises as to how

exactly we decide what constitutes good quality evi-

dence. This process is critical appraisal. The validity of

published evidence is potentially affected by the

quality of every stage of the experimental process

from aims and objectives, through design, execution,

analysis, interpretation, and finally publication. Al-

though deliberate deception is always a possibility,

the majority of problems that arise are in fact unin-

tentional. Most methodological errors may be classi-

fied as being the result of bias, confounding, or

chance. Therefore, for the purpose of this chapter,

quality will be discussed in relation to these meth-

odological issues. Other aspects of study conduct may

well be critical to the validity of a study but will not be

considered in this chapter as they will be specific for aparticular study. Such factors could include how well

treatment or supportive maintenance was provided.

Bias

Bias is a systematic error. It leads to results which are

consistently wrong in one or another direction. Bias

leads to an incorrect estimate of the effect of a risk

factor or exposure (e.g. smoking) on the development

of a disease or outcome of interest (e.g. response to

periodontal therapy). The observed effect will be

either above or below the true value. Many types of

bias have been identified, however, the main types

relate to:

how subjects were selected for inclusion in a study

(selection bias);

provision of care (performance bias);

assessment of outcomes (detection/measurement

bias);

occurrence and handling of patient attrition

(attrition bias).

Selection bias occurs when there is a systematic

difference between the characteristics of the subjectsselected for a study and the characteristics of those

who were not. For instance, selection bias will often

occur with volunteers (self-selection bias). People

who volunteer to participate in a study tend to be

different from the general population. Similarly, it is

important to consider whether people might have

selectively withdrawn from the study before its

completion (attrition bias). They may have with-

drawn at random, or because of some factor related

to the study, e.g. the treatment they were receiving

was ineffective or uncomfortable in comparison with

the alternative treatment. It is necessary to decidewhether the results of the investigation were likely to

have been compromised if one group of subjects had,

on average, a shorter follow-up as a result of more

people dropping-out.

The avoidance of selection bias is a major concern

in the design of case-control studies. In this type of

study it is essential to ensure that controls are rep-

resentative of the population from which the cases

originated. Suppose a group of researchers is con-

Assessed foreligibility (n= ... )

Excluded (n = ... )

Not meetinginclusion criteria(n = ... )

(n = ... )Refused to participate

Other reasons (n = ... )

Randomized (n = ... )

Allocated to intervention(n = ... )

intervention (n = ... )

(give reasons) (n = ... )

(give reasons) (n = ... ) (give reasons) (n = ... )

(n = ... ) (give reasons)

intervention

Received allocated

Did not receive allocated

Allocated to intervention(n = ... )

intervention (n = ... )

(give reasons) (n = ... )intervention

Received allocated

Did not receive allocated

(give reasons)Lost to follow up (n = ... )

Discontinued intervention(n = ... ) (give reasons)

(give reasons)Lost to follow up (n = ... )

Discontinued intervention

Excluded from analysis Excluded from analysis

Analysed (n = ... ) Analysed (n = ... )

Analysis

Allocation

Enrollment

Followup

(b)

Fig. 4. Continued.

22

Needleman et al.


12/17

ducting a case-control study to assess the effect of

cigarette smoking on the development of aggressive

periodontitis. In our hypothetical example, cases are

patients referred to a dental hospital with aggressive

periodontitis and controls are non-dental patients

admitted to a nearby hospital with chronic bronchi-

tis. A standard questionnaire is administered to both

cases and controls that includes questions on lifetime

smoking habits. The researchers may find no evi-dence from this study of an association between

cigarette smoking and aggressive periodontitis. Can

we accept this conclusion? The problem with this

study is that the choice of controls is biased, as the

prevalence of smoking among patients admitted with

chronic bronchitis is likely to be much higher than

among the general population resident in the catch-

ment area of the hospitals from which the cases and

controls originated. Consequently, the strength of

the association between smoking and aggressive

periodontitis will most likely be under-estimated in

this study.

Randomized controlled trials are less likely to be

affected by selection bias if the randomization is

properly conducted. Randomization is a two-stage

process. The first stage is the generation of a true

random sequence. Typically, this is achieved through

computer software or a random number table. Whilst

a tossed coin is theoretically acceptable, we suggest

using a method that can be audited later on for the

purposes of quality assessment, such as a computer

generated list.

The second stage of randomization is less wellunderstood or carried out. Once the random

sequence is generated, it must be concealed from

those selecting patients for a study until the indi-

vidual has been recruited into the trial. If not,

despite the sequence being random, the researcher

will be aware of whether the patient will be

entered into test or control group. This knowledge

provides the opportunity for selection bias, whether

intentional or not. This second stage of randomi-

zation is termed allocation concealment (see Fig. 5

for an outline of this process). The key question to

ask is, Was the recruitment of patients into a trialentirely unpredictable with respect to test or con-

trol group?

Performance bias occurs when different study

groups do not receive therapy in the same fashion or

to the same standard. This may occur if the people

providing the therapy are aware of which groups the

participants have been allocated to. Depending on

the nature of the investigation, it may be either a

relatively simple or a difficult task to ensure that

therapists remain masked (blinded) to the treatment

allocation. The use of placebo, where appropriate,

greatly facilitates masking; placebo controlled trials

are usually easy to organize in such a way as to leave

the therapist masked to the treatment allocation.

However, if the interventions to be compared are

quite dissimilar in their delivery (e.g. surgical vs.

nonsurgical therapy), then masking becomes con-

siderably more challenging. Under these circum-

stances the best available option might be to ensure

that the therapist remains masked until the last

possible moment to ensure that all therapy prior to

that point has been undertaken as even-handedly as

possible. So, for example, in a split-mouth study

comparing scaling and root planning vs. scaling and

root planning plus an adjunctive locally delivered

antimicrobial, it might be possible to complete themechanical therapy at all appropriate sites prior to

the therapist finding out which particular sites are to

receive the adjunctive therapy. However, the risk of

carry-over effects of the local antibiotic affecting the

scaling and root planning-only sites should not be

ignored.

Measurement (information) bias occurs when the

measurements of exposure and/or outcome are not

valid (i.e. they do not measure correctly what they

are supposed to measure). Errors in measurement

may be introduced by the observer (observer bias),

by the study individual (responder bias), or by theinstruments (instrument bias) used to make the

measurements (e.g. a badly designed questionnaire).

As a result of measurement errors, study partici-

pants will be misclassified in relation to their

exposure and/or outcome status. This misclassifi-

cation has particularly serious implications if the

errors in exposure measurement are related to the

participants outcome status. Ideally the person

undertaking the examination should be blinded

Step 1

Generate a true random sequence

- computer generated is best

- tossed coin is acceptable

Step 2

Allocation concealment- conceal the sequence for study recruitment

- sequentially numbered truly opaque envelopes/drug

containers, centrally kept randomization accessed by

telephone, e.g. pharmacy

Fig. 5. The two stages of randomization.

23



13/17

(masked). This is more important for measures in

which there is the potential for subjectivity (e.g.

pocket depth, colour change) than for objective

measures (e.g. tooth loss).

Bias is a consequence of defects in the design or

execution of a study. Bias cannot be controlled dur-

ing the statistical analysis of the data and cannot be

eliminated by increasing the size of the study.

Publication bias

Publication bias refers to the greater likelihood of

publication of studies with positive results than those

with neutral or negative results (5). The risk with this

type of bias is that interventions appear to perform

better than they will in clinical practice. For instance,

publication bias might mean that although several

studies were published and the data available to be

included in a meta- analysis, a larger number of

studies were actually conducted but not published.

Of these missing studies, some may show no dif-

ference between the intervention group and the

control group, or even the control group performing

better. If these additional studies had been published,

the results of the meta-analysis could have been

different. Therefore, the sample of published studies

is a biased sample and does not represent the com-

plete population of all research on this question.

Graphic and formal statistical tests are available to

investigate publication bias but need approximately

10 or more studies to have adequate power. Figure 6

illustrates this situation using hypothetical data.

Confounding

Confounding is a term that describes the situation

where an estimate of the association between an

exposure and the disease is mixed up with the real

effect of another exposure on the same disease, the

two exposures being correlated. It is a difficult con-

cept that may be illustrated with the help of the fol-

lowing example. Suppose we find that coffee drinkers

have a poorer response to periodontal therapy than

noncoffee drinkers. Does it mean that coffee drinking

affects the response to therapy? The problem here isthat there is an alternative explanation. Smoking may

be an independent risk factor for poor treatment

response and it is possible that people who drink

coffee are more likely to smoke than those who do

not. Perhaps the observed association is actually due

to smoking habits, not coffee drinking (Fig. 7).

Age and sex are the most common confounding

variables in health-related studies because these two

variables are not only associated with most exposures

we are interested in, such as diet, smoking habits,

health beliefs, etc., but are also independent risk

factors for many diseases.

Confounding can be dealt with at the design stage

of an investigation by:

Randomization By randomly allocating subjects

to study groups it is hoped that confounders are

distributed equally between the groups. This is

usually the most effective way of minimizing theproblem of confounding. If randomization is

done properly, it has the advantage that it con-

trols for both known and unknown confounders

provided the sample size is sufficiently large.

Restriction This limits participation in a study to

specific groups which are similar to each other

with respect to the confounder (e.g. if smoking is

likely to be a confounder then only nonsmokers

will be included in the study).

Matching This selects comparison groups with

similar backgrounds (e.g. nonsmokers are matched

with other nonsmokers, while smokers are mat-

ched with other smokers).

Confounding can also be controlled for in the ana-

lysis by:

Stratification Here the strength of the association

is measured separately in each well-defined sub-

group (e.g. in the smokers and the nonsmokers

separately). The results are then pooled together

using basic statistical techniques to obtain an

overall summary measure of the association adjus-

ted or controlled for the effects of the confounder.

Statistical modelling These are more sophisti-cated mathematical techniques that can simulta-

neously take into consideration the effects of

several possible confounders that have been

recorded by the investigators.

It is only possible to control for confounders in the

analysis if data on them were collected during the

study. Obviously, the extent to which confounding

can be controlled for will depend on the accuracy of

these data. However, in some situations it may be

virtually impossible to gain complete and accurate

information on confounders. Some confounders

may be so difficult to assess that even attempting toadjust for them in a statistical model will not com-

pletely control for their effect. For example, Hujoel

et al. have argued that the confounding effect of

smoking is virtually impossible to measure with

sufficient precision in studies that attempt to look at

the association between periodontal diseases and

systemic health and that such studies may only

provide valid results if they are restricted to non-

smokers (10).

24

Needleman et al.


14/17

Chance

Chance (sampling error) plays a role in most studies

of humans since it is rarely if ever possible to include

an entire population in an investigation. We therefore

attempt to infer information about the population on

the basis of information obtained from representative

samples drawn from that population. The extent to

which the sample results reflect the likely result in the

population is assessed by performing statistical sig-

nificance tests and, more importantly, by calculating

confidence intervals. A proper discussion of thesemethods is beyond the scope of this chapter, but in

general, studies with small sample sizes will be more

prone to sampling error and will provide less robust

estimates than studies with larger samples.

Interpretation

It is worth noting that authors may also fail to

interpret their experimental results correctly. So,

mean

2 1 0 1

Combined

study 15

study 14

study 13

study 12

study 11

study 10

study 9

study 8

study 7

study 6

study 5

study 4

study 3

study 2

study 1

(b)

1/standard

error

mean difference

.4 .2 0 .2 .4

0

5

10

15

20

25

30

35

40

45

50

(c)

(a)

mean

2 1 0 1

Combined

study 15

study 14study 13

study 12

study 11

study 10

study 5

study 9study 8

study 7

study 4

study 6

study 5study 3

study 4

study 2

study 1

study 3

study 2

study 1

(d)

1/standard

error

mean difference

.4 .2 0 .2 .4

0

5

10

15

20

25

30

35

40

45

50

Favours control

Favoursintervention

Favours control Favoursintervention

Fig. 6. Illustration of publication bias. a) Forest plot

showing the results of a meta-analysis of 15 studies. A verysmall improvement is indicated in favour of the test

intervention since the diamond shape (representing the

95% confidence interval for the pooled result) does not

cross the zero (no-effect) line. b) Funnel plot for these

studies. In the absence of publication bias, it is anticipated

that the plot would form a funnel shape. As no funnel

shape was produced, this indicates the possibility of

missing studies. These studies would be expected to pro-duce data points that lie somewhere within the shaded

area. c) Another Forest plot, including the data from these

extra, previously missing studies. The 95% confidence

interval for the pooled result now crosses the line of no

effect, indicating no evidence that the intervention is any

more effective than the control.

spurious associationCoffee drinking Poor treatment response

association risk factor

Smoking habits

(confounder)

Fig. 7. An example of confounding.

25



15/17

even if the study has been well conducted and

appropriately analyzed, there is still the potential to

draw incorrect conclusions from the results.

How to critically appraise?

When appraising quality it is necessary to consider

those factors that may affect the outcome of a study.These will inevitably vary according to both the topic

of the original research and the study designs

employed, so it is not possible to devise a single

system that will be appropriate for every occasion. As

a general rule, the aforementioned domains of bias,

confounding and chance will all have to be ap-

praised.

Some reviewers have attempted to devise com-

posite scales that give scores for the various quality

domains (11). These scores are then summed to an

overall summary measure for the study as a whole.

There are problems with this approach. Many

quality items may not be based on empirical evi-

dence and the scores attached to each item will

inevitably be subjective. It is also doubtful whether a

single summary score is likely to provide anadequate overall assessment of the quality of a

particular study. When different composite scales

are applied to the same studies, differing scores and

rankings may occur. For these reasons, composite

scales have largely gone out of favour. An alternative

approach is to appraise each quality component

separately (12).

Table 5. Quality assessment checklist for randomized controlled trials in periodontology used by Montenegro

et al. (18)

Item Classification Definition

Randomization Adequate If generated by random number

table (computer generated or not);

tossed coin; and shuffled cards.

Unclear Study refers to randomization but

either does not adequately explain

the method or no method was reported.

Inadequate Methods include alternate assignment,

hospital number, and odd/even birth date.

Allocation concealment Adequate Methods included central randomization

(e.g. by telephone to a pharmacy or trialoffice), pharmacy sequentially numbered/

coded containers, and sequentially

numbered opaque envelopes.

Unclear If the study referred to allocation

concealment but either did not

adequately explain the method or

no method was reported.

Inadequate Involved methods where randomization

could not be concealed, such as alternate

assignment, hospital number,

and odd/even birth date.

Blinding of patient, caregiver,

and examiner were considered

separately

Recorded as adequate, inadequate,

unclear, or for examiner blinding,

not applicable if the study

design precluded the

possibility of blinding.

Withdrawals and drop outs Were all patients who entered the

trial properly accounted for at the end?

Where dropouts occurred, the use of

analyses to allow for losses (such as

intention to treat) was noted.

26

Needleman et al.


16/17

For rigorous systematic reviews, independent

reviewers usually undertake quality appraisal in

duplicate and checklists are frequently employed for

this purpose. Two such checklists that have been

used previously are reproduced here as examples

(Tables 5 and 6) (7, 18). These checklists are based on

a combination of factors that have been shown

empirically to affect quality (such as allocation

concealment) and also topic specific factors deemed

important by the reviewers. Other checklists cover a

broad range of types of research and can be

found on the excellent Critical Appraisal Skills

Programme website (http://www.phru.nhs.uk/casp/

appraisa.htm).

The use of checklists with objective criteria helps

to safeguard the quality of the quality appraisal

process itself. The process of devising the check-list helps to ensure that all relevant quality issues

are included in the assessment. Written, piloted

checklists reduce, but can never completely elim-

inate individual subjectivity in decisions. Having a

written list means that it is more likely that the

quality assessors will be both consistent and

repeatable.

The results of the quality appraisal are used to

assess the value of the evidence and to aid clinicians

and reviewers in their efforts to place the evidence

into context. This might be a part of the formal pro-

cess of undertaking a systematic review or the

informal act of reading and assessing recently pub-

lished literature as part of everyday periodontal

practice.

Conclusions

The principles of evidence-based healthcare provide

structure and guidance to facilitate the highest levels

of patient care. There are numerous components to

evidence-based periodontology including the pro-duction of best available evidence, the critical

appraisal and interpretation of the evidence, the

communication and discussion of the evidence to

individuals seeking care and the integration of the

evidence with clinical skills and patient values. This

volume of Periodontology 2000 is mainly concerned

with the first component, i.e. the generation of best

evidence and, alone, is not enough to practise evi-

dence-based healthcare. However, an understanding

of the principles should help to underpin the latter

aspects. Evidence-based healthcare is not an easier

approach to patient management, but should provide

both clinicians and patients with greater confidence

and trust in their mutual relationship.

References

1. Alderson P, Green S, Higgins JPT, eds. Cochrane Reviewers

Handbook 4.2.2[updated. ]. In: TheCochraneLibrary,Issue1.

Chichester: John Wiley & Sons, 2004.

Table 6. Quality assessment checklist for systematicreviews in dentistry used by Glenny et al. (7)

Question (possible categories)

A. Did review address a focused question?

(yes, no, cant tell)

B. Did authors look for appropriate papers?


C. Do you think authors attempted to identify all

relevant studies?


D. Search for published and unpublished literature


E. Were all languages considered?


F. Was any hand searching carried out?


G. Was it stated that the inclusion criteria were

carried out by at least two reviewers?


H. Did reviewers attempt to assess the quality of the

included studies?(yes, no)

I. If so did they include this in the analysis?

(yes, no, cant tell, not applicable)

J. Was it stated that the quality assessment was

carried out by at least two reviewers?

(yes, no, not applicable)

K. Are the results given in a narrative or pooled

statistical analysis?

(narrative, pooled, not applicable)

L. If the results have been combined was it

reasonable to do so?


M. Are the results clearly displayed?


N. Was an assessment of heterogeneity made and

reasons for variation discussed?


O. Were results of review interpreted appropriately?


27



17/17

2. American Academy of Periodontology. Proceedings of the

1996 World Workshop in Periodontics. Lansdowne, Vir-

ginia, July 13-17, 1996. Ann Periodontol 1996: 1: 1947.

3. Antczak-Bouckoms A, Tang J, Chalmers TC. Quality

assessment of randomized controlled trials in dental

research. J Periodontal Res 1986: 21: 305314.

4. EBM Working Group. Evidence-based medicine. JAMA

1992: 268: 24202425.

5. Egger M, Davey-Smith G, Altman DG. Systematic Reviews in

Health Care, 2nd edn, London: BMJ Books, 2001.

6. Esposito M, Coulthard P, Worthington H, Jokstad A. Quality

assessment of randomized controlled trials of oral

implants. Int J Oral Maxillofac Implants2001: 16: 783792.

7. Glenny AM, Worthington H, Esposito M, Coulthard P. The

assessment of systematic reviews in dentistry. Eur J Oral

Sci 2003: 111: 8592.

8. Gray JAM. Evidence-based Healthcare. Edinburgh: Churchill

Livingstone, 1997.

9. Herrera D, Sanz M, Jepsen SJ, Needleman IG, Roldan S. A

systematic review on the effect of systemic antimicrobials

as an adjunct to scaling and root planning in perio-

dontitis patients. J Clin Periodontol 2002: 29 (Suppl 3):

136159.

10. Hujoel PP, Drangsholt M, DeRouen TA. Periodontitis

systemic disease associations in the presence of smo-

king causal or coincidental? Periodontol 2000 2002: 30:

5160.

11. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ,

Gavaghan DJ, et al. Assessing the quality of reports of

randomized clinical trials: is blinding necessary? Control

Clin Trials 1996: 17: 112.

12. Juni P, Altman DG, Egger M. Assessing the quality of con-

trolled clinical trials. Br Med J 2001: 323: 4246.

13. Khan KS, ter Riet G, Glanville J, Sowden AJ, Kleijnen J.

Undertaking systematic reviews of research on effective-

ness. Http://www.York.Ac.Uk/Institute/crd/report4.htm [2nd],

2001.

14. Labriola A, Needleman I, Moles DR. Systematic review ofthe effect of smoking on nonsurgical periodontal therapy.

Periodontol 2000 2005: 37: 124137.

15. Mathews JR. Quantification and the Quest for Medical

Certainty. Princeton: Princeton University Press, 1995.

16. Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A,

et al. Assessing the quality of reports of randomized trials:

implications for the conduct of meta-analyses. Health

Technol Assess 1999: 3: 198.

17. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M,

et al. Does quality of reports of randomized trials affect

estimates of intervention efficacy reported in meta-analy-

ses? Lancet 1998: 352: 609613.

18. Montenegro R, Needleman I, Moles D, Tonetti M. Quality

of RCTs in periodontology a systematic review. J Dent Res

2002: 81: 866870.

19. Needleman IG. A guide to systematic reviews. J Clin

Periodontol2002: 29 (Suppl. 3): 69.

20. Needleman IG. Introduction to evidence based dentistry.

In: Clarkson, J, Harrison, JE, Ismail, AI, Needleman, IG,

Worthington, H, eds. Evidence Based Dentistry for Effective

Practice. London: Martin Dunitz, 2003: 117.

21. Needleman IG, Giedrys-Leeper E, Tucker RJ, Worthington

HV. Guided tissue regeneration for periodontal infra-bony

defects (Cochrane Review). The Cochrane Library. Oxford:

Update Software. http://www.update-software.com/clibhome/

clib.htm, 2001.

22. Needleman IG, Giedrys-Leeper E, Tucker RJ, Worthington

HV. Guided tissue regeneration for infrabony defects

A systematic review. J Periodontal Res 2002: 37: 380

388.

23. Rangachari PK. Evidence-based medicine: old French wine

with a new Canadian label? J R Soc Med 1997: 90: 280284.

24. Richards D. Not all evidence is created equal so what is

good evidence? Evid Based Dent 2003: 4: 1718.

25. Roccuzzo M, Bunio M, Needleman I, Sanz M. Periodontal

plastic surgery for treatment of localized gingival reces-

sions A systematic review. J Clin Periodontol 2002: 29

(Suppl 3): 178194.

26. Schultz KF, Chalmers I, Hayes RJ, Altman D. Empirical

evidence of bias: Dimensions of methodological quality

associated with estimates of treatment effects in controlled

trials. JAMA 1995: 273: 408412.

27. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD,

Rennie D, et al. Meta-analysis of observational studies in

epidemiology: a proposal for reporting. Meta-analysis of

Observational Studies in Epidemiology (MOOSE) group.JAMA 2000: 283: 20082012.

28. Swales J. The troublesome search for evidence: three

cultures in need of integration. J R Soc Med 2000: 93: 402

407.

29. Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J.

The development of QUADAS: a tool for the quality

assessment of studies of diagnostic accuracy included in

systematic reviews. BMC Med Res Methodol 2003: 3: 25.

Needleman et al.

12 Evidence Based Period Ontology Systematic Reviews

Documents

Transcript of 12 Evidence Based Period Ontology Systematic Reviews