Can J Gastroenterol Vol 16 No 1 January 2002 17

ORIGINAL ARTICLE

A randomized survey of thepreference of

gastroenterologists for aCochrane review versus atraditional narrative review

John WD McDonald MD PhD, Jeffrey Mahon MD, Kelly Zarnke MD, Brian Feagan MD,Lorinda Simms MSc, Wayne Tucker MA

University of Western Ontario and London Health Sciences Centre, London, OntarioCorrespondence and reprints: Dr John WD McDonald, Room 5OF12, University Campus, London Health Sciences Centre, London, Ontario

N6A 5A5. Telephone 519-663-3550, fax 519-663-3232, e-mail john.mcdonald@lhsc.on.caReceived for publication June 18, 2001. Accepted September 24, 2001

JWD McDonald, J Mahon, K Zarnke, B Feagan, L Simms,W Tucker. A randomized survey of the preference ofgastroenterologists for a Cochrane review versus a traditionalnarrative review. Can J Gastroenterol 2002;16(1):17-21.

BACKGROUND: Clinicians often rely on review materialrather than analysis of primary research to guide therapy.Systematic reviews use methods to insure thoroughness and tominimize bias, but many clinicians are not familiar with system-atic reviews and continue to rely on narrative reviews.OBJECTIVES: To determine whether a traditional narrativereview or a systematic review is perceived to be more useful.METHODS: A clinical scenario (patient with chronic Crohn’sdisease considered for azathioprine therapy) was circulated togastroenterologists, along with a narrative review of therapy(including azathioprine) for inflammatory bowel disease writtenby an acknowledged expert, or with a systematic Cochrane reviewof the use of azathioprine for this disease. Whether knowledge ofauthorship and journal source influences the perception of use-fulness of a narrative review was investigated.RESULTS: The Cochrane review was rated significantly morehighly than the narrative review on a 100 mm visual analoguescale (21.3 mm; 95% CI 14.5 to 28 mm). The proportion ofrespondents who considered the review to be a useful guide wasalso higher in the group that received the Cochrane review(91%) than in the group that received the narrative review, withauthor and journal concealed (62%) or identified (70%)(P<0.001 for both comparisons). Ratings from the two groups that

received the narrative review were not significantly different.CONCLUSIONS: The focused systematic review was per-ceived to be more useful than a traditional broad narrativereview as a guide to making a decision concerning the use of spe-cific therapy. The possible strengths of systematic reviews shouldbe more fully investigated. If there is additional evidence sup-porting their greater value to clinicians, they should be mademore widely available to clinicians and their strengths should bepublicized.

Key Words: Cochrane review; Crohn’s disease; Physicianbehaviour; Review; Therapy

Enquête avec hasardisation sur la préférencedes gastro-entérologues à l’égard d’une synthèse Cochrane par rapport à une synthèsenarrative classiqueCONTEXTE : Les cliniciens se fient souvent aux synthèses plutôt qu’àl’analyse des résultats directs de la recherche pour orienter les traite-ments. Les examens systématiques font appel à des méthodes qui assurentla rigueur et réduisent les biais au minimum, mais beaucoup de cliniciensne connaissent pas vraiment les examens systématiques et continuent dese fier aux examens narratifs.OBJECTIF : Vérifier quelle forme d’examen : narrative (classique) ousystématique, est perçue comme la plus utile.

voir page suivante

Clinicians frequently rely on reviews of primary researchfor guidance in making decisions about therapy.

Review articles of health care interventions take two mainforms – traditional narrative articles and systematic reviews.The merits of each have been discussed (1-3). Cook et al(4) pointed out that systematic reviews focus on the effectsof a specific intervention, while narrative reviews can havebroader purposes. Systematic reviews, including reviewspublished in The Cochrane Library (1996 to 2000, UpdateSoftware Ltd, Oxford, United Kingdom), attempt to avoidthe weaknesses of traditional narrative reviews, such asbiased recommendations and failure to consider all relevantevidence (5,6). Reviews published in The Cochrane Libraryare also maintained with regular updates. At least some sys-tematic reviews appear to have had a powerful impact onmedical practice (7,8).

Many clinicians continue to rely on traditional narrativereviews published in conventional journals, possibly due toa lack of familiarity with or availability of systematicreviews, including Cochrane reviews, or to a preference forselecting and consulting a narrative review written by anacknowledged expert and published in a conventional med-ical journal. The preference of clinicians between the twotypes of reviews has not been studied. Therefore, theresponses of Canadian gastroenterologists to two reviewsregarding the use of azathioprine for chronic Crohn’s dis-ease – a systematic Cochrane review (9) and a traditionalnarrative review (10) – were compared. A previous study(11) found that an unexpectedly small fraction of patientswith chronic Crohn’s disease had ever received azathio-prine therapy, despite data that indicate its usefulness (12).The primary objective of the present study was to deter-mine whether the systematic Cochrane review was per-ceived as being more useful than the narrative review forguiding a decision regarding therapy. The secondary objec-tive was to determine whether identification of authorshipand journal source influences the rating by physicians of theusefulness of a traditional review.

MATERIALS AND METHODSPotential participants were practising gastroenterologistsidentified from the Canadian Association of Gastro-enterology membership list. It is estimated that approxi-

mately two-thirds of Canadian Gastroenterologists aremembers of this association. When a questionnaire wasmailed in error to a person not in practice or was returnedonly partially completed, it was excluded from the analysis.The names of potentially eligible persons were listed alpha-betically and numbered sequentially. By using a randomnumber generator, these persons were divided into threegroups and selected in a 1:1:1 ratio to receive:

• the Cochrane review (not identified as a Cochranereview, and with authors concealed) of the use ofazathioprine for induction of remission in Crohn’sdisease (9);

• the traditional narrative review of therapy (includingazathioprine) for inflammatory bowel disease (10),with the journal source and author concealed; or

• the same narrative review but with journal sourceand authorship identified.

A letter of encouragement to participate was sent by theexecutive director of the Crohn’s and Colitis Foundation ofCanada a week before the mailing, but no incentives wereoffered to complete the questionnaire. Confidential codedquestionnaires were mailed to 423 eligible persons(Tables 1,2). A follow-up mailing was sent to nonrespon-dents two months later. The allocation code was main-tained in a computer accessed only by the individual whoperformed the mailings and later by the person who per-formed the statistical analysis, and was concealed from allother investigators until the primary analysis had been per-formed.

To identify a narrative review for comparison with theCochrane review, a MEDLINE search restricted to Englishlanguage publications for the years 1996 to the preparation

McDonald et al

Can J Gastroenterol Vol 16 No 1 January 200218

MÉTHODE : On a envoyé à des gastro-entérologues un scénario clin-ique (patient atteint de la forme chronique de la maladie de Crohn, can-didat au traitement à l’azathioprine) accompagné d’un examen narratifdu traitement (dont à l’azathioprine) des maladies inflammatoires de l’in-testin, rédigé par un expert reconnu ou d’un examen systématiqueCochrane de l’utilisation de l’azathioprine dans ce contexte. On a égale-ment vérifié si le fait de connaître le nom de l’auteur et de la revue avaiteu une incidence sur la perception de l’utilité de l’examen narratif.RÉSULTATS : Les participants ont accordé une cote significativementplus élevée à la synthèse Cochrane qu’à la synthèse narrative sur uneéchelle visuelle analogue de 100 mm (21,3 mm; IC à 95 % : 14,5 à28 mm). De plus, la proportion de répondants qui ont jugé l’examen utile

a également été plus élevée dans le groupe qui avait reçu la synthèseCochrane (91 %) que dans le groupe qui avait reçu la synthèse narrative,peu importe que le nom de l’auteur et de la revue ait été dévoilé (70 %)ou non (62 %) (P<0,001 pour les deux comparaisons). L’évaluation del’examen narratif différait peu entre les deux groupes. CONCLUSION : L’examen systématique ciblé a été perçu comme plusutile que l’examen narratif classique général pour aider à la prise de déci-sion concernant le recours à un certain traitement. Les points forts possi-bles des examens systématiques devraient faire l’objet de plus derecherche. Si des données supplémentaires devaient en confirmer lasupériorité auprès des cliniciens, il faudrait offrir plus de synthèses cibléesaux intéressés et faire connaître leurs avantages.

TABLE 1Response to survey

Surveys mailed 591

Duplicates, incorrect address, not in practice 168

Mailed to eligible recipients 423

Surveys returned 264 (62.4%)

Surveys with outcome data 256 (60.5%)

date of the survey material in 1998 was conducted using theterms ‘Crohn’, ‘drug therapy’ and ‘review’. The initialsearch yielded 56 items. However, perusal of the titlesrevealed that only 12 were relevant. The remaining titleswere then examined to determine which was most likely tobe selected by gastroenterologists seeking information inthe ‘real world’. Six of the 11 titles were eliminated becausethey had been published in journals that the investigatorsconsidered to be unlikely to be consulted by the gastroen-terologists who were potential participants in the study.The remaining six reviews were examined; one was identi-fied by consensus of the investigators as most likely to beselected by practising gastroenterologists because it waspublished in a highly respected, universally available jour-nal and was written by an acknowledged authority. Theextent to which it dealt with azathioprine was similar to orgreater than that in any of the narrative reviews that wereeliminated on the grounds of authorship or journal source.The selected review also met the needs of the study becauseits length approximated that of the Cochrane review; con-siderable differences in length may have led to a differencein response rates for the two types of reviews or to biasedresponses.

Clinical scenarioThe following clinical scenario was presented to partici-pants as a focus for evaluation of the usefulness of thereviews.

A 32-year-old male patient has had Crohn’s disease forthe past four years. He has been treated with intermit-tent tapered courses of prednisone beginning at 40 mgdaily. In the past year, he has been on prednisoneapproximately half the time and has felt well with thistreatment, but his symptoms recur when the dose isreduced below 20 mg daily.

OutcomesThe primary outcome measure was the respondent’s ratingof the usefulness of the review material for guiding a deci-sion about whether to recommend azathioprine for thepatient described in the clinical scenario. The measure-ment tool was a 100 mm visual analogue scale (VAS) rang-ing from 0 (completely useless) to 100 (extremely useful).Pretesting the VAS with a sample of 10 clinicians estab-lished the variance on the VAS response and that differ-ences in responses of less than 10 mm would not beconsidered clinically important. Participants were asked toindicate whether the review was useful as a guide to therapyand whether they believed that azathioprine was indicatedfor the patient described in the case scenario by answeringdirect ‘yes’ or ‘no’ questions.

MaskingAll recipients of reviews were advised in a covering letterthat two types of review material were being compared butwere not told that the comparison involved traditional nar-

rative and Cochrane systematic reviews. The reviews werecopied using a digital scanner, and then formatted andreprinted so that they were of similar appearance andlength (Cochrane review approximately eight pages of text,four pages of references and five pages of tables; narrativereview approximately seven pages of text, five pages of ref-erences and four pages of tables. The reviews prepared inthis format are available on request.)

Statistical analysisAfter accounting for nonresponders (assumed to be 50%), asample of 378 clinicians was required to test the primaryhypothesis at 80% power (two-tailed alpha error 5%) thatthere would be no more than a 10% difference in the per-ception of the usefulness of the reviews. Continuous data(VAS) were analyzed by ANOVA using SPSS (SPSS Inc,USA). The differences between mean values were com-pared in a post hoc analysis using Tukey’s test for multiplecomparisons. Dichotomous data were analyzed by χ2 analy-sis. The possible influence of demographic characteristics ofrespondents was assessed using logistic regression analysis.

RESULTSSurveys were initially mailed to all people on the member-ship list of the Canadian Association of Gastroenterology.However, a considerable number of people were listed morethan once with more than one address, could not be locatedbecause of incorrect addresses, or were not in practice. Theresponse rate and characteristics of respondents are shownin Tables 1 and 2, respectively. The overall response ratefrom eligible recipients was 62.4%. The individual responserates from recipients of each of the three types of reviewswere very similar. Of the original 591 targeted names,responses were eventually received from 86 (32.6%) recipi-

Cochrane versus narrative review

Can J Gastroenterol Vol 16 No 1 January 2002 19

TABLE 2Characteristics of respondents

Narrative review Narrative review Cochrane review(authorship and (authorship and (authorship andjournal source journal source journal source

concealed) identified) concealed)n=86 n=92 n=86

Years since 25.7±11.2 24.4±10.8 23.6±10.6graduation(mean ± SD)

Number of CD 6.8±5.2 7.8±6.1 8.2±6.0patients seen weekly(mean ± SD)

No training 26 19 22in critical appraisal (%)

University-based 51 50 57practice (%)

CD Crohn’s disease

ents of the traditional ‘concealed’ review, 92 (33.7%) recip-ients of the traditional ‘identified’ review and 86 (32.6%)recipients of the Cochrane review. There was no evidenceof bias between early (86% of total) and late responderswhen compared by ANOVA. The mean score for earlyrespondents (mailing 1) was 59.6. The mean score for lateresponders was 56.0. ANOVA showed no statistically sig-nificant difference (F=1.019; P<0.314). Demographic dataon the nonresponders were not obtained; therefore, thepossibility of selection bias influencing the results cannotbe excluded completely. However, the randomizationprocess and the similarity of response rates from recipientsof the three types of reviews are not consistent with a biasedresponse. There were no significant differences among thegroups with respect to variables that could have been asso-ciated with a preference, including age, practice setting andtraining in critical appraisal of the medical literature.

The rating of usefulness assigned by clinicians whoreceived the Cochrane review was significantly higher thanthe rating assigned by those who received the narrativereview in either format (Figure 1). The difference betweenmean ratings of the Cochrane review and the narrativereview with authorship and journal source concealed was21.3 mm (95% CI 14.5 to 28 mm) on the 100 mm VAS.The difference between mean ratings of the Cochranereview and the narrative review with authorship and jour-nal source identified was 16.6 mm (95% CI 8.6 to24.5 mm). Regression analysis did not show that the higherrating assigned by recipients of the Cochrane review wasinfluenced by training in critical appraisal of medical litera-ture, more recent graduation from medical school or prac-tice location (community versus university based).

A significantly higher proportion of clinicians whoreceived the Cochrane review considered it to be a usefulguide to whether to use azathioprine therapy comparedwith the two groups who received the narrative review

(Table 3). Concealment or identification of authors andjournal source did not significantly affect the evaluation ofthe narrative review, although there was a trend towardboth a higher rating for the identified review and a higherproportion of clinicians responding that the review was use-ful. There was no difference in the proportion of respon-dents who stated that azathioprine was indicated for thepatient described in the clinical scenario (narrative reviewwith author concealed 97.5%, traditional review withauthor identified 97.8%, Cochrane review 98.8%).

DISCUSSIONOur results show that a systematic Cochrane review wasconsidered by Canadian gastroenterologists to be more use-ful than a traditional narrative review as a guide to specifictherapeutic decision making. The latter is clearly the pur-pose of Cochrane reviews, and these reviews are focusedand structured to provide this type of information. In ourstudy, the question posed by the clinical scenario wassharply focused, and the Cochrane review by its very natureis also focused, while the narrative review selected for com-parison is much broader in its scope. This difference is asource of potential bias toward the Cochrane review andmay lead to questions about the fairness of the comparison.However, other characteristics of the narrative review, suchas the perspective it may offer, the more familiar format orother considerations, could result in a bias in the oppositedirection. Whatever biases exist in favour of a Cochrane ornarrative review, the latter remains the established alterna-tive to a Cochrane review. It is, therefore, reasonable thatthe narrative review be the benchmark against which tocompare the value of a Cochrane review. We located a rel-evant narrative review that we thought was most likely tobe consulted in the ‘real world’. This comparison is analo-gous to a randomized trial comparing a new interventionwith a traditional therapy for Crohn’s disease. TheCochrane review may be considered to be analogous to amonoclonal antibody specifically engineered for possibleuse in inflammatory bowel disease, while the narrative

McDonald et al

Can J Gastroenterol Vol 16 No 1 January 200220

Figure 1) Rating of usefulness of reviews on 100 mm visual analoguescale. Higher scores indicate greater usefulness. Columns are mean val-ues assigned by groups receiving the specified reviews. Error bars areone standard error of the mean

TABLE 3Proportion of respondents considering the review tobe a useful guide for choosing therapy

Traditional Traditional Cochranenarrative review narrative review review

(authorship (authorship (authorshipconcealed) identified) concealed) Total

Useful guide

Yes 51 (62%) 64 (70%) 76 (91%) 191

No 31 27 7 65

Total 82 91 83 256

χ2 tests: three groups χ2=20.15, P<0.001; traditional narrative review (authorsand journal concealed) versus Cochrane review χ2=20.07, P<0.001; tradi-tional narrative review (authors and journal identified) versus Cochrane reviewχ2=12.45, P<0.001; traditional narrative review, authors and journal concealedversus identified χ2=1.28, P=0.26

review may be analogous to conventional steroid therapy.The narrative review we chose was judged most likely to beconsulted by gastroenterologists. Accordingly, the studymay be considered to be an appropriate comparison of a‘real life’, widely used intervention with a new interven-tion. We specifically asked respondents to evaluate the use-fulness of the reviews as a guide to therapeutic decisionmaking and have no information on the value they wouldplace on the narrative review for another purpose, such asmaintaining a general fund of knowledge.

Survey research using a questionnaire has inherentweaknesses. However, the response rate in our study wasrelatively high, and it appears that responders were broadlyrepresentative of gastroenterologists practising in Canada.The validity of the observed difference in rating of thereviews is strengthened by the study design. The probabilityof biased responses was reduced by the use of random allo-cation and by the provision of only a single type of reviewto each participant. Participants were unaware of the natureof the other type of review that was being evaluated. Carewas taken to conceal the origin of the reviews, and theywere formatted to ensure that they were comparable inappearance and length.

The difference between the ratings of the ‘identified’ and‘concealed’ traditional narrative reviews showed a trendtoward higher rating of the former, although this differencewas not statistically significant. Whether identification ofauthorship and journal source introduces bias into the per-ception of utility requires a study larger than ours, whichdid not have adequate statistical power to identify orexclude differences as small as those observed.

Although the generalizability of our results requires fur-ther study, we believe that they may be widely applicablebecause the responses were not dependent on age, site ofpractice or training in critical appraisal of medical literature.

Over 97% of participants believed that azathioprine wasindicated for the patient with chronic Crohn’s diseasedescribed in the case scenario; this result contrasts with pre-vious information concerning actual prescribing practices.When evaluating patients with chronic Crohn’s disease forinclusion in a clinical trial of methotrexate therapy, wefound that an unexpectedly small fraction of patients withchronic disease had ever received azathioprine (11).Although we did not address the question of whetherCochrane reviews have a greater power than narrativereviews to influence prescribing practices, this question isclearly relevant and should be explored in a future study.There is evidence that circulation of printed materials as asole strategy has little power to influence prescribing prac-tices (13). However, the availability to clinicians of com-plete, current and concise information is likely to be onecritical component of achieving optimal prescribing prac-tices. Perhaps Cochrane reviews used in conjunction withother strategies, such as presentations by opinion leaders(13), will prove to be effective. Our study indicates thatCochrane reviews may be preferred over conventional nar-rative material for this purpose. We believe that Cochranereviews should be more widely available to practising clini-cians, and that the availability and the strengths of thesereviews should be more widely publicized. We have pro-vided evidence that they are likely to be welcomed by clini-cians seeking advice for clinical decision making andpreferred over traditional narrative review materials for thispurpose.

ACKNOWLEDGEMENTS: Funding was provided by TheCrohn’s and Colitis Foundation of Canada and the LondonHealth Sciences Centre. Dr McDonald is coordinating editor ofthe Cochrane Inflammatory Bowel Diseases Review Group andreceives no financial compensation for this work.

Cochrane versus narrative review

Can J Gastroenterol Vol 16 No 1 January 2002 21

1. Bailar J. The promise and problems of meta-analysis. N Engl J Med1997;337:559-61.

2. Jefferson T. What are the benefits of editorials and non-systematicreviews? BMJ 1999;318:135.

3. Horton R. The information wars. Lancet 1999;353:164-5. 4. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: Synthesis of

best evidence for clinical decisions. Ann Intern Med 1997;126:376-80.5. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC.

A comparison of results of meta-analyses of randomized control trialsand recommendations of clinical experts. Treatments for myocardialinfarction. JAMA 1992;268:240-8.

6. Ravnskov U. Cholesterol lowering trials in coronary heart disease:frequency of citation and outcome. BMJ 1992;305:15-9.

7. Petros A, Schindler M, Pierce C, et al. Human albuminadministration in critically ill patients. BMJ 1998;317:882-6.

8. Cochrane Injuries Group Albumin Reviewers. Human albumin

administration in critically ill patients: systematic review ofrandomised controlled trials. BMJ 1998;317:235-40.

9. Sandborn W, Sutherland LR, Pearson D, et al. Azathioprine or 6MPfor inducing remission of active Crohn’s disease. Cochrane DatabaseSyst Rev 1998;1.

10. Hanauer SB. Drug therapy. Inflammatory bowel disease. N Engl JMed 1996;334:841-8.

11. Feagan BG, Vreeland MG, Larson LR, Bala MV. Annual cost of carefor Crohn’s disease: a payor perspective. Am J Gastroenterol2000;95:1955-60.

12 Pearson DC, May GR, Fick GH, Sutherland LR. Azathioprine and 6-mercaptopurine in Crohn’s disease: a meta-analysis. Ann InternMed 1995;123:132-42.

13. Freemantle EL, Wolf F, Grimshaw JM, Grilli R, Bero LA. Printededucational materials: effects on professional practice and health careoutcomes. Cochrane Database Syst Rev 2000;2.

REFERENCES

Submit your manuscripts athttp://www.hindawi.com

Stem CellsInternational

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

MEDIATORSINFLAMMATION

of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Behavioural Neurology

EndocrinologyInternational Journal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Disease Markers

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

BioMed Research International

OncologyJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Oxidative Medicine and Cellular Longevity

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

PPAR Research

The Scientific World JournalHindawi Publishing Corporation http://www.hindawi.com Volume 2014

Immunology ResearchHindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Journal of

ObesityJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Computational and Mathematical Methods in Medicine

OphthalmologyJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Diabetes ResearchJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Research and TreatmentAIDS

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Gastroenterology Research and Practice

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Parkinson’s Disease

Evidence-Based Complementary and Alternative Medicine

Volume 2014Hindawi Publishing Corporationhttp://www.hindawi.com