Does it Matter — Timeliness or Accuracy of Results?

Results of a Research Program on Rapid Reviews

Andrea C. Tricco PhD MSc

CADTH Symposium 2015

I have no actual or potential conflict of interest in relation to this presentation.

Conflict of interest

2

Background Scoping Review of Rapid Reviews International Survey of Rapid Review Producers International Consensus-building Exercise with Key

Stakeholders Ultimate Goal of Our Research Program Questions

Outline

2

Background

Rapid reviews are a form of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a timely manner [Khangura 2012]

Evidence suggests that decision-makers are currently using rapid reviews to inform their decision-making processes

Few studies have examined the methodological characteristics of rapid reviews

We aimed to conduct a research program on rapid reviews to clarify the methods and perceptions of rapid review approaches3

A scoping review of rapid review methodsSubmitted to BMC Medicine

Methods project 1:

Objective & methods Objective

– To examine rapid review approaches, guidance, impact, and comparisons through a

scoping review

Methods– Used methodologically rigorous scoping review methods proposed by Arksey and

O’Malley (2005)– MEDLINE, EMBASE, the Cochrane Library, internet websites of rapid review

producers, and reference lists were searched to identify articles for inclusion– Two reviewers independently screened citations and full-text articles, and abstracted

data

5

Results

Study flow figure

N=3397 citations from MEDLINE, EMBASE, Cochrane Library, previous systematic reviews, and grey literature

N=3135 excluded titles and abstracts

N=100 rapid reviews(plus 1 companion report)

N=262 potentially relevant full-text articles

N=161 excluded full-text reports

6

Results (continued)

Word cloud figure for the frequency of terms7

Study CharacteristicsNo. of Rapid Reviews

(n=100)

Year of Publication 

1997-2000 2

2001-2004 10

2005-2008 302009-2012 51

2013 5

Not Reported 4

Corresponding author’s continent

Europe (including UK) 58

North America (Canada & United States) 20

Australia 15

Multiple continents 3

Asia 1

South America 1

Not Reported 2

Study characteristics

Results (continued)

8

Results (continued)Study Characteristics No. of Rapid Reviews (n=100)

 Article Type

Application (82 with methods) 84

Development 7

Impact 6

Comparison 4

Topic of Review

Intervention 62 (74%)

Frequency 10 (12%)

Causal association 4 (5%)

Diagnosis 4 (5%)

Patient experience 2 (2%)

Screening 2 (2%)

Not applicable 16

Study characteristics (continued)9

Results (continued)

Methods characteristics

Duration of review

Protocol mentioned

Databases Searched

Grey literature search

Scanning references

Contacting authors

Date limit

Language limit

Titles and abstracts screening

Full-texts screening

Data abstraction

Quality appraisal

0% 20% 40% 60% 80% 100%

4%

2%

83%

70%

51%

22%

12%

17%

35%

25%

10%

17%

23%

98%

12%

24%

10%

11%

68%

49%

54%

46%

68%

57%

73%

6%

6%

40%

67%

20%

34%

17%

29%

22%

26%

SR method Streamlined method Not reported

10

Results (continued)

Approach Literature search Search limit Screening Data abstractionRisk of bias

appraisal

1>1 database,

published onlyBoth date and

languageOne reviewer

One person abstracts, other

verifies

One person assesses, other

verifies

2Updating the literature search of a previous

review, published onlyNone One reviewer One reviewer Not performed

3>1 database, grey

literatureBoth date and

languageOne reviewer One reviewer Not performed

4>1 database, grey

literatureEither date or

languageOne reviewer One reviewer Not performed

5>1 database, grey

literatureDate One reviewer One reviewer One reviewer

5 most frequent rapid review approaches11

Conclusions Numerous rapid review approaches were identified

Little consistency exists in the field

Poor quality of reporting was observed

Prospective study comparing the results from rapid reviews to those obtained through systematic reviews is warranted.

12

An international survey of rapid review producersSubmitted to the J of Clin Epi

Methods project 2:

Objective and methods Objective

– To determine different rapid review approaches used by rapid review producers

Methods– International survey of 63 organizations administered via FluidSurvey– Survey pilot-tested prior to administration– Reminders to non-respondents sent every 2 weeks– Contacted through postal mail if no response– $10 financial incentive from Amazon

14

Results

Study flow figure of participants

63 organizations contacted

41 responses (65%)

22 did not respond

15

Results (continued)

Summary of rapid review characteristics

Review Characteristics Count (%)

Duration of Review (weeks) 

1-12 62 (70)

12-26 18 (20)

26-36 6 (7)

≥ 52 2 (2)

Commissioning Agency 

Government Agencies & Health Ministries 69 (78)

Healthcare Organizations, Hospitals & Community Health 51 (58)

Healthcare Professionals 13 (15)

Industry 4 (5)

Target Audience 

Government Agencies & Health Ministries 73 (83)

Healthcare Professionals 46 (52)

Patients 19 (22)

Researchers 21 (24)

16

Results (continued)

Rationale provided Count (%)Decision-maker timeline 57 (66)

Focused or brief question 8 (9)

Lack of resources 5 (6)

Increase efficiency (including timeliness) 4 (5)

Broad understanding of an area 4 (5)

Identify topics requiring a systematic review  2 (2)

Update a systematic review 2 (2)

Well-established intervention 1 (1)

Evidence is unclear 1 (1)

Rationale for conducting rapid reviews17

Results (continued)Review Stage Most frequent streamlined approach Count (%)

Identifying relevant studies Used previous review(s) as a starting point 79 (92)

Limitations on search strategy Limited review by date of publication 75 (88)

Study selection Screening conducted by ONE reviewer only 68 (85)

Data Abstraction Data abstraction performed by ONE reviewer only 67 (84)

Quality (risk of bias) appraisal process Risk of bias assessed by ONE reviewer only 68 (86)

Synthesis Narrative summary 75 (90)

Summary results of most frequently streamlined approaches 18

Conclusions Results are consistent with scoping review of rapid reviews

Rapid reviews usually conducted in 1-12 weeks

Government agencies and health ministries are primary commissioners

Many different streamlined methods are being used.

19

International consensus-building exercise regarding rapid reviewsSubmitted to the J of Clin Epi

Methods project 3:

Objective & methods Objective

– To conduct a consensus-building exercise to select a rapid review approach that will be prospectively tested in a diagnostic study

Methods– Editors, healthcare providers, researchers, policy-makers, and industry

stakeholders (including participants of the CADTH rapid review summit)– Asked to rank the 5 most frequent rapid review approaches identified in our

scoping review and survey using FluidSurvey– Results presented to participants, followed by a facilitated discussion (online

and in-person) and re-ranking exercise using FluidSurvey

21

Results

Study flow figure of participants

26 individuals contacted

113 responses (72%)

3 did not respond

130 individuals contacted

40 did not respond

22

Online delphi In-person delphi

Summary of ranking results by approach

Results (2)Rapid review approach Feasibility Timeliness Comprehensiveness Risk of Bias

Approach 1 1st 2nd 4th 1st

Approach 2 2nd 1st 5th 5th

Approach 3 3rd 3rd 3rd 2nd

Approach 4 4th 4th 2nd 4th

Approach 5 5th 5th 1st 3rd

*Ranked based on the distribution of "very" and "extremely" on the 7-point Likert scale, except Risk of Bias was ranked on distribution of “not at all” and “very”

23

The highest ranked method was: Approach 1

‒ Most feasible (72%, n=81 out of 113 responses)

‒ Lowest perceived risk of bias (12%, n=12 out of 103)

‒ 2nd in timeliness (37%, n=38 out of 102)

‒ 5th in comprehensiveness (5%, n=5 out of 100)

We will use the information from the consensus-building exercise to select the rapid review approach for a prospective study.

Conclusion

24

Ultimate goal of this research

“We can give you results within 4 months, but the meta-analysis estimates will be inaccurate by 35%”

Rapid review definition (Shannon Kelly)

Identify 5 frequently used methods

Diagnostic study to test a rapid review approach

Identify and characterize rapid review methods

25

Will use these results from our research program to inform a diagnostic study:

− Index test: Rapid Review Approach − Reference standard: Systematic Review

Collaboration between 3 Canadian Knowledge Synthesis Centers Targeting CIHR and PCORI

Proposed diagnostic study

Diagnostic Accuracy of Rapid reviews compared To Systematic reviews (DARTS)

26

Summary This research program provides up-to-date information on rapid

review methods reported in the literature, as well as stakeholder experiences and perceptions regarding rapid reviews

Poor quality of reporting was observed in the literature

Rapid reviews have many names and approaches and some methods might be more desirable than others

A prospective study comparing the results of rapid reviews to those obtained through systematic reviews is necessary.

27

Acknowledgements Funding: Canadian Institutes of Health Research/Drug Safety

and Effectiveness Network− Operating grant to update 2 systematic reviews,

international survey, Delphi− New investigator award

Research team: Jesmin Antony, Wasifa Zarin

Co-investigators: Drs. Straus, Moher, Hutton, Sherifali

28

Questions?

triccoa@smh.ca

30