Does it Matter —

Timeliness or Accuracy of Results?

Results of a Research Program on Rapid Reviews

Andrea C. Tricco PhD MSc

CADTH Symposium 2015

I have no actual or potential conflict of

interest in relation to this presentation.

Conflict of interest

2

Background

Scoping Review of Rapid Reviews

International Survey of Rapid Review Producers

International Consensus-building Exercise with Key

Stakeholders

Ultimate Goal of Our Research Program

Questions

Outline

2

Background

Rapid reviews are a form of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a timely manner [Khangura 2012]

Evidence suggests that decision-makers are currently using rapidreviews to inform their decision-making processes

Few studies have examined the methodological characteristics of rapid reviews

We aimed to conduct a research program on rapid reviews to clarify the methods and perceptions of rapid review approaches

3

A scoping review

of rapid review methodsSubmitted to BMC Medicine

Methods project 1:

Objective & methods

Objective – To examine rapid review approaches, guidance, impact, and comparisons

through a scoping review

Methods– Used methodologically rigorous scoping review methods proposed by

Arksey and O’Malley (2005)

– MEDLINE, EMBASE, the Cochrane Library, internet websites of rapid review

producers, and reference lists were searched to identify articles for inclusion

– Two reviewers independently screened citations and full-text articles, and

abstracted data

5

Results

Study flow figure

N=3397 citations from MEDLINE, EMBASE, Cochrane Library,

previous systematic reviews, and grey literature

N=3135 excluded titles and abstracts

N=100 rapid reviews(plus 1 companion report)

N=262 potentially relevant full-text articles

N=161 excluded full-text

reports

6

Results (continued)

Word cloud figure for the frequency of terms7

Study CharacteristicsNo. of Rapid Reviews

(n=100)

Year of Publication

1997-2000 2

2001-2004 10

2005-2008 30

2009-2012 51

2013 5

Not Reported 4

Corresponding

author’s

continent

Europe (including UK) 58

North America (Canada & United States) 20

Australia 15

Multiple continents 3

Asia 1

South America 1

Not Reported 2

Study characteristics

Results (continued)

8

Results (continued)Study Characteristics No. of Rapid Reviews (n=100)

Article Type

Application (82 with methods) 84

Development 7

Impact 6

Comparison 4

Topic of Review

Intervention 62 (74%)

Frequency 10 (12%)

Causal association 4 (5%)

Diagnosis 4 (5%)

Patient experience 2 (2%)

Screening 2 (2%)

Not applicable 16

Study characteristics (continued)9

Results (continued)

Methods characteristics

4%

2%

83%

70%

51%

22%

12%

17%

35%

25%

10%

17%

23%

98%

12%

24%

10%

11%

68%

49%

54%

46%

68%

57%

73%

6%

6%

40%

67%

20%

34%

17%

29%

22%

26%

0% 20% 40% 60% 80% 100%

Duration of review

Protocol mentioned

Databases Searched

Grey literature search

Scanning references

Contacting authors

Date limit

Language limit

Titles and abstracts screening

Full-texts screening

Data abstraction

Quality appraisal

SR method Streamlined method Not reported

10

Results (continued)

Approach Literature search Search limit Screening Data abstractionRisk of bias

appraisal

1>1 database,

published only

Both date and

languageOne reviewer

One person

abstracts, other

verifies

One person

assesses, other

verifies

2

Updating the literature

search of a previous

review, published only

None One reviewer One reviewer Not performed

3>1 database, grey

literature

Both date and

languageOne reviewer One reviewer Not performed

4>1 database, grey

literature

Either date or

languageOne reviewer One reviewer Not performed

5>1 database, grey

literatureDate One reviewer One reviewer One reviewer

5 most frequent rapid review approaches11

Conclusions

Numerous rapid review approaches were identified

Little consistency exists in the field

Poor quality of reporting was observed

Prospective study comparing the results from rapid reviews to

those obtained through systematic reviews is warranted.

12

An international survey

of rapid review producersSubmitted to the J of Clin Epi

Methods project 2:

Objective and methods

Objective– To determine different rapid review approaches used by rapid review

producers

Methods– International survey of 63 organizations administered via FluidSurvey

– Survey pilot-tested prior to administration

– Reminders to non-respondents sent every 2 weeks

– Contacted through postal mail if no response

– $10 financial incentive from Amazon

14

Results

Study flow figure of participants

63 organizations contacted

41 responses (65%)

22 did not respond

15

Results (continued)

Summary of rapid review characteristics

Review Characteristics Count (%)

Duration of Review

(weeks)

1-12 62 (70)

12-26 18 (20)

26-36 6 (7)

≥ 52 2 (2)

Commissioning Agency

Government Agencies & Health Ministries 69 (78)

Healthcare Organizations, Hospitals & Community Health 51 (58)

Healthcare Professionals 13 (15)

Industry 4 (5)

Target Audience

Government Agencies & Health Ministries 73 (83)

Healthcare Professionals 46 (52)

Patients 19 (22)

Researchers 21 (24)

16

Results (continued)

Rationale provided Count (%)

Decision-maker timeline 57 (66)

Focused or brief question 8 (9)

Lack of resources 5 (6)

Increase efficiency (including timeliness) 4 (5)

Broad understanding of an area 4 (5)

Identify topics requiring a systematic review 2 (2)

Update a systematic review 2 (2)

Well-established intervention 1 (1)

Evidence is unclear 1 (1)

Rationale for conducting rapid reviews17

Results (continued)Review Stage Most frequent streamlined approach Count (%)

Identifying relevant studies Used previous review(s) as a starting point 79 (92)

Limitations on search strategy Limited review by date of publication 75 (88)

Study selection Screening conducted by ONE reviewer only 68 (85)

Data Abstraction Data abstraction performed by ONE reviewer only 67 (84)

Quality (risk of bias) appraisal process

Risk of bias assessed by ONE reviewer only 68 (86)

Synthesis Narrative summary 75 (90)

Summary results of most frequently streamlined approaches 18

Conclusions

Results are consistent with scoping review of rapid reviews

Rapid reviews usually conducted in 1-12 weeks

Government agencies and health ministries are primary

commissioners

Many different streamlined methods are being used.

19

International consensus-building

exercise regarding rapid reviewsSubmitted to the J of Clin Epi

Methods project 3:

Objective & methods

Objective – To conduct a consensus-building exercise to select a rapid review approach

that will be prospectively tested in a diagnostic study

Methods– Editors, healthcare providers, researchers, policy-makers, and industry

stakeholders (including participants of the CADTH rapid review summit)

– Asked to rank the 5 most frequent rapid review approaches identified in our

scoping review and survey using FluidSurvey

– Results presented to participants, followed by a facilitated discussion (online

and in-person) and re-ranking exercise using FluidSurvey

21

Results

Study flow figure of participants

26 individuals

contacted

113 responses (72%)

3 did not respond

130 individuals

contacted

40 did not respond

22

Online delphi In-person delphi

Summary of ranking results by approach

Results (2)Rapid review approach

Feasibility Timeliness Comprehensiveness Risk of Bias

Approach 1 1st 2nd 4th 1st

Approach 2 2nd 1st 5th 5th

Approach 3 3rd 3rd 3rd 2nd

Approach 4 4th 4th 2nd 4th

Approach 5 5th 5th 1st 3rd

*Ranked based on the distribution of "very" and "extremely" on the 7-point Likert scale, except Risk of Bias was ranked on distribution of “not at all” and “very”

23

The highest ranked method was: Approach 1

‒ Most feasible (72%, n=81 out of 113 responses)

‒ Lowest perceived risk of bias (12%, n=12 out of 103)

‒ 2nd in timeliness (37%, n=38 out of 102)

‒ 5th in comprehensiveness (5%, n=5 out of 100)

We will use the information from the consensus-building exercise

to select the rapid review approach for a prospective study.

Conclusion

24

Ultimate goal of this research

“We can give you results within 4 months, but the meta-

analysis estimates will be inaccurate by 35%”

Rapid review definition (Shannon Kelly)

Identify 5 frequently used methods

Diagnostic study to test a rapid review approach

Identify and characterize rapid review methods

25

Will use these results from our research program to inform a

diagnostic study:

− Index test: Rapid Review Approach

− Reference standard: Systematic Review

Collaboration between 3 Canadian Knowledge Synthesis Centers

Targeting CIHR and PCORI

Proposed diagnostic study

Diagnostic Accuracy of Rapid reviews compared To

Systematic reviews (DARTS)

26

Summary

This research program provides up-to-date information on rapid

review methods reported in the literature, as well as stakeholder

experiences and perceptions regarding rapid reviews

Poor quality of reporting was observed in the literature

Rapid reviews have many names and approaches and some

methods might be more desirable than others

A prospective study comparing the results of rapid reviews to those

obtained through systematic reviews is necessary.

27

Acknowledgements

Funding: Canadian Institutes of Health Research/Drug Safety

and Effectiveness Network

− Operating grant to update 2 systematic reviews,

international survey, Delphi

− New investigator award

Research team: Jesmin Antony, Wasifa Zarin

Co-investigators: Drs. Straus, Moher, Hutton, Sherifali

28

Questions?

triccoa@smh.ca

30