Theme 5. Cochrane Reviews: updating
-
Upload
cade-newman -
Category
Documents
-
view
35 -
download
1
description
Transcript of Theme 5. Cochrane Reviews: updating
Theme 5. Cochrane Reviews: updating
• 9:30 – 10:10 Theme overview and CRG experiences of using tools/frameworks• 10:10 – 10:30: Q&A• 10:30 – 11:00 Working groups• 11:00 – 11:15 Feedback
Overview
1. How often should Cochrane Reviews be updated?
2. Which Cochrane Reviews should be a priority for updating?
3. How should updates of Cochrane Reviews, and Cochrane Reviews no longer being updated, be classified in The Cochrane Library?
Updating frequency
• CRGs are not able to update all Cochrane Reviews every two years. The reasoning behind updating decisions is not transparent to readers.
• The burden of updating is ever increasing• Cochrane Reviews may become out of date at different times• Updating too soon may introduce bias• 80% of 617 respondents moderately or strongly supported a
move from the current policy of aiming to update all Cochrane Reviews every two years to a ‘needs-based’ approach
Prioritising Cochrane Reviews
Prioritising Cochrane Reviews
Classification framework
Experiences
Criteria area Question Yes(=1)/No(=0) Strategic importance Is condition of strategic important to MSK Group or funders or users 0/1 Is intervention of strategic important to MSK Group or funders or users 0/1 Patient importance/ impact Does review measure patient important outcomes? i.e. clinical outcomes 0/1 Does intervention have a high potential impact on quality of life? 0/1 Does intervention potentially significantly lower mortality 0/1 Leading cause of burden of disease in Europe (WHO burden of disease DALY list) 0/1 National Spend High spend on condition (including societal costs)? 0/1 High spend on intervention or on currently used alternatives? 0/1 Presence on list of 20 drugs which had the greatest number of items dispensed, OR greatest Net
ingredient cost 0/1 Incidence prevalence High incidence or prevalence? 0/1 Recent increase in incidence/prevalence? 0/1
Emerging evidence/remaining uncertainty Current review conclusions reflect uncertainty? 0/1 Significant RCTs/other eligible trials published since last update? 0/1 SRs/HTAs covering same ground NOT published (or about to be published) since last update? 0/1 Additional Cochrane criteria Leading cause of burden of disease in the rest of the world i.e. outside of Europe (WHO burden of
disease DALY list) 0/1 Review content out of date 0/1 Review has attracted > 10 citations 0/1 Review in top 10 most accessed MSK reviews 0/1 Review addresses important equity or Global Health Issues 0/1 Author available for update 0/1 Total score: 0
Original criteria
Question
ConditionBurden of disease, spend, and incidence/prevalence related to the condition
Is condition of strategic importance to CCDAN Group or funders or usersLeading cause of burden of disease in Europe (WHO burden of disease DALY list)Leading cause of burden of disease in the rest of the world i.e. outside of Europe (WHO burden of disease DALY list)High spend on condition (including societal costs)?High incidence or prevalence?Recent increase in incidence/prevalence?
Evidence Emerging evidence / remaining uncertainty
Current review conclusions reflect uncertainty?Significant RCTs/other eligible trials published since last update?Are there new trials addressing a specific call for research in previous version of reviewSRs/HTAs covering same ground NOT published (or about to be published) since last update?
Stakeholder
NIHR / DOH / NHS
High spend on intervention or on currently used alternatives?
Presence on list of 20 drugs which had the greatest number of items dispensed, OR greatest Net ingredient costReview has contributed to a NICE guidelinesRelated NICE guideline pending / update due soonIs intervention of strategic importance to fundersHas intervention been highlighted in guidelines or elsewhere as requiring more research
Patients
Does intervention have a high potential impact on quality of life?Is the intervention recommended at all in NICE guidelinesIs the intervention a first line treatment according to NICE guidelines?Current treatment options have evidence of effectiveness but are not well tolerated by patientsDoes the intervention have significant known adverse events (score -1)Intervention discontinued / no longer prescribed (score -1)
Cochrane Collaboration
CCDAN
Review has attracted > 10 citationsReview in top 10 most accessed CCDAN reviewsReview never cited (score -1)Review rarely accessed (score -1)Could this contribute to an MTM / overview
Rest of the World (WHO as proxy)
Review addresses important equity or Global Health IssuesIs there potential for collaboration with or benefit for WHO or other similar body
Does this review directly contribute to Evidence Aid
The criteria below are to be considered once a review has been prioritised according to the above and so these criteria do not contribute to the scoring
Review Quality
Methodology
Full risk of bias tool used
Summary of findings tables used
Methods up to date and written in RevMan 5 headings
Is the scope of the review question sensible (should it be split/merged)
Does the review address its objectives
Patient important changes
Are patient important outcomes addressed - if not they must be.
Are adverse events dealt with - if not they must be
Revised criteria
Cochrane Airways Group Prioritisation exercise using the decision flowchart and
STATA program
Context
• ~250 reviews• 166 reviews over 2 years out of date• Don’t have a prioritisation policy and make
decisions to update based on author availability
• Looking to: – make decisions transparent– direct limited resources at most important topics
What we did
• Considered 21 reviews with the highest number of hits and cites for prioritisation
• TSC ran searches• ME/TSC/RA screened titles and abstracts• Used prioritisation flow chart and STATA tool• Extracted outcome data for the STATA tool
where possible even if review currently being updated
Flow chart
Issues with flow chart
• Seemed very complicated at first, but was OK to use• We created a spreadsheet to reflect the flow of work
from the decision tool and record comments on reviews
• We didn't go searching for abstracts or papers if they weren't available; however we felt that the new studies identified represent the bulk of new research for the review
• Some updates were already in progress and we consulted authors to try to get the number of hits
Innovated table to record decisions
Issues with STATA tool
Possibly too crude?– need to select a single outcome– extract outcome data often from abstract only.
• Unresolved issues with continuous data and GIV
• However, once data had been extracted the STATA tool was easy to use and may find application in some circumstances
Reviews that reached stage x in the flowchart
• Stage 1: 4 review questions appeared answered or partly answered
• Stage 2: 13 reviews needed some sort of work. 7 were unclear. Most needed SOF
• Stage 3: We did searches for all of them to test out the STATA tool (so could save time here in future). Some searches challenging to go through
• STATA: we got sufficient data to run 6 studies through STATA tool but could not analyse the continuous data or the GIV data for unresolved technical reason
Outcomes• 8 indicated for update for other reasons, 5
possible updates and 8 not to be updated (already up-to-date, irrelevant review question, not enough new data)
• We didn’t feel that the STATA tool was worth the additional workload and that it was too crude for our needs, however now we know how it works, may use in future
Issues/comments
• Estimate it took around 5 days in total to do this work. Plus 2 hours with a statistician on STATA.
• This is a reasonable amount of time compared to the editorial process for an ill-conceived update!
• The flow-chart tool provided – a helpful structure to assess each review – a transparent way to summarise evidence to
stakeholders
Issues/comments
• Other groups may need to modify tool and consider which of their reviews to put through the tool
• We would need a further conversation with an advisory board on the obtained information to rank them/decide relevance of review question
• Need a transparent way to select reviews for this tool• With more time, may be possible to screen
references carefully enough to be of use to authors
Objectives To assess whether CRGs other than the CIDG
could classify their Cochrane Reviews using the framework
To identify any problems with the framework and potential solutions
To determine how long it takes CRGs to classify Cochrane Reviews
Methods CRGs who nominated themselves classified a
sample of 20 Cochrane Reviews selected by CEU, and fed back experiences
Classification framework pilot
Results Thirteen CRGs completed the pilot CRGs were able to classify Cochrane Reviews of
interventions, and found the process manageable and useful
CRGs faced several issues when making judgements, and we adjusted guidelines and guidance
CRGs varied in the amount of time they took to classify Cochrane Reviews
Next steps Consider whether framework is suitable for non-
intervention reviews User test framework on readers of CDSR, with mock-ups of
potential views Implementation
Results and next steps
How should Cochrane Reviews be classified if there is more than one comparison, and the comparisons have different categories?
How should a Cochrane Review be classified if trials are ongoing? Could there be more subdivisions in the categories, for example,
to highlight whether an author is or isn’t available? When should a Cochrane Review be classified as up to date? If
there are no new studies, can the Cochrane Review be labelled ‘Current, Up to date’? And what should happen if a few small studies are identified that are unlikely to change the conclusions?
Is the term ‘Current, Up to date’ misleading when a full systematic update search as not been done?
How frequently should these statuses be applied? For the classification ‘Current, No update intended’, how should
CRGs make this judgement? Should there be a final search before a Cochrane Review is
classified as ‘Historical, No update intended’. Should authors be consulted before making a judgement about
the classifications?
Queries from CRGs
Revised framework
Revised framework in light of queries
Recommendations
“Prioritise Cochrane Reviews for updating at least every two years, using methods such as the Updating prioritisation tool or the Updating decision tool...”
“Replace the current guidance to update all Cochrane Reviews every two years, in favour of prioritising updates (see above)...”
“Ensure decisions around prioritisation are transparent to readers.”
“Classify Cochrane Reviews of interventions using the classification framework, at least every two years...”
Working groups
Group 1: “Prioritise Cochrane Reviews for updating at least every two years, using methods such as the Updating prioritisation tool or the Updating decision tool...” AND “Ensure decisions around prioritisation are transparent to readers.”
Group 2:“Replace the current guidance to update all Cochrane Reviews every two years, in favour of prioritising updates (see above)...”
Group 3: “Classify Cochrane Reviews of interventions using the classification framework, at least every two years...”
To consider in working groups• Agree/disagree? Consensus?• If agreement, high or low priority (compared with other strategic session
paper recommendations)? How urgent?• If disagreement, tweak, rework, or discount?• Implementation?
– Changes to tools or framework?– Affect on CRGs’ monitoring?– Changes to Review Manager and Archie?– Changes to The Cochrane Library website?– Changes to CRG entity websites?– Changes to resources (e.g. Cochrane Handbook)?
• Barriers?– Methods?– External perceptions?