1 Eduardo Ortiz, M.D., M.P.H. National Heart, Lung, and Blood Institute National Institutes of...
-
Upload
candace-smith -
Category
Documents
-
view
216 -
download
3
Transcript of 1 Eduardo Ortiz, M.D., M.P.H. National Heart, Lung, and Blood Institute National Institutes of...
1
Eduardo Ortiz, M.D., M.P.H.
National Heart, Lung, and Blood InstituteNational Institutes of Health
May 10, 2011
IOM Standards for Systematic Reviews: Finding and Assessing Individual Studies
2
Acknowledgements• Colleagues at the Division for the Application of Research
Discoveries at NHLBI: Denise Simons-Morton, MD, PhD (Division Director) Glen Bennett, MS Janet de Jesus, MS, RD Karen Donato, SM, RD Rob Fulwood, PhD, MSPH Edward Donnell Ivy, MD Chi Onyewu, MD, PhD Susan Shero, MS, RN Joylene John-Sowah, MD Sid Smith, MD Zhi-Jie Zheng, MD, PhD
3
Joint National Committee on Prevention, Detection, Evaluation, &Treatment of High Blood Pressure
JNC 7: 2003 JNC 6: 1997 JNC 5: 1992 JNC 4: 1988 JNC 3: 1984 JNC 2: 1980 JNC 1: 1976
Detection, Evaluation, &Treatment of High Blood Cholesterol in Adults (ATP - Adult Treatment Panel)
ATP III Update: 2004 ATP III: 2002 ATP II: 1993 ATP I: 1988
Clinical Guidelines on the Identification, Evaluation, & Treatment of Overweight and Obesity in Adults Obesity 1: 1998
3
NHLBI Cardiovascular Clinical Guidelines
4
Guidelines for the Diagnosis and Management of Asthma (EPR-3) EPR-3: 2007 EPR-2 Update: 2002 EPR-2: 1997 EPR-1: 1991
The Diagnosis, Evaluation, and Management of Von Willebrand Disease
2006
Management of Sickle Cell Disease
2002 1995 1989 1984
4
Other NHLBI Clinical Guidelines
5
Current Guideline Efforts
High Blood Pressure Cholesterol Overweight/Obesity Adult Cardiovascular Risk Reduction Crosscutting Work Groups to support our
Adult CVD Guideline efforts: Lifestyle, Risk Assessment,
ImplementationPediatric Cardiovascular Risk Reduction Sickle Cell Disease
66
Guidelines Development Process
Expert PanelSelection
External Reviewwith Revisions
as Needed
Literature Search
ScreeningData Abstraction
Study Quality Grading
Create Evidence Tables;
Grade Body of Evidence
Dissemination Implementation
Evaluation
Evidence Statementsand
Recommendations
Topic Area Identification
Analytic Models and Critical Questions
777
Evidence-Based Systematic Review Process Step 1 – Develop analytic framework and critical
questionsStep 2 – Establish inclusion & exclusion criteria Step 3 – Screen titles, abstracts, and full text to
identify relevant studies for inclusionStep 4 – Rate the quality of each individual studyStep 5 – Abstract data for studies rated good or fairStep 6 – Create evidence tables Step 7 – Create summary tables Step 8 – Create narrative summary Step 9 – Rate the quality of the overall body of
evidence for each critical question
888
Developing the Recommendations
Step 10 – Review the evidence Step 11 – Develop evidence statements, including
rating the quality of evidence for each statementStep 12 – Develop recommendations, including
grading the recommendations and making sure they are supported by the evidence
Step 13 – GLIA assessment of recommendationsStep 14 – Public comment period, with invitations for
review Step 15 – Review comments and revise
recommendationsStep 16 – Final recommendations Step 17 – Dissemination, implementation, and
evaluation
9
Developing the Critical Questions
Critical questions are developed by the expert panels working collaboratively with the methodology team and NHLBI leads
PICOTS format
Predefined inclusion and exclusion criteria
10
Screening
Each study is screened by two independent reviewers using the I/E criteria
Review titles and abstracts, followed by full text
If they disagree, they discuss and try to reach consensus
If they do not achieve consensus or need additional input, there is 3rd party adjudication
Panel members can appeal a decision Re-assessed and adjudicated by a 3rd party, but
the panel cannot override a decision made by the reviewers.
11
Lessons Learned and Challenges
Issues:
Sometimes you get the prespecified I/E criteria wrong and have to be practical and make post-hoc adjustments
Despite your best efforts, it is sometimes difficult to figure out whether a study should be included or excluded
12
Rating the Evidence
Quality of each included study is rated by two trained independent reviewers at the time of data abstraction. No satisfactory tools were available for assessing
study quality, so we developed our own Controlled intervention studies, cohort studies,
case control studies, and systematic reviews / meta-analyses
Despite your best efforts, it is sometimes difficult to determine the quality rating of an individual study
Panel members can appeal a decision Re-assessed and adjudicated by a 3rd party, but
the panel cannot override a decision made by the reviewers.
13
Lessons Learned and Challenges
We used one abstractor and a reviewer to check the abstracted data for accuracy and completeness Consistent with the Buscemi study, we
experienced a substantial number of errors
Dependent on the individual abstractor and reviewer
Takes a lot of time and can create a bottleneck
Using two independent abstractors would not have been feasible
14
Rating the Evidence
Overall body of evidence is rated for each critical question using a standardized rating instrument We reviewed GRADE, USPSTF, ACC-AHA, and
many other systems Hybrid model similar to USPSTF High, Moderate, Low
We need a better, standardized, user-friendly approach to rating evidence and grading the strength of recommendations
15
Lessons Learned and Challenges
For guidelines, you need to answer many questions
Conducting multiple systematic reviews to answer these questions is very challenging and requires a lot of time, effort, expertise, manpower, and money
How do we develop high-quality, credible systematic reviews to support our guidelines that can be completed and updated in a timely manner? Is there a sweet spot to aim for between evidence-based rigor and practicality?
16
Lessons Learned and Challenges
Standard 3.2 – Take action to address potentially biased reporting of research results Grey literature and other sources of unpublished studies Contacting researchers and asking them to clarify study
related information Asking study sponsors to submit unpublished data Conference abstracts Studies not published in English
It takes a lot of time and effort to search the published literature and conduct all the other steps in the SR process without searching all these additional resources
Some of these issues hopefully will be addressed by investigators and publishers, as it is not realistic to expect most groups developing SRs to be able to do all this
17
Lessons Learned and Challenges
Updating searches during and after the review process Very important, but once again practical
considerations come into play A considerable amount of time can lapse between
the initial search and completion of the review If you have to go back and update the material, it
is not just the search that has to be repeated, but all the other steps in the EB review process, which then takes more time and can lead to a vicious cycle.
How do we deal with this from a practical perspective, yet maintain a reasonable time frame and budget?
Lessons Learned and Challenges
Lots of personnel, time, effort and costs to screen studies for inclusion/exclusion, assess study quality, adjudicate decisions, abstract data, create evidence tables and summaries, etc.
If you don’t have all the needed expertise to do this in-house, contracting out the work can be challenging High costs, coordination, and decision-making across
organizations and individuals Getting screeners, reviewers, and abstractors that have
enough methodological expertise and clinical knowledge to understand important contextual issues and other nuances can be a challenge, especially when conducting multiple SRs
Variability in the quality of the reviewers and methodologists Reviewers and methodologists can differ in their
knowledge, perspectives, biases, attention to detail, etc., so the quality of reviews can vary, depending on the individuals
Final Comments
Support the IOM recommendationsHopefully will improve the quality and
consistency of systematic reviewsReport is comprehensive and represents
an ideal process, but it would have been helpful to provide more practical or “at a minimum” recommendations to factor in real-world limitations facing most organizations
Stronger linkage between the SR and CPG reports, including more practical recommendations to assist those of us who conduct SRs for the purpose of developing guidelines
20
Thank you!
Eduardo Ortiz, M.D., M.P.H.
NHLBI Guideline Information: http://www.nhlbi.nih.gov/guidelines/index.htm