Critical appraisal of published research: introductory guidelines
Critical appraisal of published medical research
-
Upload
tarek-tawfik-amin -
Category
Health & Medicine
-
view
447 -
download
0
Transcript of Critical appraisal of published medical research
Critical Appraisal of Published Medical Research
Dr. Tarek AminProfessor of Public Health
Cairo University
Background
• Every day … – ~ 46 randomized clinical trials are
published – ~ 1000 new Medline articles – ~ 6,000 new articles in biomedical
journals• Every year … – ~ 3 million articles published in ~
30,000 journals
Background
• Most research published in medical journals is either
– Too poorly done – Insufficiently relevant to be
clinically useful• Besieged with too much information
to keep up to date. • High quality information is often not
easy to find.
Critical appraisal is not
o Negative dismissal of any piece of research
o Assessment on results aloneo Based entirely on statistical analysiso Undertaken by experts only
Why critically appraise?
• To find out the validity of the study– Are the methods robust?
• To find out the reliability of the study– What are the results and are they credible?
• To find out the applicability of the study– Is it important enough to change my
practice?
What do I need to know?
o Awareness of study designso Levels of evidenceo Statistics!!o CA checklistso CA resources
Roadmap
1. Describe the evidence2. Internal validity3. External validity4. Comparison with other evidence5. Resources
1 -Describe the evidence
What relationship being evaluated and what hypothesis was tested?
What were the exposure and the outcome variable? What was the study design? Case report, series Survey Clinical trial Case-control Prospective or retrospective cohort study Cross sectional study or Others
1 -Describe the evidence Definition of participants in terms of: – Source populations – Time frame – Eligibility criteria – Participation rates of the different groups
compared Summary of the main results: What is the result in terms of association between
exposure and outcome?
Should be possible to express the main result in a simple table and obtain from the paper the means to calculate the appropriate measure of association.
2- Internal Validity
o The truthfulness of inferences about the study population.
o Causal relationship between exposure and outcome or just an association?
2 -Internal Validity
Two aspects of internal validity1. Non-causal explanations2. Causal explanations
Non-causal explanations
• Bias• Confounding• Chance variation
Non-causal explanations
• Bias – Selection bias, surveillance,
diagnosis, referral, non-response, length of stay, survival bias
– Misclassification bias: recall, interviewer, improper analysis, etc.,
Confounding
Smoking is a risk factor for cancer of the larynx
• we’d like to quantify the strength of the association between smoking and laryngeal cancer, but …
• many smokers are also drinkers (which is also a risk factor for cancer of the larynx)
• drinking is said to confound the association between smoking and risk of laryngeal cancer
Confounding definition
Confounding (the formal definition)• The effect of an extraneous variable
that wholly or partially accounts for the apparent effect of the study exposure, or masks an underlying true association
A variable is confounder
A variable is a confounder if:1. It is causally associated with the
outcome; and2. It is non-causally associated with
the exposure; and3. It and the exposure variable are on
two separate causal pathways
Confounding.
A confounding variable is associated with the exposure and it affects the outcome, but it is not an intermediate
link in the chain of causation between exposure and outcome.
Myocardial infarctionOral contraceptive
Smoking
IUD insertion
STDs
Salpingitis
Chance variation
A relationship between exposure and outcome identified by chance?
Type I error: null hypothesis is rejected when, in reality, it is true.
Non-causal explanations
The order of these non-causal explanations is important:
o Observation (information) bias, analytical manipulation of the data will not overcome the problem
o Confounding, then appropriate analysis will (in most cases) overcome the problem
Five aspects of causal explanations
1. Is there a correct temporal relationship?
2. Is the relationship strong?3. Is there a dose-response
relationship?4. Consistency of the association?5. Specificity of association
1 .Is there a correct temporal relationship?
o The exposure must act before the outcome occurs
o No problem with prospective study designs
o Difficult in retrospective studies
2 .Is the relationship strong?
Larger relative risks (and Odds) are more likely to reflect causal relationships.
3 .Is there a dose-response relationship?
The greater the exposure, the greater the risk of disease.
4 .Consistency of the association
Expected to apply across a wide range of subjects.
An association identified in one study that is consistent with the same association identified in a different groups of subjects.
5 .Specificity of association
Specificity: exposure produces a specific outcome (e.g. asbestos and mesothelioma)
3 -External Validity
External validity: can the results be applied to populations other than that which was studied?
• If the internal validity of a study is poor, the answer is no
Aspects of external validity:1. Applied to the eligible population?2. Applied to the source population?3. Applied to other relevant populations?
1.Can the results be applied to the eligible population?
– The relationship between the study population (the population from which samples are taken) and the eligible population (those that met the study inclusion criteria but did not take part) should be well documented.
– Non-participation have to be considered carefully as they are likely to be non-random.
2 .Can the results be applied to the source population?
Whether the association between outcome and exposure given by the study participants is likely to apply to other groups
3 .Can the results be applied to other relevant populations?
The difficulties of applying results from one group of subjects to another will be minimal for issues of basic physiology and maximal for effects in which cultural and psycho-social aspects are dominant
4 -Comparison with other evidence
Useful to consider a hierarchy of evidence .
1. Randomized [clinical] trials2. Cohort and case-control studies3. Other comparative studies4. Descriptive studies, case series,
case studies, clinical experience
Three aspects of comparison should be considered:
1. Results consistent with other evidence?
2. Results plausible biologically?3. Coherency with the existing
knowledge.
Are the results consistent with other evidence?
• Most important characteristic used in the judgment that an association is causal
• Lack of consistency argues against causality
Plausibility
Is the observed association biologically understandable?
Coherency
• An association is regarded as coherent if it fits the general features of the distribution of both the exposure and the outcome under assessment.
Summary
1. Describe the evidence• – type of study, outcome measure, population investigated,• results2. Internal validity• – non-causal explanations• • bias• • confounding• • chance• – causal explanations• • temporal relationship• • strength of relationship• • dose-response• • consistency• • specificity
Summary
3. External validity• – can the results be applied to the eligible population?• – can the results be applied to the source population?• – can the results be applied to other relevant populations?4. Comparison of the results with other evidence• – are the results consistent with other evidence?• – are the results plausible biologically?• – is there coherency with the distribution of the exposure
and the• outcome?• Can we apply these results to other populations? Are
the findings reported here consistent with other studies that looked at the same thing?
Resources
Web sites– Users’ Guides to Evidence-Based Practice
http://www.cche.net/usersguides/main.asp– A Student’s Guide to the Medical Literature
http://grinch.uchsc.edu/sg/– Pearls for Residents: Annotated Critical
Appraisal References http://www.cfpc.ca/English/cfpc/cme/pearls/pearls%20residents/default.asp?s=1
• Critical Appraisal of Bio-medical Literature http://www.shef.ac.uk/scharr/ir/units/critapp/resources.htm
• Critical Appraisal Resources for Assessing Health and Medical Research http://www.etsu.edu/health/index_files/harvill_handout.pdf
• Bandolier http://www.medicine.ox.ac.uk/bandolier/
Thank you