An Introduction to Critical Appraisal Dr Ghamdi.pdf
-
Upload
matthew-mendez -
Category
Documents
-
view
219 -
download
0
Transcript of An Introduction to Critical Appraisal Dr Ghamdi.pdf
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 1/51
An Introduction to Critical Appraisal
Kholoud S. Al Ghamdi (MD, PhD)
November 27, 2014
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 2/51
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 3/51
اشراحرإن
بإاقذك
عروبقيقإواسثء
به
بجداقأداإذا
.اقد
د
.سدعا
Criticism is important in the
continuation of the humans’ life
cycle, since every human, without
exception, has his/her own
shortage in one or more aspects of
life and every person admits this. So
as long as shortage exists, there isalways place for criticism.
Dr. Salman Alowdah
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 4/51
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 5/51
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 6/51
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 7/51
“It is astonishing with how little reading a doctor can practice
medicine, but it is not astonishing how badly he may do it”
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 8/51
What is evidence based practice?
Evidence-based practice is the integration of
• individual clinical expertise
with the
• best available external clinical evidence from systematicresearch
and
• patient’s values and expectations
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 9/51
What is the Relation Between…
AND
?
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 10/51
The evidence-based practice (EBP) process includes:
• Decision or question arising from a patient’s care.
• Formulate a focused question.
• Search for the best evidence.
• Appraise the evidence.
• Apply the evidence.
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 11/51
Do NOT believe anything people write until
you’ve convinced yourself it was a well donestudy with valid conclusions.
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 12/51
What is critical appraisal?
It is the process of systematically examining research evidence to assess itsvalidity, results, and relevance before using it to inform a decision”
(Hill and Spittlehouse, 2001)
A critical review must identify the strengths and weaknesses in a piece of researchand this should be carried out in a systematic manner
(Eachus, 2003)
It is a skill that needs to be practiced by all health professionals as part
of their work.
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 13/51
Why do we need to critically appraise?
studies which don't report their methods fully overstate the benefits oftreatments by around 25%
Khan et al. Arch Intern med, 1996; Maher et al, Lancet 1998.
studies funded by a pharmaceutical company were found to be 4 timesas likely to give results that were favourable to the company than
independent studiesLexchin et al, BMJ, 2003
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 14/51
How do I appraise?
You need to know if this research is:
Valid & Relevant
• Mostly common sense.
• You don’t have to be a statistical expert!
• Checklists help you focus on the most important aspects ofthe article.
• Different checklists for different types of research.
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 15/51
• What is the objective/hypothesis of this manuscript?
•What outcomes are being measured?What is the data type gathered?
• Is the study biased, is there confounding, can the results be explained by chance?
Subject selection, data collection proper?
•Are the conclusions supported by the study data?
Appropriate statistics, adequate power?
• Was the study ethical and without conflict of interest?
General Questions to Ask
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 16/51
Research methods
Quantitative
• Uses numbers to describe and analyse
• Useful for finding precise answers to defined questions
Qualitative
• Uses words to describe and analyse
• Useful for finding detailed information about people’s perceptions and attitudes
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 17/51
What type of study is it?
• Descriptive studies – Data used for descriptive purposes and not used to make predictions
– Correlational studies, case reports or series, cross sectional surveys
– Measures of central tendency (mean, median, mode)
• Inferential studies – Use data from study sample to derive conclusions and/or make
predictions about the population
– Statistics used to prove or reject hypothesis
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 18/51
What type of study is it?
• Systematic reviews
• Randomized controlled trials
• Prospective studies (cohort studies)
• Retrospective studies (case control)
• Case series and reports
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 19/51
What is the intervention?
What are the:
Dependent
Independent
Confounding
variables of the study?
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 20/51
Who are the subjects?
The ideal patients to study is from a random sample
most studies do not use a totally random sample
selection bias!
Subject inclusion/exclusion criteria?
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 21/51
Is the study internally valid?Do I believe it?
Assess hypothesis/objective of the study
Research hypothesis what the researcher predicts
Null hypothesis (Ho) there is no difference in outcome between the two
groups; in general expect the null hypothesis to be rejected because researcher
usually predicts a difference between groups
Alternate hypothesis (H1) there is a difference between the groups; typically,researcher expects this to be supported so this is the research hypothesis
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 22/51
Systematic reviews
• Thorough search of literature
•All RCTs (or other studies) on a similar subject synthesised and summarised.
• Meta-analysis to combine statistical findings of similar studies.
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 23/51
Appraising systematic reviews
• Was a thorough literature search carried out ?
•How was the quality of the studies assessed?
• If results were combined, was this appropriate?
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 24/51
Randomised controlled trials (RCTs)
• Normal treatment/placebo versus new treatment.
•Participants are randomised.
• If possible should be blinded:
• Best design is when neither the investigator nor the subject know whichgroup they are in (double blinding)
• Intention to treat analysis
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 25/51
Appraising RCTs• Recruitment and sample size
• Randomisation method and controls
• Confounding factors: factors that 'get in the way‘
• Blinding
• Follow-up (flow diagram)
• Intention to treat analysis
• Adjusting for multiple analyses
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 26/51
“As a non-statistician I tend only to look for 3 numbers
in the methods :
• Size of sample
• Duration of follow-up
• Completeness of follow-up”
Greenhalgh, 2010
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 27/51
Assess adequate sample size
• Especially important for descriptive statistics
– 83% success rate in 6 patients is different from same success rate in 600 patients
• In general, inferential statistics take sample size into consideration
– Results may trend towards significance (p = 0.05) with low sample size but becomesignificant if more subjects were enrolled
• In research design, however, power calculations should be done to assess adequatesample size
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 28/51
Cohort studies• Prospective
• groups (cohorts)
• exposure to a risk factor
• followed over a period of time
• compare rates of development of an outcome of interest
• Confounding factors and bias
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 29/51
Case-control studies• Retrospective
• Subjects confirmed with a disease (cases) are compared with non-diseased
subjects (controls) in relation to possible past exposure to a risk factor.
• Confounding factors and bias
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 30/51
Appraising cohort/case control studies
• Recruitment – selection bias
•Exposure - measurement, recall or classification bias
• Confounding factors & adjustment
• Time-frames
• Plausibility: a relationship between a putative cause and an outcome
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 31/51
Publication bias/ Results
papers with more ‘interesting’ results are more likely to be:
– Submitted for publication
– Accepted for publication
– Published in a major journal
– Published in the English language
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 32/51
Results• How was data collected?
• Which statistical analyses were used?
• How precise are the results?
• How are the results presented?
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 33/51
Statistical testsType of test used depends upon:
• Type of data – categorical, continuous etc
• One- or two-tailed (or sided) significance
• Independence and number of samples
• Number of observations and variables
• Distribution of data, e.g. normal
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 34/51
P values
• P stands for probability - how likely is the result to have occurred by chance?
• P value of less than 0.05 means likelihood of results being due to chance is lessthan 1 in 20 = “statistically significant”.
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 35/51
• When a difference is shown, it could be due to
(1) chance or
(2) a true finding
– Chance (Type I error), False positive
– Generally we accept less than 5% chance of type I error; so check that alpha
level (P value) is set at ≤0.05 for level of significance.
P values
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 36/51
• When no difference is found (accept null hypothesis), it could be:
(1) The truth
(2) False negative (Type II error, beta)
– Beta is typically set at 20%
– Power of the study is defined as the probability of true positive (accept
alternate hypothesis), typically 0.80 (i.e. there is an 80% chance of detectingthe difference if one truly exists)
P values
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 37/51
Statistical significance
• statistical significance does not necessarily equal clinical significance
• statistical non-significance may be due to small sample size
• A bigger sample may result in a statistically significant difference
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 38/51
You also will need to look for the following:
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 39/51
Look for Sources of Bias in the Study
• Bias
– Things that may influence the research and lead to a systematicdeviation from the truth
– May occur in each stage of data manipulation:
• Collection
• Analysis
• Interpretation
• Publication
• Review
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 40/51
• Design bias
• Sampling bias• Observer bias
• Reviewer bias
Examples of Bias in a Study
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 41/51
Sampling bias is introduced when the sample used is not representative of the
population or inappropriate for the question asked.
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 42/51
Look for Confounding Variables
• A confounding variable is one that is associated with the predictor variableand is a cause of the outcome variable
– Example: An association was seen in a study between coffee drinking and MI.
However, if more coffee drinkers were also smokers then smoking is theconfounding variable
- So need to know other risk factors for disease
- Randomization reduces confounding but this research design is not alwayspossible
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 43/51
Look for adequate follow-up
• In general the follow-up should be at least 80%.
• Inadequate follow-up or too many loss to follow-up is a serious flaw in
research;
– What if only the happy patients followed up with the study?
• The authors should account for all patients lost to follow-up, and at least
discuss the potential bias and data scenarios
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 44/51
Look for statistical measures used and findings
• What is the data type measured for outcomes? – Nominal (categorical)
– Ordinal (rank order)
– Continuous
– Ratio
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 45/51
• Check that proper statistical analysis was performed; example,
– Categorical data chi square
– Ordinal dataMann - Whitney U Test, spearman’s rho, weighted kappa
– Continuous data t-test, Z-test
• Keep a statistical reference book handy to review new statistical terms whilereading journal articles until familiar with it
Look for statistical measures used and findings
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 46/51
Look for Disclosure?
• Disclosure
– “the act of revealing something”
• Medical Disclosure
– Author, editor, and reviewer must disclose any financial or personal
relationships that inappropriately influence (bias) his or her actions
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 47/51
"Moderation of Critical Appraisal session/ Journal club
The role of a moderator of a session:
1: Identifying or help in selection of paper. The paper must not be a 'perfect'
paper published in 'Nature' or 'Lancet', It must have some deficiencies to
generate critical appraisal/discussion
2: Following the critical appraisal checklists as regards critique of different
items: critique of title, abstract, introduction, methods, results, discussion,
conclusions, references.
3: Minimal interference. Listening and interfering only when the presentingstudent or attending students make a blunder or attending faculty do not
contribute in a matter of difference of opinion. Let the presenting student
'present' and attending students discuss
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 48/51
"Moderation of Critical Appraisal session/ Journal club
The role of a moderator of a session:
4: Preparation: so that moderator is capable of supporting or negating some
views if the need is there.
5: Awareness of the methods and particularly statistics used in the paper.
6: Evaluation of presenting and attending students.
7: Advertising: Ensuring that the student sends information/invitation andreminder to all students and faculty. Encourage faculty to attend, add
'attractants' like prizes or refreshments!
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 49/51
http://www.sign.ac.uk/methodology/checklists.html
Critical Appraisal: Online Notes and Checklists
7/26/2019 An Introduction to Critical Appraisal Dr Ghamdi.pdf
http://slidepdf.com/reader/full/an-introduction-to-critical-appraisal-dr-ghamdipdf 50/51
Dr. Sarah [email protected]
Dr. Samira [email protected]
Dr. I. [email protected]