AHCJ 2012 Atlanta conf. talk
-
Upload
gary-schwitzer -
Category
News & Politics
-
view
1.870 -
download
1
description
Transcript of AHCJ 2012 Atlanta conf. talk
What are your criteria in reporting on health care research?
Gary Schwitzer, Publisher, HealthNewsReview.org
Background on HealthNewsReview.org, what we do, how we do it, why, and what we’ve found
Recurring pitfalls we’ve seen in 6 years of daily monitoring
An editor’s perspective on addressing these issues
We’ll review a story – together – using HealthNewsReview.org criteria.
Plenty of time for Q & A – tapping Ivan’s editor expertise
Handouts: Summary of our 6-year, 1,700 story experience Resources for reporting on studies Story to review & scoresheet
Our plan for next 2 hours:
We review stories that include claims about…
Medical treatments
Tests
Products
Procedures
Site stats
28 reviewers: 19 clinicians or researchers, 7 journalists, 2 breast cancer survivors
Site launched April 17, 2006 – 6th anniversary week!
>1,700 stories reviewed
Blog launched in 2004 - >2,500 posts
Sole funding from the Informed Medical Decisions Foundation, founded by Jack Wennberg et al
= HealthNewsReview.org (U.S.), Media Doctor sites in Australia, Canada, Germany, Hong Kong, Japan plus somewhat similar sites in UK and Austria
= talks/workshops in Beijing, UK, Spain, Brazil, Argentina, Mexico, Puerto Rico only in two years’ time
Global Reach for such efforts
Our criteria: Does the story explain…
What’s the total cost?
How often do benefits occur?
How often do harms occur?
How strong is the evidence?
Is the condition exaggerated?
Is this really a new approach?
Is it available?
Are there alternative choices?
Who’s promoting this?
Do they have a financial conflict of interest?
After 6 years and 1,700 stories
70% fail to adequately discuss costs.
66% fail to quantify benefit – often exaggerating potential benefit
65% fail to quantify harm - often minimizing potential harm
62% fail to evaluate the quality of the evidence
57% fail to compare new idea with existing options
Kid-in-candy-store picture of U.S. health care
Everything is terrificNothing is riskyNo price tags
Recurring themes seen after 6 years of daily
monitoringFailure to explain limitations of
observational studies. Stories that conflate association and causation.
Framing numbers to exaggerate benefit, minimize harms
Stories about screening tests that emphasize only benefits, minimizing or ignoring potential harms
Be skeptical about observational studies
Remind readers that the findings may be more about the people being studied than the "exposure".
They can point to a strong statistical association, but they cannot prove cause and effect.
White rice increases risk of type 2 diabetes
Thursday, March 15, 2012
The risk of type 2 diabetes is significantly increased if white rice is eaten regularly, claims a study published today on bmj.com.
More than 400 words in the news release but NONE about the limitations of such an observational study.
Does your language fit the evidence of observational studies?
Absolute versus relative risk
Following slides courtesy Drs. Steve Woloshin & Lisa
SchwartzDartmouth Medical School
White River Junction, Vermont VA
Part of their syllabus at: NIH Medicine in the MediaMIT Medical Evidence Boot
Camp
Nolvadex (tamoxifen)Reducing breast cancer risk by 50
percent.
For the first time, there is a clinically proven way for many women at high risk of developing breast cancer to significantly reduce that risk.
The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.
48% of what ???
Women who took Nolvadex had 48% fewer breast cancers….
48
Extremely Fancy Store
%OFF
On selected items!
It's like a coupon…
What if selected items were..
TV’s, washing machines?
Things like a pack of gum?
save $100s
save pennies
“48% of what” matters!
Know the REGULAR price!
What is the
48% off coupon really worth?
How much do you save?
"Breast cancer risk"
NOLVADEX???
Placebo
???
Chance of getting breast cancer
The chance of getting breast cancer over 6 years
in the PLACEBO group
3.3%
The REGULAR price!
PLACEBO
The "base rate"
Absolute risk in the placebo group
Event rate in the placebo group
NOLVADEX
The chance of getting breast cancer over 6 years
in the INTERVENTION group
3.3%
The SALES price!The REGULAR price!
PLACEBO
1.7%
Chance of getting breast cancer (over 6 years) with NOLVADEX was 1.6 % points lower than with placebo
NOLVADEX
3.3%
PLACEBO
1.7%
What is the effect of Nolvadex?How good is the sale?
Savings = Regular price – Sales price
How much do you save?
If 100 women took NOLVADEX instead of placebo for 6 years, there would be about 2 fewer cases of breast
cancer.
The SALES price!The REGULAR price!
= 1.6%
3.3% 1.7% Absolute risk reduction
NOLVADEX
3.3%
PLACEBO
1.7%
What is the effect of Nolvadex?How good is the sale?
Relative Risk =Chance of outcome (intervention)
Chance of outcome (control)
NOLVADEX
3.3%
PLACEBO
1.7%
Relative Risk = = 0.52 ???
???
1.7%
3.3%
Describing the effect of NOLVADEX
"% Lower" format =1 - RR =1 - .52 =.48
So finally....this is how you get to
the 48% off sale! It's the relative risk reduction
At 6 years, the chance of breast cancer for women taking NOLVADEX was 48 % lower than that of women taking placebo.
48
Extremely Fancy Store
%
OFF
On selected items!
1.6%
Extremely Fancy Store
SAVINGS
On selected items!How you say it matters!"Framing"
Two ways of saying the same thing:
the benefit of NOLVADEXOne feels
bigOne feels
small
??
Relative vs. absolute risk reductions
Relative Absolute
30% 10%
Chance of death at 1 year
Risk reduction
67%
67%
67%
3% 1%
0.003% 0.001%
??20%
Placebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG)
2%
0.002%
"% off" "savings"
The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.
”Strokes, cataracts more common with Nolvadex. Most women experience some level of hot flashes and vaginal discharge".
"Nolvadex isn't for every woman…In the study women taking Nolvadex were 2 to 3 times more likely to develop uterine cancer or blood clots in the lung and legs, although each occurred in less than 1% of women".
BenefitHarm
No numbers
Present benefits and harms the same way
210% more uterine cancer and potentially life threatening blood clots in the lung and legs.
210% more
The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.
Harm
6-yr chance of venous thromboembolic event:
Like inflation…doubled the price!
NOLVADEX
0.5%
PLACEBO
1.0%
Over the next 6 years, what happened when women…
Benefits: Nolvadex lowered chance
Getting breast cancer
Harms: Nolvadex increased chance
Having a serious blood clot
3.3% 1.7%
0.5% 1.0%
Net effect of Nolvadex for every 1000 women: 16 fewer women get breast cancer 5 more women get serious blood clots 6 more get uterine cancer
Getting uterine cancer 0.5% 1.1%
NOLVADEX PLACEBO
Take home messagesExtrapolate with caution! Don't tell people what to worry about – or do –
based on very preliminary animal / lab science.
Recognize pseudo-evidence Publication in a medical journal - even "the New England Journal of Medicine" - does not guarantee the findings are true (or even important).
Be wary of inherently weak science Without a comparison group, it is impossible to be sure if the drug was responsible for the findings.
Pay attention to the outcome Surrogate outcomes (like tumor shrinkage) do not reliably translate into clinically meaningful outcomes (longer life).
Be skeptical about observational studies Remind readers that the findings may be more about the people being studied than the "exposure".
Avoid exaggerated numbers Use absolute risks for both benefits and harms.
Pitfalls of a steady diet of journal
stories
PLoS Med 2005; 2(8): e124
The problem begins with the public’s rising expectations of science. Being human, scientists are tempted to show that they know more than they do. The number of investigators—and the number of experiments, observations and analyses they produce—has also increased exponentially in many fields, but adequate safeguards against bias are lacking. Research is fragmented, competition is fierce and emphasis is often given to single studies instead of the big picture. Much research is conducted for reasons other than the pursuit of truth. Conflicts of interest abound, and they influence outcomes.
Spinning results of randomized clinical
trialsBoultron et al, JAMA May 26, 2010, 303 (20): 2058
In this representative sample of RCTs published in 2006 with statistically non-significant primary outcomes, the reporting and interpretation of findings was frequently inconsistent with the results.
Reporting bias in medical research - a narrative
review
McGauran et al. Trials 2010, 11:37
We identified reporting bias in 40 indications comprising around 50 different pharmacological, diagnostic, and preventive interventions. Many cases involved the withholding of study data by manufacturers and regulatory agencies or the active attempt by manufacturers to suppress publication. The ascertained effects of reporting bias included the overestimation of efficacy and the underestimation of safety risks of interventions.
• What Ivan’s project has shown about retractions, research fraud, fabrication, falsification of data
• Unpublished data (BMJ recently published 8 articles in one edition on “the extent, causes and consequences of unpublished evidence”)
• Conflicts of interest – guideline setting
BE AWARE OF…..
• Commercialization of research: contract research organizations, commercial IRBs or institutional review boards, medical education and communication companies (Carl Elliott)
• Ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)
• The focus on surrogate markers in many studies may be hurting patient care. (primer in HealthNewsReview.org online toolkit)
News coverage & poor public discussion of screening tests is one of the most concerning public policy issues.
Worst, most biased coverage I’ve seen in 37 years
“The decision to start regular, biennial screening mammography before the age of 50 years should be an individual one and take patient context into account, including the patient's values regarding specific benefits and harms.”
What the Task Force actually wrote:
As 37-year ChiTrib & NYT vet John Crewdson wrote in The Atlantic…
“There are multiple reasons women are ill-informed about breast cancer. The fault lies primarily with their physicians, the cancer establishment, and the news media--especially the news media. Until coverage of breast cancer rises above the level of scary warnings mixed with heartwarming stories of cancer survivors, women are likely to go on being perplexed."
Other examples:
Chicago Sun-Times
Wall Street Journal
Washington Post
New York Daily News
Minneapolis Star Tribune
All TV networks
The ink isn’t even
dry on studies before
marketin
g begins
HealthDay wire service didn’t challenge researcher promoting universal pancreatic cancer screening for everyone over 50.
After study of tissue from 7 people!
We could have given the finger to this story
"Relative finger length could be used as a simple test for prostate cancer risk, particularly in men aged under 60," said one of the researchers.
Screening Madness
Crusading
one-sided
advocacy
Why don’t we deliver this
message?
“All screening tests cause harm;
some may do good.”
But much health journalism consistently emphasizes benefits & minimizes harms
A form of disease-mongering
Selling sickness
Selling the search for weapons of mass destruction inside everyone
Dr. Gil Welch in NYT – If You Feel OK, Maybe You Are OK
“Screening the apparently healthy potentially saves a few lives. But it definitely drags many others into the system needlessly — into needless appointments, needless tests, needless drugs and needless operations. This process doesn’t promote health; it promotes disease. People suffer from more anxiety about their health, from drug side effects, from complications of surgery. A few die. And remember: these people felt fine when they entered the health care system.”
Communication of the evidence is a key health policy issue
“I honestly believe it is better to know nothing than to know what ain’t so.”
Josh Billings (pen name of humorist Henry Wheeler Shaw, 1818 – 1885)
Thank you