1
Foolish & Fatal Flaws
When medical science goes bad
2
Outline
Background on statistical and methodological error On Pigs and PCCs
Crossing the species barrier Correct group assignment
What? You mean some of the control patients actually got transfused?
Large starting group consents, but few randomized & deceptive statistics You mean only 6.8% of patients consenting to the trial
were actually randomized? You mean there are missing statistical calculations?
‘Temporal ambiguity’ Outcome before exposure – What? The patient got
pneumonia before the transfusion?
3
The use of statistics in medical diagnoses and biomedical research may affect whether individuals live or die, whether their health is protected or jeopardized, and whether medical science advances or gets sidetracked. [...] Because society depends on sound statistical practice, all practitioners of statistics, whatever their training and occupation, have social obligations to perform their work in a professional, competent, and ethical manner.”[Ethical Guidelines for Statistical Practise,
American Statistical Association,1999].
4
When transfusion medicine gets sidetracked Recombinant factor VIIa Transfusion-related
immunomodulation Formula-driven resuscitation Albumin use in critical care RBC transfusion in patients with
ischemic heart disease
5
Where can you go wrong?Strasak AM, et al. Swiss Med Wkly 2007; 137: 44-49
Study design Failure to a priori define outcomes Failure to perform sample calculations Failure to blind (or disclose who the blinding was done) Control and treatment groups not comparable
Data analysis Messing up on simple statistical tests – i.e. using a t-test
without meeting test requirements or appropriate correction tests
When comparing multiple groups – can’t use ‘two-group’ tests
Post-hoc subgroup analysis – ‘shopping’ for statistically significant results
6
Where can you go wrong?Strasak AM, et al. Swiss Med Wkly 2007; 137: 44-49
Documentation ‘Where appropriate’ use of a statistical test Failure to state the number of ‘tails’ used
for a specific analysis Presentation
Correct use of standard deviation vs. 95% confidence interval vs. inter-quartile ranges
Median vs. mean P values – record exactly – not ‘ns’, ‘<0.05’,
‘>0.05’ – no cheating when you round your p values
7
Incongruence between test statistics and P values in medical papersBMC Med Res Methodol. 2004; 4: 13.
11.6% and 11.1% of the statistical results published in Nature and BMJ respectively during 2001 were incongruent, mostly due to rounding, transcription, or type-setting errors
At least one such error appeared in 38% and 25% of the papers of Nature and BMJ, respectively
In 12% of the cases, the significance level might change one or more orders of magnitude
The frequencies of the last digit of statistics deviated from the uniform distribution and suggested ‘digit preference’ in rounding and reporting (a.k.a. lying)
8
Should see uniform distribution
9
More haste, less science?Nature. 1999 Aug 5;400(6744):498
10
Name Example
Centripetal Bias Healthcare access bias
Mode for mean bias Frequency-quantity Q to assess EtOH intake, subject reports modal rather than mean intake (closer to zero)
Obsequiousness bias Subjects alter responses in the direction they perceive is desired by the investigator
Details 74 types of medical bias
11
What are the Contributing Factors to Misuse?
Pressures to publish, produce results, or obtain grants
Career ambitions or aspirations Conflicts of interest and economic
motives Inadequate supervision, education, or
training
Gardenier JS, Resnik DB. The misuse of statistics: concepts,tools, and a research agenda. Account Res. 2002;9:65–74.
12
Bottom line: Be on the look out for medical science flaws!
13
On Pigs and PCCs
Prothrombin complex concentrate vs fresh frozen plasma for reversal of dilutional coagulopathy in a porcine trauma model.
Dickneite G, Pragst I.Brit J Anesthesia 2009; 102: 345-
54. The study was funded and conducted by representatives of the company that makes the PCC product
14
The study design
47 pigs, 20-30 kg, anesthetized, 70% isovolemic blood loss with replacement with RBCs/HES
The pigs were randomized to (5-7 each group): 15 mL/kg of porcine FFP 40 ml/kg of porcine FFP 25 U/kg of human-derived PCC (Beriplex P/N) 15 mL/kg saline
Porcine FFP was used instead of human FFP because infusion of human FFP into pigs can result in transfusion reactions
Following this resuscitation, the pigs underwent a controlled injury 3 mm hole into the femur or 7 cm x 1cm incision into
the spleen
15
Outcomes – measured by blinded observers
Time to hemostasis and blood loss were monitored for 120 min after injury
Skin bleeding time (SBT) in duplicate Blood samples were collected at baseline,
after the completion of hemodilution and 5 min after study treatment administration
Coagulation factors (2/7/9/10) were measured They did not measure the non-vitamin K
dependent factors
16
The effect on the PT – all 3 work
17
Factor X – only PCC works
Human equiv11 units FFP
Expectedresults
18
Proportion with hemostasisBetter off with saline than FFP?
PCC
Saline
FFP - low
FFP – high
19
Spleen blood loss
20
Bone blood loss
Saline
FFP-low FFP-highPCC
21
Skin bleeding time
22
Fibrinogen level
2 ½ dosesOf FFP?
23
The investigators conclusion In view of the unmet clinical
need for more efficacious haemostatic agents in such patients, clinical studies are now justified to confirm the observed favorable effects of PCC in the present preclinical model system
24
A lot of unanswered questions The laboratory tests used to measure clotting factor
levels are based on human factor-deficient plasma Do the lab assays work for human and pig samples for all
factor assays performed? Perhaps this explains why PT corrects but not factor
assays Why would massive FFP exposure increase blood
loss in these pigs? Was there something wrong with their pig FFP? Is hemodilution with FFP a bad thing? Did they fail to give enough RBCs?
Why would human PCC immediately stop bleeding in these pigs and FFP would make you bleed more than saline?
25
Comment
Pig vs. Human comparison Infusion of human factors into pig recipients via PCC may
have induced enhanced coagulation compared with what those same proteins would achieve in a human recipient?
The clinical bleeding outcomes observed between animals infused with pig FFP versus animals infused with human PCC in this experiment are difficult to compare directly
The real value of PCCs compared with FFP in human trauma cannot be answered by this experimental design
If you are a pig and you get injured, you want human PCC!
26
Caution
Make sure your trauma surgeons do not take this study at face value
27
Correct group assignmentWhen you untransfused control group might be transfused
Intraoperative transfusion of 1 U to 2 U packed red blood cells is associated with increased 30-day mortality, surgical-site infection, pneumonia and sepsis in general surgery patients.
Bernard AC,Davenport DL,Chang PK, et al. J Am Coll
Surg 2009; 208: 931-37
28
Design
Prospective study of patients undergoing major surgical procedures at 121 hospitals
Nurses prospectively collected preoperative, intraoperative and postoperative variables for 30 days after the operation on the first 40 operations in each 8-day cycle
Database was queried for 05-06 for all general surgery procedures
29
Transfused vs. not transfused ‘Transfusion’ =
number of PRBC units transfused intraoperatively
Transfused >4 U RBCs is the first 72 hours post-op
2nd group = yes/no definition, not number of units
30
Time course
Intraoperative 30 days
72 hours
>4 unitsyes/no
>4 unitsyes/no
numbertransfused
Data variable only
=Transfused
OR
31
Outcomes over 30 days
Composite surgical-site infection (superficial, deep, or organ/space)
Urinary tract infection Pneumonia (without preoperative pneumonia) Sepsis/septic shock Composite morbidity – all of the above 1 or more of 20 adverse events uniformly
defined by the ACS-NSQIP, excluding bleeding requiring transfusion
Mortality
32
Analysis
Estimated probability for a patient to receive a transfusion (propensity) was calculated by MLR analysis of all the available patient and operative risk factors
Risks for outcomes by level of intraoperative transfusion were calculated using logistic regression, with adjustment for: transfusion propensity, procedure group, and complexity other ACS-NSQIP risk factors operative duration (proxies for technique), level of transfusion received intraoperatively postoperative transfusion of >4 U PRBCs
33
Results
125,223 general surgery patients at 121 hospitals were retrieved
4,788 patients (3.8%) received intraoperative RBC or >4 in the 72 hours post
Risk variables most predictive of transfusion were inpatient procedure, procedure group, ASA class, hematocrit >38!, preoperative transfusion >4 U, emergent procedure, esophageal varices, and age
34
Results
Patients receiving a single unit of intraop RBCs had higher rates of surgical-site infections, urinary tract infection, pneumonia, sepsis/shock, composite morbidity, and 30-day mortality
After adjustment for transfusion propensity, procedure group and complexity, wound class and operative duration, and all other important risk variables, transfusion significantly (p<0.05) increased the risk of mortality (OR 1.32), composite morbidity (OR 1.23), pneumonia (OR 1.24), and sepsis/shock (OR 1.29), but not surgical-site infection
35
Their conclusions – Can’t argue with these motherhood statements RBC transfusions should be used very selectively
during surgical procedures. Mild anemia should be tolerated. Blood-conservation strategies and appropriate
indicators for transfusion should be used. Additional studies must determine the
mechanisms by which transfusion of PRBCs and other blood components contribute to poor patient outcomes.
Although deleterious effects are evident and some mechanisms have been suggested, reversible causes and effective treatments have yet to be definitively determined.
36
The flaw
Their statistical model was based on patients requiring intraoperative transfusions and if a patient was given >4 transfusion within 72 hours post-operatively.
As such, a patient could have received up to 8 transfusions in the perioperative period and would have been considered as not having a transfusion!
37
Large population screened (few enrolled)Deceptive statistics, ‘underpowered’
Safety and efficacy of recombinant activated
factor VII. A randomized placebo-controlled trial in
the setting of bleeding after cardiac surgery. Gill R, Herbertson M, Vuylsteke A, et al. Circulation 2009;
120: 21-27.industry-funded
38
Design
Phase 2 dose-escalation study
Safety and possible benefits 30 sites in 13 countries Aug 2004 - Nov 2007
placebo
Cohort 1 (n=70) 1:1 randomization
40 ug/kg rVIIa
Cohort 2a (n=51)1:2 randomization
placebo 80 ug/kg rVIIa
Cohort 3 (planned, but not done)Steering/safety Comm advised against
Placebo vs. 160ug/kg
Cohort 2b (n=51)DSMB required repeat due to safety concerns
placebo 80 ug/kg rVIIa
39
Primary endpoint
The primary endpoint was the incidence of critical serious adverse events (death, myocardial infarction, stroke, and venous thromboembolic complications)
Note: Don’t get too excited – the study is stopped early before we get the answer (maybe)
40
Patient population
Inclusion criteria: Pt admitted to CVICU 30 min (excluded patients
bleeding intraop) Post-op bleeding into drains in the cardio-thoracic cavity:
200 ml/hr or 2ml/kg for 2 consecutive hours Urgent re-op not required ‘per investigator judgment’Examples of exclusion criteria: History of CVA/DVT/PE Hereditary thrombophilia VAD, ECMO, aortic arch +/or descending thoracic aorta 1st time CABG + none or only 1 antiplatelet medication
within 5 days Unacceptable thrombotic risk ‘as per site investigator’
41
Patient outcome
Randomized (n=179)
Randomized and dosed (n=172)
Randomized and not dosed (n=7)
Placebo (n=68)Death (n=4, 6%)
40 ug/kg (n=35)Death (n=4, 11%)
80 ug/kg (n=69)Death (n=6, 9%)
Study terminated prior to introduction of cohort 3“based on the data within the expanding cardiac literature in which doses of rVIIa were in the range of 60 ug/kg” ???The time to drug administration was 2.8 hrs arriving in ICU
Consented (n=2619) 6.8%
42
Overview of adverse events
SAEs Placebo (68)
40 ug/kg (35)
80 ug/kg (69)
Death 4 (6) 4 (11) 6 (9)
Cerebral infarction
0 2 (6) 2 (3)
MI 1 (2) 3 (9) 1 (1)
PE 0 0 0
Other TEs
0 1 (3) 2 (3)
P 0.25 0.43
OR 2.16 (.6-8.1)
1.61 (.5-5.3)
43
Sample size calculationPoorly written Sample size was chosen to have <20% risk of
seeing >14 (of 35) on active versus <7 (of 35) on placebo in cohort 1 What they saw: 5/68 (7.35%) placebo vs. 10/35
(28.57%) treatment – 21.22% absolute risk increase <16.7% risk of 13 (of 34) on active versus 2 (of
17) on placebo or 7 on active versus 8 on placebo in cohorts 2a, 2b and 3, all assuming no differences and 21 events in cohort 1 and 15 events in cohorts 2a, 2b, and 3. What they saw: 5/68 (7.35%) placebo vs. 11/69
(15.94%) treatment – 8.59% absolute risk increase
Data on 1st 35 controlNot provided
44
Sample size - continued
“Additionally, the sample size was chosen to give adequate power to detect a 35% reduction in the need for any allogeneic transfusions.” The power for the efficacy evaluation is based on a
comparison of (all) placebo patients with the highest dose of rFVIIa (ie, cohort 3) – never done
11% (placebo) vs. 32% (40 ug/kg) vs. 27% (80 ug/kg) – 20% absolute risk reduction
“This simple comparison between 2 groups (86 on placebo versus 34 on rFVIIa) then has 80% power assuming 80% transfusion rate on placebo and a 35% reduction to 52%” (89% placebo, 68% on 40 ug/kg, 73% on 80 ug/kg)
45
Placebo Drug
Any SAE* 5 21
No SAE 63 83
Total 68 104
* SAE = Death, Cerebral Infarction, MI, PE, or other TE
p=0.0284, Fisher
r7a was associated with a statistically higher frequency of Serious Adverse Events
46
Efficacy
placebo
40 ug/kg
80 ug/kg
Re-op for bleeding (%) 25 14
P=0.21
12
P=0.04
Volume allogeneic blood (ml)
825 640
P=0.05
500
P=0.04
Drainage from intra-thoracic cavity 15mins-4hrs post dose (ml/hr)
51 35
P=0.76
24
P=0.018
47
Re-op rates
48
Avoiding RBCs after treatment
49
Major limitations
Statistically increased serious adverse events in rVIIa groups, although they failed to do a simple Fisher Exact test
Potential efficacy only seen at 80 ug/kg dose Very select group of 6.8% of their population Limitations:
Small sample size Industry sponsored trial Authors “compensated” by company Study never completed
50
Outcome before intervention(a.k.a. Temporal ambiguity)
Transfusion and pneumonia in the trauma
intensive care unit: An examination of the
temporal relationship. Vandromme MJ, McGwin G,
Marques MB, et al. J Trauma 2009; 67: 97-101
51
Background
Retrospective studies have found an association between transfusion in ICU patients and pneumonia
Associations between transfusion and ALI, MOF, tumor recurrence and mortality
These reports have been used as evidence to support the notion of ‘transfusion-related immunomodulation’ or TRIM
There are 2 limitations with these studies1. Retrospective and therefore this association may not
represent true causation2. The timing of transfusion is often unknown and
therefore potentially some of these patients developed pneumonia before they even received the first unit of RBCs
52
Causation or Association?
53
Design
Patients admitted to the trauma ICU at the University of Alabama University Hospital between 2004-07 who had an overall length of stay of >4 days, and who had spent 1 or more days in the ICU receiving mechanical ventilation
All transfusions to these patients were also categorized as ‘young’ (<14 days) or ‘old’ (>14 days) – as if something magical happens on day 14
All RBC units were prestorage leukoreduced The incidence of pneumonia was the primary outcome of
interest, and was defined as positive culture >105 cfu/mL on bronchoalveolar lavage
The analysis of the effect of the transfusion on pneumonia was performed for all units transfused (like all the studies that came before), and then only for units that were transfused before the first episode of pneumonia
54
Patient population
1,615 patients met their study criteria Of these patients, 73% were transfused at least 1 U
of RBC The mean number of units transfused per transfused
patient was 8.94 units A total of 270 (16.7%) patients developed
pneumonia The population was as expected for a civilian trauma
cohort (age of 40 years, 74% male, 81% with a blunt injury)
The length of stay was 22 days, with a mean duration of ventilation of 12 days
55
Pneumonia analysis
The overall adjusted relative risk for pneumonia for all transfusions was 1.99 (95% CI 1.39-2.86)
When transfusions that occurred after the diagnosis of pneumonia were excluded, the adjusted relative risk for pneumonia was no longer statistically significant (1.33, 95% CI 0.98-1.80)
56
Age of blood
They also analyzed the effect of different ages of blood on the risk of pneumonia This analysis was problematic because those who
received exclusively young blood were transfused only a mean of 4.24 units, compared to 4.74 units for those receiving old blood, and 12.07 units for those receiving a combination of old and young units (different populations of patients)
The only statistically significant relative risk they could find was that the receipt of exclusively old blood, compared to no blood, increased the risk of pneumonia, albeit only slightly (RR 1.42, 95% CI 1.01-2.02)
There was no difference between patients who received exclusively young, when compared to those who received exclusively old blood
Statistically significant ‘shopping’ may be going on here
57
Author’s conclusions
The finding of an association between transfusion and pneumonia may be an erroneous one, reflecting transfusions received as a consequence of developing pneumonia during a long stay in ICU
These authors methodologically improved on previous TRIM reports by excluding transfusions that occur after pneumonia onset, and not surprisingly the effect is less prominent
58
Some tips when reading
Read from cover to cover – don’t be an abstract reader
Think really hard about how they got their results if the results are surprising or unexpected or too good to be true
Look for statistically significant ‘shopping’ Do your own calculations Check tables for errors Read the stats section and look for missing
info
59
Any questions?
Top Related