THE  IDEA  

THE  DATA  

THE  IMPLICATIONS  

The  Cultural  Evolu8on  of  Ineffec8ve  Medicine  Mícheál  de  Barra1,  Pontus  Strimling  &  Kimmo  Eriksson  

Everyone who is treated

health outcome (+ive = health improved)

Freq

uenc

y

−60 −40 −20 0 20 40 60

020

6010

0

People who tell others about treatment

health outcome (+ive = health improved)

Freq

uenc

y

−60 −40 −20 0 20 40 60

020

6010

0

Cumula8ve   cultural   evolu8on   creates   excellent   solu8ons   to  many  of   life's   problems.  Hun8ng   tools,  

s8tched   clothing,   protec8ve   shelters   and   agricultural   techniques   show   a   gradual   ratche8ng   up   in  

complexity,   efficiency   and   u8lity   (Richerson   and   Boyd   2005).   But   in   another   domains   of   life   the  

process  of  cultural  evolu8on  rarely   leads  to  adap8ve  behavior.  Most  notably,  tradi8onal,  alterna8ve  

and  historical  medical  treatments  are  typically  ineffec8ve  and  oYen  harmful  (Woo[on  2006).    

So   why   do   harmful   or   neutral   treatments   achieve   cultural   success?   The   outcome   of   a   disease   is  

determined   by   many   factors,   and   so   when   a   treatment   is   applied   there   will   be   a   distribu8on   of  

outcomes:   some  people  will   get   be[er   and   some  people  will   get  worse.   If   the   people  with   be[er  

outcomes   are  more   inclined   to   share   their   experience  with   others,   then   a   treatment  may   retain   a  

posi8ve  reputa8on  despite  a  nega8ve  effect.  If  this  is  the  case,  the  reputa8on  of  medical  treatments  

will  be  more  posi8ve  than  the  clinical  evidence  warrants.    

We   tested   this   idea   using   diet   books.   1,360   people   had   reviewed   the   Atkins   diet   book   on  

amazon.com,  and  540  of  these  state  both  an  exact  weight  change  and  a  8me  period  over  which  this  

change  happened  (e.g.  "OMG  I  lost  20lb  in  3  months  -­‐  this  diet  is  fantas8c").  These  reviews  allow  an  

es8ma8on  of  the  diets  real-­‐world  reputa8on.  Four  clinical  trials  have  examined  weight  change  on  the  

Atkins   diet   or   similar   and   recorded   weight   change   at   mul8ple   8me   points.   Despite   somewhat  

different  interven8ons  and  samples,  these  studies  show  a  consistent  pa[ern  of  moderate  loss.  Fig  2  

plots  weight  loss  at  each  8me  period  and  shows  that  the  average  weight  loss  reported  in  amazon.com  

reviews  of  the  Atkins  diet  are  consistently  larger  than  seen  in  clinical  trials.    

after 2 months

05

1015

2025

(n=72)

after 6 months

(n=71)

after 12 months

(n=69)

−40 −30 −20 −10 0 10

05

1015

2025

(n=81)

−40 −30 −20 −10 0 10

(n=47)

−40 −30 −20 −10 0 10

(n=42)

weight change (kg)

frequ

ency

(peo

ple)

a

maz

on.c

omcl

inic

al tr

ial

Atkins diet debiased 17 day−diet debiased

condition

num

ber o

f peo

ple

010

2030

4050

Picked AtkinsPicked 17−day diet

Biased  repor8ng  will  influence  cultural  evolu8on  to  the  extent  that  the  reputa8on  influences  subsequent  decisions.  We  performed  

an  online  experiment  where  each  par8cipant  saw  a  picture  of  The  Atkins  Diet  book  plus  three  reviews,  and  The  17-­‐day  Diet  plus  

three  reviews.  In  one  condi8on  the  Atkins  book  reviews  were  “debiased”  by  (a)  drawing  reviews  from  a  sample  of  reviews  with  an  

average   of   3.5   stars   (this   is   the   average   sa8sfac8on   ra8ng   given   to   diets   in   a   longitudinal   study   by   Baldwin   et   al.   2009)   and   (b)  

changing   the   weight   change   to   the   mean   weight   change   at   that   8me   point   as   measured   in   clinical   trials.   In   the   other   other  

condi8on,  the  17-­‐day  Diet  was  debiased  using  the  same  procedure.  Order  of  diet  presenta8on  was  counterbalanced.  Fig  4.  shows  

that  par8cipants  were  significantly  less  likely  to  pick  the  pick  the  debiased  diet.    

vs,  

Gardner  et  al.  2007  shared  individual   level  data  from  their  clinical  trial  of  311  

women,   including  77  on  the  Atkins  diet.  The   interven8on  entailed  gelng  the  

Atkins  Diet  Book  plus  8  weekly  mee8ngs  with  a  die8cian   to  discuss   the  book  

and   progress.   Weight   change   was   significantly   larger   in   the   amazon.com  

reviews   than   in   the  clinical   trial   (ps  <<   .01).  At  6  months,  27%  of  clinical   trial  

par8cipants   and   88%   of   amazon.com   reviewers   experienced   weight   loss   of  

10kg  or  more.  Larger  variance  in  clinical  outcome  was  associated  with  a  larger  

bias.  

The   online   reputed   benefit   of   weight-­‐loss   diets   is   larger   than   its   real   effect,   most   likely   because  

people  with  typical  or  poorer  outcomes  are  less  likely  to  tell  others  about  their  experiences.  Thus  the  

real-­‐world  reputa8on  of  medical  treatments  may  be  subject  to  a  publica8on  bias  akin  to  that  seen  in  

science   (Easterbrook   et   al.   1991).   This   kind   of   bias   is   more   likely   to   operate   when:   (a)   treatment  

depends   on   word-­‐of-­‐mouth   reputa8on   (b)   treated   individuals   with   poor   outcomes   can   remain  

“invisible”,   and   (c)   outcomes   are   a   func8on   of   many   factors,   ie.   there   is   a   broad   distribu8on   of  

outcomes.  When   this   is   the   case,   ineffec8ve   or   even   harmful   treatments  may  maintain   a   posi8ve  

reputa8on.  This  may  partly  explain  the  persistence  and  prevalence  of  bad  medicine.    

Man  being  harmed  by  a  doctor:  1860s  bloodlelng  photo2.  

Fig  1.  Imagined  data  illustra8ng  how  an  ineffec8ve  treatments  can  seem  beneficial  when  people  with  posi8ve  outcomes  are  more  inclined  to  tell  others  about  experience.  Red  lines  indicate  mean.    

Fig  2.  Average  weight  loss  on  Atkins  diet  reported  in  four  clinical  trials  and  on  amazon.com.    Amazon  data  points  calculated  by  averaging  reviews  diets  of  similar  dura8on.  

0 2 4 6 8 10 12

−30

−25

−20

−15

−10

−50

duration of diet (months)

mea

n ch

ange

in w

eigh

t (kg

)

amazon.comShai et al. 2008Gardner et al. 2007Truby et al. 2006Foster et al. 2003

Fig  3.  Comparison  of  weight  change  in  Gardner  et  al.  2007  (top  row)  and  amazon.com  reviews  (bo[om  row)  at  three  8me  points.    Red  lines  indicate  mean.  

Fig  4.  Which  diet  would  you  choose?  Results  of  experiment  where  par8cipants  choose  between  a  diet  with  typical  reviews  or  a  diet  with  “debiased”  reviews.      

Atkins,  R.  C.  (2002).  Dr.  Atkins'  new  diet  revolu8on,  M.  Evans.  

Baldwin,   A.   S.,   A.   J.   Rothman,   et   al.   (2009).   "Sa8sfac8on  with  weight   loss:   examining   the   longitudinal   covaria8on   between  

people's  weight-­‐loss-­‐related  outcomes  and  experiences  and  their  sa8sfac8on."  Ann  Behav  Med  38(3):  213-­‐24.  

Easterbrook,  P.  J.,  J.  A.  Berlin,  et  al.  (1991).  "Publica8on  bias  in  clinical  research."  Lancet  337(8746):  867-­‐72.  

Foster,  G.  D.,  H.   R.  Wya[,   et   al.   (2003).   "A   randomized   trial   of   a   low-­‐carbohydrate   diet   for   obesity."  N   Engl   J  Med  348(21):  

2082-­‐90.  

Gardner,  C.  D.,  A.  Kiazand,  et  al.   (2007).  "Comparison  of  the  Atkins,  Zone,  Ornish,  and  LEARN  diets   for  change   in  weight  and  

related   risk   factors   among  overweight  premenopausal  women:   the  A  TO  Z  Weight   Loss   Study:   a   randomized   trial."   JAMA  

297(9):  969-­‐77.  

Moreno,  M.  (2011).  The  17  Day  Diet:  A  Doctor's  Plan  Designed  for  Rapid  Results,  Free  Press.  

Richerson,  P.  J.  and  R.  Boyd  (2005).  Not  by  genes  alone:  How  culture  transformed  human  evolu8on.  Chicago,  IL,  US,  University  

of  Chicago  Press.  

Shai,   I.,  D.  Schwarzfuchs,  et  al.   (2008).   "Weight   loss  with  a   low-­‐carbohydrate,  Mediterranean,  or   low-­‐fat  diet."  N  Engl   J  Med  

359(3):  229-­‐41.  

Truby,   H.,   S.   Baic,   et   al.   (2006).   "Randomised   controlled   trial   of   four   commercial  weight   loss   programmes   in   the  UK:   ini8al  

findings  from  the  BBC  "diet  trials"."  BMJ  332(7553):  1309-­‐14.  

Woo[on,  D.  (2006).  Bad  medicine:  doctors  doing  harm  since  Hippocrates,  Oxford  University  Press,  USA.  1  Contact  author  at  mdebarra@gmail.com  2  Image  source:  www.burnsarchive.com/  via  wikipedia.