Empirische wetenschap onder vuur
description
Transcript of Empirische wetenschap onder vuur
![Page 1: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/1.jpg)
Empirische wetenschaponder vuur
SW Alumni dag March 22nd 2014 @TimSmitsTimhttp://www.slideshare.net/timsmitstim
![Page 2: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/2.jpg)
Vuur. Of toch veel rook.
![Page 3: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/3.jpg)
Diederik Stapel
Dirk Smeesters
Yoshitaka Fuji
![Page 4: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/4.jpg)
Diederik Stapel
Dirk Smeesters
Yoshitaka Fuji
![Page 5: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/5.jpg)
![Page 6: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/6.jpg)
Heel veel rook dus.Maar wat brand er eigenlijk?
![Page 7: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/7.jpg)
FRAUD
THE POPE ; JESUS
Diederik Stapel (50)
Yoshitaka Fuji (183)
Dirk Smeesters
We? (communication sciences; social sciences, KULeuven)
Data fabricationPaper duplicationPlagiarismLack of IRB approval
P-hackingFile drawerOne-sided lit. reviewBiased content analysisBiased interviews
Other questionable research practices (e.g., not understanding science)
![Page 8: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/8.jpg)
OPINIE…
… Het is geen probleem van individuele fraudeurs. Die zullen er altijd zijn en de kans dat het ongestraft gebeurt, is kleiner dan ooit.
… Het is geen discipline-specifiek probleem (sociale psychologie) of een bepaalde onderzoeksmethode (experimenteel; kwantitatief)(Stroebe, Postmes & Spears, 2012)
![Page 9: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/9.jpg)
DUS … IS HET SYSTEMISCH
- Publish or perish. Met beoordeling op ondermaatse metingen: aantal publicaties.
![Page 10: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/10.jpg)
![Page 11: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/11.jpg)
DUS … IS HET SYSTEMISCH (2)
- Publicatiebias en significantie-fetish (maar dat is slechts deels systemisch)
![Page 12: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/12.jpg)
(SYSTEMISCHE) OPLOSSINGEN of EVOLUTIES:
-Retractions van frauduleuze of questionable papers; mega-correcties van gepubliceerde artikels
-Onderzoek over fraude en QRP (questionable research practices)
-Oproep tot replicatie studies; pre-registratie van onderzoek & pre-acceptatie van artikels
-Open science networks: bv. publication/replication depositories – open science framework (osf.org)
-Post-publication review: blogs, Twitter, etc.; HIBAR (Had I Been A Reviewer)
-Sancties voor fraudeurs
-…
![Page 13: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/13.jpg)
![Page 14: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/14.jpg)
From retractionwatch.wordpress.com
![Page 15: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/15.jpg)
![Page 18: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/18.jpg)
![Page 19: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/19.jpg)
![Page 20: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/20.jpg)
![Page 22: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/22.jpg)
![Page 24: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/24.jpg)
DRINGEND GEZOCHT: GEZOND VERSTANDSoms is onderzoek gewoon “TOO GOOD TO BE TRUE”
Extreem voorbeeld: Greg Francis’ research (ook onder kritiek a.o. Uri Simonsohn)
***For the following slides:ALL CREDITS to Greg’s presentation on Febr 5th 2013 in Brussels***
![Page 25: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/25.jpg)
Experimental methods• Suppose you hear about two sets of experiments that investigate
phenomena A and B• Which effect is more believable?
Effect A Effect B
Number of experiments 10 19
Number of experiments that reject H0
9 10
Replication rate 0.9 0.53
![Page 26: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/26.jpg)
• Effect A is Bem’s (2011) precognition study that reported evidence of people’s ability to get information from the future– I do not know any scientist who believes this effect is real
• Effect B is from a meta-analysis of a version of the bystander effect, where people tend to not help someone in need if others are around– I do not know any scientist who does not believe this is a real effect
• So why are we running experiments?
Effect A Effect B
Number of experiments 10 19
Number of experiments that reject H0
9 10
Replication rate 0.9 0.53
![Page 27: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/27.jpg)
Hypothesis testing (for means)• We start with a null hypothesis: no effect, H0
• Identify a sampling distribution that describes variability in a test statistic
![Page 28: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/28.jpg)
Hypothesis testing (for two means)• We can identify rare test statistic values as those in the tail of the sampling distribution
• If we get a test statistic in either tail, we say it is so rare (usually 0.05) that we should consider the null hypothesis to be unlikely
• We reject the null
H0
![Page 29: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/29.jpg)
Alternative hypothesis• If the null hypothesis is not true, then the data came from some other
sampling distribution (Ha)
H0 Ha
![Page 30: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/30.jpg)
Power• If the alternative hypothesis is true
• Power is the probability you will reject H0
• If you repeated the experiment many times, you would expect to reject H0 with a proportion that
reflects the power
H0 Ha
![Page 31: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/31.jpg)
Power• Use the pooled effect size to compute the pooled power of each
experiment (probability this experiment would reject the null hypothesis)
• Pooled effectsize – g*=0.1855
• The sum of the power
values (E=6.27) is the
expected number of times
these experiments would
reject the null hypothesis (Ioannidis & Trikalinos, 2007)
Sample size
Effect size (g)
Power
Exp. 1 100 0.249 0.578
Exp. 2 150 0.194 0.731
Exp. 3 97 0.248 0.567
Exp. 4 99 0.202 0.575
Exp. 5 100 0.221 0.578Exp. 6 Negative 150 0.146 0.731
Exp. 6 Erotic 150 0.144 0.731
Exp. 7 200 0.092 0.834
Exp. 8 100 0.191 0.578
Exp. 9 50 0.412 0.363
![Page 32: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/32.jpg)
Take-home-message of Greg’s studies
-The file drawer phenomenon might be immense. Don’t put your money on published studies-Think not only about the p of your failed studies, but also their power. -For most studies in our discipline, there is about a 50% chance to discover an true phenomenon (since many studies are underpowered)-Increase your N per hypothesis! It increases your “power” to discover an effect (Ha= true) and (a bit) to refute an effect’s existence (H0= true)
Note: To “detect” that men weigh more than women at an adequate power of .8, you need to have n=46!!! (Simmons et al., 2013). Are we studying effects that are stronger than men outweighing women??
![Page 33: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/33.jpg)
Fout begrip van wetenschap
Bij publiek, journalistiekén wetenschap
![Page 36: Empirische wetenschap onder vuur](https://reader035.fdocuments.net/reader035/viewer/2022062405/554e8e41b4c90573338b4c2a/html5/thumbnails/36.jpg)
Wetenschap…
… zal een stuk saaier moeten worden. Minder nieuwigheden, meer repliceren en synthetiseren
… zal een stuk minder duidelijk moeten worden. Gewicht van conclusies in functie van effectgrootte, aantal studies, omstandigheden