Unsw reproducibility workshop

19
Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php A new era of large-scale replication and collaboration? [email protected] @ceptional http://www.slideshare.net/holcombea/

Transcript of Unsw reproducibility workshop

Page 1: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

A new era of large-scale replication and collaboration?

[email protected]

@ceptional

http://www.slideshare.net/holcombea/

Page 2: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

False positives zone Look for areas wherein victory declared whenever null hypothesis is rejected, where size of effect, plausibility considered unimportant.

Where science is not self-correcting

Page 3: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

"This one's Psych Science. 1998. Put your money on ‘will replicate’?”

"What, you think I was born yesterday?”

"Fine," muttered the bookie.

#AcademicNoir

Page 4: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Well, Mr. P. Hacker, your luck is about to run out… down at the journals, they’re running a new game. They call it ‘preregistration’.

Page 5: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

5

https://osf.io/tvyxz/wiki/home/

Page 6: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

6

https://osf.io/tvyxz/wiki/home/

Page 7: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

7

https://osf.io/tvyxz/wiki/home/

100% P-hacking FreeHere, check our numbers.

Here’s how you can replicate our result.

Page 8: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

"This one's Psych Science. 1998. Put your money on ‘will replicate’?”

"What, you think I was born yesterday?”

"Fine," muttered the bookie.

Page 9: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 9

Page 10: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 10

Failed to replicate my result?

Have you considered

the unknown moderators?

Page 11: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 11

Any one of us alone, he’ll say we messed up. But if we all run the replication together, we

can show the big guy we’re right.

Page 12: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Meta-analysis forest plot

Page 13: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Meta-analysis forest plot

Failed to replicate my result?

Have you considered the unknown moderators?

You messed up!

Page 14: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Page 15: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Page 16: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Page 17: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

PreregistrationLarge-scale replicationOpen dataOpen protocol and code

A new era of large-scale replication

and collaboration?

Can we do this bottom-up?

Page 18: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Page 19: Unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php