Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 ·...
Transcript of Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 ·...
![Page 1: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/1.jpg)
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤
❆r♥❛✉❞ ▲❡❣r❛♥❞
❏♦✉r♥é❡ ❚r❛❝❡s✱ ●r❡♥♦❜❧❡▼❛r❝❤ ✷✹✱ ✷✵✶✺
❆r♥❛✉❞ ▲❡❣r❛♥❞ ✶ ✴ ✶✼
![Page 2: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/2.jpg)
❆ ❋❡✇ ❊❞✐❢②✐♥❣ ❊①❛♠♣❧❡s
◆❛✐❝❦❡♥✱ ❙t❡♣❤❡♥ ❡t ❆❧✳✱ ❚♦✇❛r❞s ❨❡t ❆♥♦t❤❡r P❡❡r✲t♦✲P❡❡r❙✐♠✉❧❛t♦r✱ ❍❊❚✲◆❊❚s✬✵✻✳
From 141 P2P sim.papers, 30% use a custom tool,50% don’t report used tool
25
50
75
0/100
Type
Chord−(SFS)
Custom
NS−2
Other
Unspecified
Simulator usage [Naicken06]
❈♦❧❧❜❡r❣✱ ❈❤r✐st✐❛♥ ❡t ❆❧✳✱ ▼❡❛s✉r✐♥❣ ❘❡♣r♦❞✉❝✐❜✐❧✐t② ✐♥ ❈♦♠♣✉t❡r ❙②st❡♠s ❘❡s❡❛r❝❤✱ ❤tt♣✿✴✴r❡♣r♦❞✉❝✐❜✐❧✐t②✳❝s✳❛r✐③♦♥❛✳❡❞✉✴
601
NC63
HW30
508
Article85
Web54
EMyes
87
EX106
226
OK≤30
130OK
>30
64OK
A u
23
Buildfails9
176
EMno
146EM
∅
30
8 ACM conferences (❆❙P▲❖❙✬✶✷✱❈❈❙✬✶✷✱ ❖❖P❙▲❆✬✶✷✱ ❖❙❉■✬✶✷✱ P▲❉■✬✶✷✱
❙■●▼❖❉✬✶✷✱ ❙❖❙P✬✶✶✱ ❱▲❉❇✬✶✷) and 5journals
EM♥♦= the code cannot be provided
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✷ ✴ ✶✼
![Page 3: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/3.jpg)
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Thanks for your interest in the implementation of our paper. The goodnews is that I was able to find some code. I am just hoping that it isa stable working version of the code, and matches the implementationwe finally used for the paper. Unfortunately, I have lost some data whenmy laptop was stolen last year. The bad news is that the code is notcommented and/or clean.
Attached is the 〈system〉 source code of our algorithm. I’m not very surewhether it is the final version of the code used in our paper, but it shouldbe at least 99% close. Hope it will help.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
![Page 4: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/4.jpg)
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Unfortunately, the server in which my implementation was stored had adisk crash in April and three disks crashed simultaneously. While the helpdesk made significant effort to save the data, my entire implementationfor this paper was not found.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
![Page 5: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/5.jpg)
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Unfortunately the current system is not mature enough at the moment,so it’s not yet publicly available. We are actively working on a numberof extensions and things are somewhat volatile. However, once thingsstabilize we plan to release it to outside users. At that point, we wouldbe happy to send you a copy.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
![Page 6: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/6.jpg)
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
I am afraid that the source code was never released. The code was neverintended to be released so is not in any shape for general use.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
![Page 7: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/7.jpg)
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
〈STUDENT〉 was a graduate student in our program but he left a whileback so I am responding instead. For the paper we used a prototypethat included many moving pieces that only 〈STUDENT〉 knew how tooperate and we did not have the time to integrate them in a ready-to-share implementation before he left. Still, I hope you can build on theideas/technique of the paper.
Unfortunately, the author who has done most of the coding for this paperhas passed away and the code is no longer maintained.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
![Page 8: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/8.jpg)
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Since this work has been done at 〈COMPANY〉 we don’t open-sourcecode unless there is a compelling business reason to do so. So unfortu-nately I don’t think we’ll be able to share it with you.
The code owned by 〈COMPANY〉, and AFAIK the code is not open-source. Your best bet is to reimplement :( Sorry.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
![Page 9: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/9.jpg)
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Unfortunately, the 〈SYSTEM〉 sources are not meant to be opensource(the code is partially property of 〈UNIVERSITY 1〉, 〈UNIVERSITY 2〉and 〈UNIVERSITY 3〉.)
If this will change I will let you know, albeit I do not think there is anintention to make the 〈SYSTEM〉 sources opensource in the near future.
If you’re interested in obtaining the code, we only ask for a descriptionof the research project that the code will be used in (which may leadto some joint research), and we also have a software license agreementthat the University would need to sign.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
![Page 10: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/10.jpg)
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
In the past when we attempted to share it, we found ourselves spendingmore time getting outsiders up to speed than on our own research. SoI finally had to establish the policy that we will not provide the sourcecode outside the group.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
![Page 11: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/11.jpg)
2 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Data Intensive
Obtain
Data
Analyze/
Visualize
User studies
Sensors
Web
Databases
Simulations
Publish/
Share
Sequencing
machines
Particle
colliders
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✹ ✴ ✶✼
![Page 12: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/12.jpg)
3 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Data + Computing Intensive
Obtain
Data
Analyze/
Visualize
User studies
Sensors
Web
Databases
Simulations
Publish/
Share
Sequencing
machines
Particle
colliders
AVS
Taverna
VisTrails
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✺ ✴ ✶✼
![Page 13: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/13.jpg)
4 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Data + Computing Intensive
Obtain
Data
Analyze/
Visualize
User studies
Sensors
Web
Databases
Simulations
Publish/
Share
Sequencing
machines
Particle
colliders
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✻ ✴ ✶✼
![Page 14: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/14.jpg)
5 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Data + Computing Intensive
Obtain
Data
Analyze/
Visualize
User studies
Sensors
Web
Databases
Simulations
Publish/
Share
Sequencing
machines
Particle
colliders
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✼ ✴ ✶✼
![Page 15: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/15.jpg)
6 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Incomplete Publications
Publications are just the tip of the iceberg
- Scientific record is incomplete---
to large to fit in a paper
- Large volumes of data
- Complex processes
Can’t (easily) reproduce results
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✽ ✴ ✶✼
![Page 16: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/16.jpg)
7 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Incomplete Publications
Publications are just the tip of the iceberg
- Scientific record is incomplete---
to large to fit in a paper
- Large volumes of data
- Complex processes
Can’t (easily) reproduce results
“It’s impossible to verify most of the results that computational scientists present at conference
and in papers.” [Donoho et al., 2009]
“Scientific and mathematical journals are filled
with pretty pictures of computational experiments
that the reader has no hope of repeating.” [LeVeque, 2009]
“Published documents are merely the
advertisement of scholarship whereas the
computer programs, input data, parameter
values, etc. embody the scholarship itself.” [Schwab et al., 2007]
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✾ ✴ ✶✼
![Page 17: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/17.jpg)
❊✈✐❞❡♥❝❡ ❢♦r ❛ ▲❛❝❦ ♦❢ ❘❡♣r♦❞✉❝✐❜✐❧✐t②
❙t✉❞✐❡s s❤♦✇✐♥❣ t❤❛t s❝✐❡♥t✐✜❝ ♣❛♣❡rs ❝♦♠♠♦♥❧② ❧❡❛✈❡ ♦✉t ❡①♣❡r✐♠❡♥t❛❧❞❡t❛✐❧s ❡ss❡♥t✐❛❧ ❢♦r r❡♣r♦❞✉❝t✐♦♥ ❛♥❞ s❤♦✇✐♥❣ ❞✐✣❝✉❧t✐❡s ✇✐t❤ r❡♣❧✐❝❛t✐♥❣♣✉❜❧✐s❤❡❞ ❡①♣❡r✐♠❡♥t❛❧ r❡s✉❧ts✿
J.P. Ioannidis. Why Most Published Research Findings Are False PLoSMed. 2005 August; 2(8)
❍✐❣❤ ♥✉♠❜❡r ♦❢ ❢❛✐❧✐♥❣ ❝❧✐♥✐❝❛❧ tr✐❛❧s✳
Do We Really Know What Makes Us Healthy?, New-York Times —September 16, 2007Lies, Damned Lies, and Medical Science, The Atlantic. 2010, Nov.
■♥❝r❡❛s❡ ✐♥ r❡tr❛❝t❡❞ ♣❛♣❡rs✿
Steen RG, Retractions in the scientific literature: is the incidence ofresearch fraud increasing? J Med Ethics 37: 249–253.
❈♦✉rt❡s② ♦❢ ❆❞❛♠ ❏✳ ❘✐❝❤❛r❞s❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✶✵ ✴ ✶✼
![Page 18: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/18.jpg)
❊①♣❡r✐♠❡♥t✐♥❣ ✐♥ t❤❡ ❲✐❧❞
❊①♣❡r✐♠❡♥ts ✐♥ ❍P❈✿ ❊✈❡♥ ❲♦rs❡✦
❘❡❧② ♦♥ ❧❛r❣❡✱ ❞✐str✐❜✉t❡❞✱ ❤②❜r✐❞✱ ♣r♦t♦t②♣❡ ❤❛r❞✇❛r❡✴s♦❢t✇❛r❡
▼❡❛s✉r❡ ❡①❡❝✉t✐♦♥ t✐♠❡s ✭♠❛❦❡s♣❛♥s✱ tr❛❝❡s✱ ✳ ✳ ✳ ✮
▼❛♥② ♣❛r❛♠❡t❡rs✱ ✈❡r② ❝♦st❧② ❛♥❞ ❤❛r❞ t♦ r❡♣r♦❞✉❝❡
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❲❤❛t ❛❜♦✉t ✉s ❄ ✶✶ ✴ ✶✼
![Page 19: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/19.jpg)
❘❡♣r♦❞✉❝✐♥❣ ✐♥ t❤❡ ❲✐❧❞
■♥ ♠② ❈♦♠♠✉♥✐t②
❘❡♣r♦❞✉❝✐♥❣ r❡s✉❧ts ♦❢ ♦t❤❡rs ✐s ♥♦t ✐♥ t❤❡ ❝✉❧t✉r❡
❘❡✇❛r❞s ❢♦r ♣♦s✐t✐✈❡ r❡s✉❧ts ❛♥❞ ♥♦✈❡❧t② r❛t❤❡r t❤❛♥ ❝♦♥s♦❧✐❞❛t✐♥❣ ❡①✐st✲✐♥❣ r❡s✉❧ts
❊①❝✉s❡ ★✶✿ ❍❛r❞✇❛r❡ ❛♥❞ s♦❢t✇❛r❡ ❡✈♦❧✈❡ ✈❡r② q✉✐❝❦❧②
❊①❝✉s❡ ★✷✿ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤ ✐s ♥♦t ♠❛♥❞❛t♦r②
❲❤② ✐s ✐t s♦ ❍❛r❞ ✐♥ ●❡♥❡r❛❧❄
❈♦♣②r✐❣❤t✴❝♦♠♣❡t✐t✐♦♥ ✐ss✉❡ ✭✇♦✉❧❞ ❣✐✈❡ ❛❞✈❛♥t❛❣❡ t♦ ✧♦t❤❡rs✧✮
P✉❜❧✐❝❛t✐♦♥ ❜✐❛s ✭♦♥❧② t❤❡ ✐❞❡❛ ♠❛tt❡rs✱ ♥♦t t❤❡ ❣♦r② ❞❡t❛✐❧s✮
❊①♣❡r✐♠❡♥t❡r ❜✐❛s
Pr♦❣r❛♠♠✐♥❣ ❡rr♦rs ♦r ❞❛t❛ ♠❛♥✐♣✉❧❛t✐♦♥✴❛♥❛❧②s✐s ♠✐st❛❦❡s
▲❛❝❦ ♦❢ ❡❛s②✲t♦✲✉s❡ t♦♦❧s
❲❤② ❜♦t❤❡r❄❆r♥❛✉❞ ▲❡❣r❛♥❞ ❲❤❛t ❛❜♦✉t ✉s ❄ ✶✷ ✴ ✶✼
![Page 20: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/20.jpg)
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣
❘❡❛❞❡r
❆✉t❤♦r
✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮
Pr♦t♦❝♦❧
❙❝✐❡♥t✐✜❝
◗✉❡st✐♦♥
P✉❜❧✐s❤❡❞
❆rt✐❝❧❡
◆❛t✉r❡✴❙②st❡♠✴✳✳✳
■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✸ ✴ ✶✼
![Page 21: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/21.jpg)
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣
❆♥❛❧②t✐❝
❉❛t❛
❈♦♠♣✉t❛t✐♦♥❛❧
❘❡s✉❧ts
▼❡❛s✉r❡❞
❉❛t❛
◆✉♠❡r✐❝❛❧
❙✉♠♠❛r✐❡s
❋✐❣✉r❡s
❚❛❜❧❡s
❚❡①t
❘❡❛❞❡r
❆✉t❤♦r
✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮
Pr♦t♦❝♦❧
❙❝✐❡♥t✐✜❝
◗✉❡st✐♦♥
P✉❜❧✐s❤❡❞
❆rt✐❝❧❡
◆❛t✉r❡✴❙②st❡♠✴✳✳✳
■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✸ ✴ ✶✼
![Page 22: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/22.jpg)
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣
❚r② t♦ ❦❡❡♣ tr❛❝❦ ♦❢ t❤❡ ✇❤♦❧❡ ❝❤❛✐♥
❊①♣❡r✐♠❡♥t ❈♦❞❡
✭✇♦r❦❧♦❛❞ ✐♥❥❡❝t♦r✱ ❱▼ r❡❝✐♣❡s✱ ✳✳✳✮
Pr♦❝❡ss✐♥❣
❈♦❞❡
❆♥❛❧②s✐s
❈♦❞❡
Pr❡s❡♥t❛t✐♦♥
❈♦❞❡
❆♥❛❧②t✐❝
❉❛t❛
❈♦♠♣✉t❛t✐♦♥❛❧
❘❡s✉❧ts
▼❡❛s✉r❡❞
❉❛t❛
◆✉♠❡r✐❝❛❧
❙✉♠♠❛r✐❡s
❋✐❣✉r❡s
❚❛❜❧❡s
❚❡①t
❘❡❛❞❡r
❆✉t❤♦r
✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮
Pr♦t♦❝♦❧
❙❝✐❡♥t✐✜❝
◗✉❡st✐♦♥
P✉❜❧✐s❤❡❞
❆rt✐❝❧❡
◆❛t✉r❡✴❙②st❡♠✴✳✳✳
■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✸ ✴ ✶✼
![Page 23: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/23.jpg)
▼②t❤❜✉st❡rs✿ ❙❝✐❡♥❝❡ ✈s✳ ❙❝r❡✇✐♥❣ ❆r♦✉♥❞
![Page 24: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/24.jpg)
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣✧❚r✐❝❦②✧♣❛rt
✧❊❛s②✧ ♣❛rt
✧❚r✐❝❦②✧ ❛♥❞ ✧❊❛s②✧ r❡❢❡r t♦
♣❛r❛❧❧❡❧ ❝♦♠♣✉t❡r s❝✐❡♥t✐st ✉s❡ ❝❛s❡s
❊①♣❡r✐♠❡♥t ❈♦❞❡
✭✇♦r❦❧♦❛❞ ✐♥❥❡❝t♦r✱ ❱▼ r❡❝✐♣❡s✱ ✳✳✳✮
Pr♦❝❡ss✐♥❣
❈♦❞❡
❆♥❛❧②s✐s
❈♦❞❡
Pr❡s❡♥t❛t✐♦♥
❈♦❞❡
❆♥❛❧②t✐❝
❉❛t❛
❈♦♠♣✉t❛t✐♦♥❛❧
❘❡s✉❧ts
▼❡❛s✉r❡❞
❉❛t❛
◆✉♠❡r✐❝❛❧
❙✉♠♠❛r✐❡s
❋✐❣✉r❡s
❚❛❜❧❡s
❚❡①t
❘❡❛❞❡r
❆✉t❤♦r
✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮
Pr♦t♦❝♦❧
❙❝✐❡♥t✐✜❝
◗✉❡st✐♦♥
P✉❜❧✐s❤❡❞
❆rt✐❝❧❡
◆❛t✉r❡✴❙②st❡♠✴✳✳✳
■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✺ ✴ ✶✼
![Page 25: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/25.jpg)
❙♦ ▼❛♥② ◆❡✇ ❚♦♦❧s
New Tools for Computational Reproducibility
• Dissemination Platforms:
• Workflow Tracking and Research Environments:
• Embedded Publishing:
VisTrails Kepler CDE
Galaxy GenePattern Synapse
Sumatra Taverna Pegasus
Verifiable Computational Research Sweave knitR
Collage Authoring Environment SHARE
ResearchCompendia.org IPOL Madagascar
MLOSS.org thedatahub.org nanoHUB.orgOpen Science Framework The DataVerse Network RunMyCode.org
❈♦✉rt❡s② ♦❢ ❱✐❝t♦r✐❛ ❙t♦❞❞❡♥ ✭❯❈ ❉❛✈✐s✱ ❋❡❜ ✶✸✱ ✷✵✶✹✮
❆♥❞ ❛❧s♦✿ ❋✐❣s❤❛r❡✴❩❡♥♦❞♦✱ ❆❝t✐✈❡P❛♣❡rs✱ ❊❧s❡✈✐❡r ❡①❡❝✉t❛❜❧❡ ♣❛♣❡r✱✳ ✳ ✳
❆r♥❛✉❞ ▲❡❣r❛♥❞ ▼❛♥② ❆❧t❡r♥❛t✐✈❡s ❢♦r ❘❡♣r♦❞✉❝✐❜❧❡ ❆♥❛❧②s✐s ✶✻ ✴ ✶✼
![Page 26: Reproducible Research - imagpolaris.imag.fr/arnaud.legrand/blog/2015/03/24/... · 3/24/2015 · The Dog Ate my Homework !!! Versionning Problems Bad Backup Practices Code Will be](https://reader034.fdocuments.net/reader034/viewer/2022042911/5f442a503888c6153236abdd/html5/thumbnails/26.jpg)
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ t❤❡ ◆❡✇ ❇✉③③✇♦r❞❄
❍✷✵✷✵✲❊■◆❋❘❆✲✷✵✶✹✲✷✵✶✺
❆ ❦❡② ❡❧❡♠❡♥t ✇✐❧❧ ❜❡ ❝❛♣❛❝✐t② ❜✉✐❧❞✐♥❣ t♦ ❧✐♥❦ ❧✐t❡r❛t✉r❡ ❛♥❞ ❞❛t❛
✐♥ ♦r❞❡r t♦ ❡♥❛❜❧❡ ❛ ♠♦r❡ tr❛♥s♣❛r❡♥t ❡✈❛❧✉❛t✐♦♥ ♦❢ r❡s❡❛r❝❤ ❛♥❞
r❡♣r♦❞✉❝✐❜✐❧✐t② ♦❢ r❡s✉❧ts✳
▼♦r❡ ❛♥❞ ♠♦r❡ ✇♦r❦s❤♦♣s
❲♦r❦s❤♦♣ ♦♥ ❉✉♣❧✐❝❛t✐♥❣✱ ❉❡❝♦♥str✉❝t✐♥❣ ❛♥❞ ❉❡❜✉♥❦✐♥❣ ✭❲❉❉❉✮ ✭✷✵✶✹ ❡❞✐t✐♦♥✮
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚♦♦❧s ❛♥❞ ❙tr❛t❡❣✐❡s ❢♦r ❙❝✐❡♥t✐✜❝ ❈♦♠♣✉t✐♥❣✭✷✵✶✶✮
❲♦r❦✐♥❣ t♦✇❛r❞s ❙✉st❛✐♥❛❜❧❡ ❙♦❢t✇❛r❡ ❢♦r ❙❝✐❡♥❝❡✿ Pr❛❝t✐❝❡ ❛♥❞ ❊①♣❡r✐❡♥❝❡s ✭✷✵✶✸✮
❘❡♣r♦❞✉❝✐❜✐❧✐t②❅❳❙❊❉❊✿ ❆♥ ❳❙❊❉❊✶✹ ❲♦r❦s❤♦♣
❚❘❯❙❚ ✷✵✶✹
❘❊PP❆❘✬✶✺✿ ✷♥❞ ■♥t❡r♥❛t✐♦♥❛❧ ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜✐❧✐t② ✐♥ P❛r❛❧❧❡❧ ❈♦♠♣✉t✐♥❣
❘❡♣r♦❞✉❝✐❜✐❧✐t② ♦❢ ❈♦♠♣✉t❛t✐♦♥ ❇❛s❡❞ ❘❡s❡❛r❝❤✴◆❚▼❙✬✶✺
❙❤♦✉❧❞ ❜❡ s❡❡♥ ❛s ♦♣♣♦rt✉♥✐t✐❡s t♦ s❤❛r❡ ❡①♣❡r✐❡♥❝❡✳
r❡❝❤❡r❝❤❡✲r❡♣r♦❞✉❝t✐❜❧❡❅❧✐st❡s✳✉♥✐✈✲♦r❧❡❛♥s✳❢r
❆r♥❛✉❞ ▲❡❣r❛♥❞ ▼❛♥② ❆❧t❡r♥❛t✐✈❡s ❢♦r ❘❡♣r♦❞✉❝✐❜❧❡ ❆♥❛❧②s✐s ✶✼ ✴ ✶✼