Responding to reproducibility challenges: from physics to social … · 2019-10-15 ·...
Transcript of Responding to reproducibility challenges: from physics to social … · 2019-10-15 ·...
![Page 1: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/1.jpg)
Responding to reproducibility challenges:
from physics to social sciences
Ana Trisovic IQSS, Harvard University
![Page 2: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/2.jpg)
Agenda
• Reproducibility and replicability
• Preliminary results reveal that more than 75% of deposited research code (in R) is not easily re-executable
• Possible reproducibility solutions for
• social science research
• particle physics research
![Page 3: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/3.jpg)
Reproducibility and replicability
• "Reproducibility (computational) is obtaining consistent results using the same input data, computational steps, methods and code"
• "Replicability is obtaining consistent results across studies aimed at answering the same scientific questions, each of which has obtained its own data"
National Academies of Sciences, Engineering, and Medicine. 2019. https://doi.org/10.17226/25303.
![Page 4: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/4.jpg)
Necessary conditions for reproducibility
• Research code
• Code dependencies (packages, system dependencies etc.)
• Input Data
• Research workflow (i.e., sequence of analysis steps)
• Other (computer infrastructure dependencies, OS, contextual information etc.)
![Page 5: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/5.jpg)
Current practice of research preservation for reproducibility • Typically code and data are preserved, together with
instructions and metadata
![Page 6: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/6.jpg)
A reproducibility study• Goal: find success rate of easily re-executable research code
that uses R programming language and is publicly available
• Mimics the situation in which a third-party researcher downloads a dataset, installs the used packages and tries to rerun the code
• R programming language
• free software environment for statistical computing and graphics
• widely used among statisticians, data scientists and academics
Inspired by Coding BetteR: Assessing and Improving the Reproducibility of R-Based Research with containR by Chris Chen
![Page 7: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/7.jpg)
Dataverse
a free and open-source web application to preserve, share, explore, analyse research data. 48 institutions around the globe run Dataverse
instances as their official data repository.
![Page 8: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/8.jpg)
Implementation• One job executes one Dataverse
dataset
• A dataset contains: code, data, documentation
• Allocated time to run one job is 1 hour
• Some jobs ran for 17 hours or longer
• The study is implemented with AWS Batch
• 3685 R files from 1568 Dataverse datasets were studied
![Page 9: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/9.jpg)
DOI
(Digital Object Identifier) is a globally unique persistent identifier that references digital objects.
![Page 10: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/10.jpg)
ImplementationList of DOIs
![Page 11: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/11.jpg)
ImplementationList of DOIs
![Page 12: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/12.jpg)
ImplementationList of DOIs
DOI
![Page 13: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/13.jpg)
ImplementationList of DOIs
DOI
![Page 14: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/14.jpg)
Results
![Page 15: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/15.jpg)
Results
![Page 16: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/16.jpg)
Results
![Page 17: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/17.jpg)
Re-execution rate• Re-execution rate of 3208 R
files• Re-execution rate of 1091
Dataverse datasets
![Page 18: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/18.jpg)
Re-execution rate• Re-execution rate of 3208 R
files• Re-execution rate of 1091
Dataverse datasets
![Page 19: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/19.jpg)
Re-execution rate per year of publishing
![Page 20: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/20.jpg)
Re-execution rate per field of study
![Page 21: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/21.jpg)
Re-execution rate per publisher
![Page 22: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/22.jpg)
Discussion• Limitations of the language
• R is not backward compatible
• setwd considered a good practice in R
• More sophisticated analysis
Error in setwd(~/Dropbox/Pollution and Public Perceptions in China/Data and Analysis/Data) : [newline] cannot change working directory Error in setwd(~/Dropbox/Current projects/1953/Replication archive) : [newline] cannot change working directory Error in setwd(C:/Users/Shawna/Desktop/PA replic) : [newline] cannot change working directory Error in setwd(/Users/ericguntermann/Documents/Papers/Representation of Party Preferences/Replication) : [newline] cannot change working directory Error in setwd(~replication) : cannot change working directory ...
![Page 23: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/23.jpg)
Discussion• Limitations of the language
• R is not backward compatible
• setwd considered a good practice in R
• More sophisticated analysis
Error in setwd(~/Dropbox/Pollution and Public Perceptions in China/Data and Analysis/Data) : [newline] cannot change working directory Error in setwd(~/Dropbox/Current projects/1953/Replication archive) : [newline] cannot change working directory Error in setwd(C:/Users/Shawna/Desktop/PA replic) : [newline] cannot change working directory Error in setwd(/Users/ericguntermann/Documents/Papers/Representation of Party Preferences/Replication) : [newline] cannot change working directory Error in setwd(~replication) : cannot change working directory ...
![Page 24: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/24.jpg)
Mini summary:
The way in which we currently deposit research code in data repositories is not adequate to guarantee its re-execution. On average, only about 15% of the R
code files can be easily re-executed.
![Page 25: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/25.jpg)
Characterising social science research
• Typically small research groups
• Use both survey data and big data
• Often use sensitive data and proprietary software
• Workflows include the use of:
• Dropbox
• git
• sometimes continuous integration
![Page 26: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/26.jpg)
What went wrong?
• There is a culture for sharing research materials
• Often imposed by top journals and funding bodies
• Computational and system dependencies are often overlooked
![Page 27: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/27.jpg)
Possible solution• Container-based reproducibility platforms
• Allow seamless use of the container technology
• Containers capture necessary system dependencies and can vastly help with reproducibility
• Examples:
• Whole Tale
• Code Ocean (proprietary software)
• Binder (Jupyter)
![Page 28: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/28.jpg)
Docker containers provide a OS-level virtualisation that bundle software, libraries and configuration
files in an isolated environment. They are typically more lightweight than virtual machines.
![Page 29: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/29.jpg)
The Whole Tale project
• A representation of the entire research activity: data, software, workflows, provenance into a single executable package
• External data sources: Globus, DataONE, Dataverse
• http://wholetale.org
Metadata
Frontends
Environment
Provenance
Data
Published
![Page 30: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/30.jpg)
![Page 31: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/31.jpg)
Code Ocean• Research collaboration platform integrated into a web
browser
• Encourages users to define research workflow
• Support for a number of programming languages:
• python
• R
• stata
• MATLAB
![Page 32: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/32.jpg)
Code Ocean
![Page 33: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/33.jpg)
Binder
![Page 34: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/34.jpg)
Functionality of reproducibility platforms
• Interactive environments
• Provide versioning and provenance tools
• Support for verification, review, licensing
• Assigning permissions (for accessing proprietary data)
• Support cloning and republishing
![Page 35: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/35.jpg)
Integration with Dataverse
Data +Code
+ Reproducible Capsule
Data +Code
Reproducibility verification
Data +Code
Execute and curate code
By Mercè Crosas @mercecrosas
![Page 36: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/36.jpg)
Mini summary:
The use of container- and cloud-based reproducibility platforms is a promising solution that can adequately preserve a wide variety of
research studies across sciences.
![Page 37: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/37.jpg)
Characterising particle physics research
• Large experiments producing petabytes of data
• Large collaborations
![Page 38: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/38.jpg)
Characterising particle physics research
• Typically use free and open source software
• Data to be released open access due to funding-body policies
Source: https://github.com/atrisovic/thesis-plots
Data production Data analysis
PB TB GB MBDat
a co
mpl
exity
PublicationsRaw experimental data
Data size
![Page 39: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/39.jpg)
Characterising particle physics research
• Typically use free and open source software
• Data to be released open access due to funding-body policies
Source: https://github.com/atrisovic/thesis-plots
Data production Data analysis
PB TB GB MBDat
a co
mpl
exity
PublicationsRaw experimental data
Data size
![Page 40: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/40.jpg)
CERN Analysis Preservation• Digital repository for describing and capturing physics
analyses
• Supports integration with databases of the LHC collaborations (analysis databases)
• Supports integration with GitLab, GitHub, DockerHub
![Page 41: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/41.jpg)
![Page 42: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/42.jpg)
![Page 44: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/44.jpg)
Reana supports tools for workflow design and execution
![Page 45: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/45.jpg)
CERN Open Data
![Page 46: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/46.jpg)
Flight 795
(By Tibor Simko)
![Page 47: Responding to reproducibility challenges: from physics to social … · 2019-10-15 · reproducibility • Research code • Code dependencies (packages, system dependencies etc.)](https://reader034.fdocuments.net/reader034/viewer/2022050410/5f8799c8b5cb3137ff1ab01a/html5/thumbnails/47.jpg)
Conclusions
• Computational reproducibility is a hard problem that is likely becoming worse
• Container-based reproducibility platforms as a possible solution
• Some disciplines (like particle physics) require tailored solutions