What is Reproducibility? The R* brouhaha

WhatisReproducibility?TheR*brouhaha(andhowResearchObjectscanhelp)

ProfessorCaroleGobleTheUniversityofManchester,UKSoftwareSustainabilityInstitute,UKELIXIR-UK,[email protected]

SciReproWorkshop,TPDL,9Sept2016,Hannover,Germany

Acknowledgements•  DagstuhlSeminar16041,January2016

–  http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16041

•  ATISymposiumReproducibility,SustainabilityandPreservation,April2016–  https://turing.ac.uk/events/reproducibility-sustainability-and-preservation/–  https://osf.io/bcef5/files/

•  CTitusBrown•  JulianaFreire•  DavidDeRoure•  StianSoiland-Reyes•  BarendMons•  TimClark•  DanielGarijo•  NormanMorrison

“WhenIuseaword,"HumptyDumptysaidinratherascornfultone,"itmeansjustwhatIchooseittomean-neithermorenorless.”

Carroll,ThroughtheLookingGlass

re-compute

replicatererun

repeat

re-examine

repurpose

recreate

reuse

restore reconstruct review

regeneraterevise

recycle

redo

robustness tolerance

verifica(oncompliancevalida(onassurance

remix

ReproducibilityofReproducibilityResearch

ComputationalScience

http://tpeterka.github.io/maui-project/

From:TheFutureofScientificWorkflows,ReportofDOEWorkshop2015,http://science.energy.gov/~/media/ascr/pdf/programdocuments/docs/workflows_final_report.pd

1.  Observational,experimental

2.  Theoretical3.  Simulation4.  Dataintensive

BioSTIF

ComputationalScience

Scientificpublicationsgoals:(i)announcearesult(ii)convincereadersitscorrect.

Papersinexperimentalscienceshoulddescribetheresultsandprovideaclearenoughprotocoltoallowsuccessfulrepetitionandextension.

Papersincomputationalscienceshoulddescribetheresultsandprovidethecompletesoftwaredevelopmentenvironment,dataandsetofinstructionswhichgeneratedthefigures.

VirtualWitnessing*

*LeviathanandtheAir-Pump:Hobbes,Boyle,andtheExperimentalLife(1985)ShapinandSchaffer.

JillMesirov

DavidDonoho

Datasets,DatacollectionsStandardoperatingproceduresSoftware,algorithmsConfigurations,Toolsandapps,services

Codes,codelibrariesWorkflows,scriptsSystemsoftwareInfrastructureCompilers,hardware

10“Simple”RulesforReproducibleComputationalResearch:RACE

1.  ForEveryResult,KeepTrackofHowItWasProduced

2.  AvoidManualDataManipulationSteps3.  ArchivetheExactVersionsofAllExternal

ProgramsUsed4.  VersionControlAllCustomScripts5.  RecordAllIntermediateResults,WhenPossiblein

StandardizedFormats6.  ForAnalysesThatIncludeRandomness,Note

UnderlyingRandomSeeds7.  AlwaysStoreRawDatabehindPlots8.  GenerateHierarchicalAnalysisOutput,Allowing

LayersofIncreasingDetailtoBeInspected9.  ConnectTextualStatementstoUnderlying

Results10.  ProvidePublicAccesstoScripts,Runs,and

Results

SandveGK,NekrutenkoA,TaylorJ,HovigE(2013)TenSimpleRulesforReproducibleComputationalResearch.PLoSComputBiol9(10):e1003285.doi:10.1371/journal.pcbi.1003285

RecordEverythingAutomateEverythingContainEverythingExposeEverything

Preparationpainindependenttestingtrialsandtribulations

[NormanMorrison]

replicationhostilitynofunding,time,recognition,placetopublishresourceintensiveaccesstothecompleteenvironment

Reproducibility=HardWork

Data sets

Analyses

Open-Paper

Open-Review

DOI:10.1186/2047-217X-1-18 >11000 accesses

Open-Code

8 reviewers tested data in ftp server & named reports published

DOI:10.5524/100044

Open-PipelinesOpen-Workflows

DOI:10.5524/100038 Open-Data

78GB CC0 data

Code in sourceforge under GPLv3: http://soapdenovo2.sourceforge.net/

>5000 downloads

Enabled code to being picked apart by bloggers in wiki http://homolog.us/wiki/index.php?title=SOAPdenovo2

[Scott Edmunds]

WhatisreproducibilityItshardsoletsbesurewearedoingtherightthing,the

usefulthingandthetractablething

Weakerandstrongerformsofreproducibilitydependentonexpectations,methods,tasks,churnturnaround,howmuch

weactuallyknowandcanrestrict

LabAnalogy:Witnessing“Datascopes”

InputData

Software

OutputData

ConfigParameters

Methodstechniques,algorithms,spec.ofthesteps,models

Materialsdatasets,parameters,algorithmseeds

Instrumentscodes,services,scripts,underlyinglibraries,workflows,,refresources

Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatformscomputationalenvironment

“Micro”Reproducibility

“Macro”Reproducibility

TRUST

Validate

Verify

Repeat,Replicate,Robust

[CTitusBrown]

https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html

Whythedifferences?

Reproduce,Trust

“anexperimentisreproducibleuntilanotherlaboratorytriestorepeatit”

AlexanderKohnRepeatability:

“Sameness”Sameresult1Lab1experiment

Reproducibility:

“Similarity”Similarresult>1Lab>1experiment

Validate

Verify

MethodReproducibilitytheprovisionofenoughdetailaboutstudyproceduresanddatasothesameprocedurescould,intheoryorinactuality,beexactlyrepeated.

ResultReproducibility(akareplicability)obtainingthesameresultsfromtheconductofanindependentstudywhoseproceduresareascloselymatchedtotheoriginalexperimentaspossible

Whatdoesresearchreproducibilitymean?StevenN.Goodman,DanieleFanelli,JohnP.A.IoannidisScienceTranslationalMedicine8(341),341ps12.[doi:10.1126/scitranslmed.aaf5027]http://stm.sciencemag.org/content/scitransmed/8/341/341ps12.full.pdf

ProductivityTrackdifferences

Validate

Verify

reviewerswantadditionalworkstatisticianwantsmorerunsanalysisneedstoberepeatedpost-docleaves,studentarrivesnew/reviseddatasetsupdated/newversionsofalgorithms/codessamplewascontaminatedbetterkit-longersimulationsnewpartners,newprojects

PersonalProductivity

PublicGoodReproducibility

“Datascope”LabAnalogy



Instrumentscodes,services,scripts,underlyinglibraries,workflows,refdatasets


“Datascope”LabAnalogy





Form

Function

“Datascope”Practicalities





Change DependenciesScience,methods,datasetsquestionsstay,answerschange

breakage,labsdecay,servicesandtechniquescomeandgo,newinstruments,updateddatasets,services,codes,hardware

Oneoffs,streams,stochastics,sensitivities,scale,non-portabledatablackboxes

supercomputeraccessnon-portablesoftwarelicensingrestrictionsunreliableresourcesblackboxes

T1 T2

evolvingrefdatasets,newsimulationcodes

EnvironmentArchivedvsActive

ContainedvsDistributedMulti-owned

Dependencies->ManageBlackboxes->Expose

Change->FixReliability

ReplicateharderthanReproduce?Repeatingtheexperimentorthesetup?

ContainerConundrum ResultswillVary

ReplicabilityWindowAllexperimentsbecomelessreplicableovertime

Preparetorepair

LevelsofComputationalReproducibility

Coverage:howmuchofanexperimentisreproducible

Orig

inalExp

erim

ent

Simila

rExp

erim

ent

Differen

tExp

erim

ent

Portab

ility

Depth:howmuchofanexperimentisavailable

Binaries+Data

SourceCode/Workflow+Data

Binaries+Data+Dependencies

SourceCode/Workflow+Data+Dependencies

VirtualMachineBinaries+Data+Dependencies

VirtualMachineSourceCode/Workflow+Data+Dependencies

Figures+Data

[Freire,2014]

Minimum:dataandsourcecodeavailableundertermsthatpermitinspectionandexecution.

MeasuringInformationGainfromReproducibility

Researchgoal

Method/Alg.

Platform/ExecEnv

DataParameters

Inputdata

Actors

InformationGain

Implementation/Code

Nochange

Change

Don’tcare

https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/http://www.dagstuhl.de/16041

How? PreservebyReporting,ReproducebyReading

ArchivedRecord

DescriptionZoostandards,commonmetadata

How? PreservebyMaintaining,Repairing,ContainingReproducebyRunning,Emulating,Reconstructing

ActiveInstrument BytelevelBuildabilityZoo

provenance

portability,preservation

robustness,versioning

access descriptionstandards

commonAPIslicensing,identifiers

standards,commonmetadata

changevariationsensitivitydiscrepancyhandling

packaging,containers

FAIRRACEReproducibilityDimensions

dependencies

steps

ResearchObjectStandards-basedmetadataframeworkforlogicallyand

physicallybundlingresourceswithcontext,http://researchobject.org

Biggerontheinsidethantheoutside

externalreferencing

ManifestConstruction

Aggregateslinkthingstogether

Annotationsaboutthings&their

relationships

Container

ResearchObjectStandards-basedmetadataframeworkforlogicallyandphysicallybundlingresourceswithcontext,http://researchobject.org

Packagingcontent&links:Zipfiles,BagIt,Dockerimages

Catalogues&CommonsPlatforms:FAIRDOM

ManifestDescription

Dependencieswhatelseis

needed

Versioningitsevolution

Checklistswhatshouldbethere

Provenancewhereit

camefrom

Identificationlocatethings

regardlesswhere

id

SystemsBiologyCommonsfair-dom.orgfairdomhub.org

•  Linkdata,modelsandSOPs

•  Standards•  Snapshot+DOIs•  Spanresources

http://data.datacite.org/10.15490/seek.1.investigation.56

CitationG.Penkler;F.duToit;W.Adams;M.Rautenbach;D.C.Palm;D.D.vanNiekerk;J.L.Snoep;(2014):GlucosemetabolisminPlasmodiumfalciparumtrophozoites;FAIRDOMHub.http://dx.doi.org/10.15490/seek.1.investigation.56

SystemsBiologyCommons•  Linkdata,models

andSOPs•  Standards•  Spanresources•  Snapshot+DOIs•  Bundleandexport•  Logicalbundles

Belhajjameetal(2015)Usingasuiteofontologiesforpreservingworkflow-centricresearchobjects,JWebSemanticsdoi:10.1016/j.websem.2015.01.003

application/vnd.wf4ever.robundle+zip

WorkflowResearchObjectsexchange,portabilityandmaintenance

*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html

AsthmaResearche-Lab

Datasetbuildingandreleasing

StandardisedpackingofSystemsBiologymodels

EuropeanSpaceAgencyROLibrary

Largedatasetmanagementforlifescienceworkflows

LHCATLASexperiments

NotreDame URostock

EncyclopediaofDNAElements

PeptideAtlas

ROTypes:ManifestContentProfilesminimal,maximal,extensible

PID

Citation

Checklist

Version

Prov

enan

ce

Dependencies

JATS

Commonproperties

amongcontenttypes

Minimuminformationforonecontenttype

Wordsmatter.

Reproducibilityisnotaend.

Itsameanstoanend.Bewarereproducibilityzealots.

50ShadesofReproducibility.

formvsfunction

Aconundrum:bigco-operativedata-drivensciencemakesreproducibility

desirablebutalsomeansdependencyandchangearetobe

expected.

Labanalogyforcomputationalscience

Belhajjameetal(2015)Usingasuiteofontologiesforpreservingworkflow-centricresearchobjects,JWebSemanticsdoi:10.1016/j.websem.2015.01.003

application/vnd.wf4ever.robundle+zip

WorkflowResearchObjects


[C.TitusBrown*]

RepeatableEnvironments


[C.TitusBrown*]

MetadataObjects:ReproducibleReporting,Exchange

ChecklistProvenanceTracking

VersioningDependencies

container

Summary

https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/http://www.dagstuhl.de/16041

•  VALIDATION

•  DISCREPANCIES

•  DEPENDENCIES&CHANGE

•  MAINTENANCE!Tokeepitrunning

•  INTEROPERABILITY

https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/

RewardPeerreviewCitationmechanismsTransitivecreditmapsCitingchangingresources

Cite….Original?MostRecent?AVersion?Dockerimage?Description?

BornReproducible

It’saspectrumWeakerforms

ResearchObjects,MetadataObjectsCitableReproduciblePackaging

InputData WorkflowDescription

Provenancetrace

VersionofCodes/Services

Outputs Configs

Executionenvironment

IntermediatesORCIDS URIs DOIs

T1 T2

evolvingreferencedatasets,newsimulationcodes

Dependencies

ZhaoetalWhyworkflowsbreak-UnderstandingandcombatingdecayinTavernaworkflows,8thIntlConfe-Science2012

Blackboxes

?JoppaetalSCIENCE3402013,MorinetalSCIENCE3362012

Rawdata

Secondarydata

Rawdata

Tertiarydata

Source

SecondarydatasetsSecondaryanalysis

MetaAnalysis

ModelDrivenScience–canIrerunmymodel?ModelSweeps,Whatarethesensitivities?

ReproducibilityRampancy

FAIRnessFindable

Accessible

Interoperable

ReusableIntelligible

Reproducible

Citable

Track&Countable

ReproducibilityRealities:ItsHard

Social

Technical

Methodological

recomputation.org

sciencecodemanifesto.org

“Datascope”Entropy->Preservation“Replication/ReproducibilityWindow”

Form

Function




Laboratoryswandhwinfrastructure,systemssoftware,integrativeplatforms,computationalenvironment

Analogy:TheLabdatascience|data-drivenscience

1.Why

2.How

What is Reproducibility? The R* brouhaha

Documents

Transcript of What is Reproducibility? The R* brouhaha