Download - .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Transcript
Page 1: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

RRR 4 MORReplicability, Reproducibility and Reusability for Model Order Reduction

Jorg Fehr, Jan Heiland, Christian Himpe, Jens Saak

2017-01-12

3rd Workshop on Model Reduction of Complex Dynamical SystemsMODRED 2017

Page 2: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Best Practices for RRR in MOR

An open-access review article, doi.org/bsb2

Jens Saak, [email protected] RRR 4 MOR 2/17

Page 3: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Introduction: MotivationObservation (of an inadequacy):

Numerical experiments gain importance as part of science:

1971 Nitsche: Uber ein Variationsprinzip zur Losung vonDirichlet-Problemen bei Verwendung von Teilraumen, die keinenRandbedingungen unterworfen sind— no numerical experiments reported

1986 Saad & Schultz: GMRES: A Generalized Minimal ResidualAlgorithm for Solving Nonsymmetric Linear Systems— 2/14 pages are devoted to numerical experiments

2010 Chataranbutat & Sorensen: Nonlinear Model Reduction viaDiscrete Empirical Interpolation— consists to more than 30% of numerical examples or reasoningsbased on numerical experiments

Still, there is no standard defining how to present, document andpublish numerical experiments.

Jens Saak, [email protected] RRR 4 MOR 3/17

Page 4: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Introduction: Motivation

Observation (of a chance):

Science builds on previous findings

in theory – using e.g. theoremsand in practice – using established methods

Often, the first step of a new scientific endeavor is the reproduction ofprevious results

Building upon numerical investigations can be incredibly easy

software is easy to share and to modifyhardware is easy to replace

Jens Saak, [email protected] RRR 4 MOR 4/17

Page 5: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Introduction: Motivation

Observation (of a chance):

Science builds on previous findings

in theory – using e.g. theoremsand in practice – using established methods

Often, the first step of a new scientific endeavor is the reproduction ofprevious results

Building upon numerical investigations could be incredibly easy

software is easy to share and to modifyhardware is easy to replace

Jens Saak, [email protected] RRR 4 MOR 4/17

Page 6: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Introduction: Our Aim

Improve Computer-Based Experiments (CBEx):

Create problem-awareness and ensure scientificity

Define terminology

Establish best-practices

Formulate discipline-agnostic practical guidelines

Jens Saak, [email protected] RRR 4 MOR 5/17

Page 7: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Introduction: Our Aim

Improve Computer-Based Experiments (CBEx):

Create problem-awareness and ensure scientificity

Define terminology

Establish best-practices

Formulate discipline-agnostic practical guidelines

Jens Saak, [email protected] RRR 4 MOR 5/17

Page 8: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Outline

1. Best Practices for RRR in MOR

2. Introduction

3. RRRReplicabilityReproducibilityReusabilityThe Road to Reusability

4. Proposed Best Practices

Jens Saak, [email protected] RRR 4 MOR 6/17

Page 9: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

RRR

We argue that Good Scientific Practice is all about

Replicability,

Reproducibility, and

Reusability

We discuss

what RRR looks like for CBEx,

how RRR forms a hierarchy, and

how RRR can be implemented up to a desired or feasible level.

Jens Saak, [email protected] RRR 4 MOR 7/17

Page 10: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

RRR

We argue that Good Scientific Practice is all about

Replicability,

Reproducibility, and

Reusability

We discuss

what RRR looks like for CBEx,

how RRR forms a hierarchy, and

how RRR can be implemented up to a desired or feasible level.

Jens Saak, [email protected] RRR 4 MOR 7/17

Page 11: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

RRR

We argue that Good Scientific Practice is all about

Replicability,

Reproducibility, and

Reusability

We discuss

what RRR looks like for CBEx,

how RRR forms a hierarchy, and

how RRR can be implemented up to a desired or feasible level.

Jens Saak, [email protected] RRR 4 MOR 7/17

Page 12: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

RRR : Replicability

Definition

The attribute Replicability describes the ability to repeat a CBEx and tocome to the same (in a numerical sense) results. Sometimes the equivalentterm Repeatability is used for this experimental property.

Replicability is a basic requirement of reliable software as well as of itsresult as it shows a certain robustness of the procedure against

statistical influences

and bias of the observer.

Also, only replicable CBEx can serve as a benchmark to which newmethods can be compared, cf. [Vitek & Kalibera ’11].

Jens Saak, [email protected] RRR 4 MOR 8/17

Page 13: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

RRR : Reproducibility

Definition

Reproducibility of a CBEx means that it can be repeated by a differentresearcher in a different computer environment.

This is an adaption of the general concept of Reproducibility

that is key in any science that relies on experiments,

that is a subject in the theory of science, and

which absence in a significant fraction of publications in manyresearch areas has shaped the term Reproducibility crisis in recentyears [Marcus ’13]; cf. also [Collberg, Proebsting, & Warren ’04] onReproducibility in computer science.

Jens Saak, [email protected] RRR 4 MOR 9/17

Page 14: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

RRR : Reusability

Definition

In the sphere of CBEx, Reusability refers to the possibility to reuse thesoftware or parts thereof for different purposes, in different environments,and by researchers other than the original authors.

In particular, Reusability enables the utilization of the test setup orparts of it for other experiments or related applications.

Although theoretically, any bit of a software can be reused for differentpurposes, here, Reusability applies only for reproducible parts.

Jens Saak, [email protected] RRR 4 MOR 10/17

Page 15: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

The Road to Reusability

Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

• ReplicabilityRequired: Basic Documentation

Recommended: Automation & Testing

• ReproducibilityRequired: Extensive Documentation

Recommended: Availability

• ReusabilityRequired: Accessibility

Recommended: Software Management,Modularity & Licensing

Jens Saak, [email protected] RRR 4 MOR 11/17

Page 16: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

The Road to Reusability

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Replicability

← This is a sanity check

Reproducibility

← This makes it science

Reusability

← This is a competitive advantage

Replicability Reproducibility Reusability

Jens Saak, [email protected] RRR 4 MOR 12/17

Page 17: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

The Road to Reusability

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Replicability ← This is a sanity check

Reproducibility

← This makes it science

Reusability

← This is a competitive advantage

Replicability Reproducibility Reusability

Jens Saak, [email protected] RRR 4 MOR 12/17

Page 18: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

The Road to Reusability

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Replicability ← This is a sanity check

Reproducibility

← This makes it science

Reusability

← This is a competitive advantage

Replicability Reproducibility Reusability

Jens Saak, [email protected] RRR 4 MOR 12/17

Page 19: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

The Road to Reusability

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Replicability ← This is a sanity check

Reproducibility ← This makes it science

Reusability

← This is a competitive advantage

Replicability Reproducibility Reusability

Jens Saak, [email protected] RRR 4 MOR 12/17

Page 20: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

The Road to Reusability

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Replicability ← This is a sanity check

Reproducibility ← This makes it science

Reusability

← This is a competitive advantage

Replicability Reproducibility Reusability

Jens Saak, [email protected] RRR 4 MOR 12/17

Page 21: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

The Road to Reusability

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Replicability ← This is a sanity check

Reproducibility ← This makes it science

Reusability ← This is a competitive advantage

Replicability Reproducibility Reusability

Jens Saak, [email protected] RRR 4 MOR 12/17

Page 22: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Outline

1. Best Practices for RRR in MOR

2. Introduction

3. RRR

4. Proposed Best PracticesCode Availability SectionBasic DocumentationCode Meta Data

Jens Saak, [email protected] RRR 4 MOR 13/17

Page 23: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Proposed Best Practices

Code Availability Section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Introduced by Nature (doi:10.1038/514536a, doi:10.1038/sdata.2015.4).

An additional section stating the availability of source code.

Code should be shared [LeVeque’13]

Obvious at a glance.

Code Availability SectionThe source code of the implementations used to compute thepresented results can be obtained from:

doi:???????/???????? and is authored by: X Y, A B.

Please contact X Y for licensing information.

Jens Saak, [email protected] RRR 4 MOR 14/17

Page 24: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Proposed Best Practices

Basic Documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

README - Every code should have a README file

Title, Version, Release-Date, Summary, Table-of-Contents, ...

RUNME - Every scientific code should have a script reproducing its results

LICENSE - Licensing contents

AUTHORS - List of authors and contributors

CITATION - How to properly cite the code

DEPENDENCIES - required hardware & software

CODE - Code meta-data (see next slide)

More: CHANGELOG, FAQ, INSTALL, TODO

Source file headers:

Project, Authors, Summary, ...

All in plain text!possibly using, e.g. markdown

Jens Saak, [email protected] RRR 4 MOR 15/17

Page 25: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Proposed Best Practices

Basic Documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

README - Every code should have a README file

Title, Version, Release-Date, Summary, Table-of-Contents, ...

RUNME - Every scientific code should have a script reproducing its results

LICENSE - Licensing contents

AUTHORS - List of authors and contributors

CITATION - How to properly cite the code

DEPENDENCIES - required hardware & software

CODE - Code meta-data (see next slide)

More: CHANGELOG, FAQ, INSTALL, TODO

Source file headers:

Project, Authors, Summary, ...

All in plain text!possibly using, e.g. markdown

Jens Saak, [email protected] RRR 4 MOR 15/17

Page 26: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Proposed Best Practices

Code Meta Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Proposed by: [Katz & Smith’15] We suggest: .ini format

Sample keys:name

shortname

version

release-date

id

id-type

authors

orcids

associated

topic

type

license

license-type

repository

repository-type

languages

dependencies

systems

website

keywords

Jens Saak, [email protected] RRR 4 MOR 16/17

Page 27: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Open For Discussions

Suggestions? / Remarks? Talk to us, e.g. over dinner tonight. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Jorg Fehr

Jan Heiland

Christian Himpe

Jens Saak

Thank very much for your attention!

Questions?

Jens Saak, [email protected] RRR 4 MOR 17/17

Page 28: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Open For Discussions

Suggestions? / Remarks? Talk to us, e.g. over dinner tonight. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Jorg Fehr

Jan Heiland

Christian Himpe

Jens Saak

Thank very much for your attention!

Questions?

Jens Saak, [email protected] RRR 4 MOR 17/17

Page 29: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

Our Questions

Public code is a major recommendation of our guide.

Can you think of situations in which code publicationis not possible or desired?

Do you have additions to the list of suggesteddocumentation files?

Do you have additions to the code meta data tags?

Do you miss any information?

Jens Saak, [email protected] RRR 4 MOR 1/4

Page 30: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

References IS. Chaturantabut and D. C. Sorensen.Nonlinear model reduction via discrete empirical interpolation.SIAM J. Sci. Comput., 32(5):2737–2764, 2010.

C. Collberg, T. Proebsten, and A. M. Warren.Repeatability and benefaction in computer systems research.Technical report, University of Arizona, 2014.Accessed: 2016-09-22.

J. Fehr, J. Heiland, C. Himpe, and J. Saak.Best practices for replicability, reproducibility and reusability of computer-basedexperiments exemplified by model reduction software.AIMS Mathematics, 1(3):261–281, 2016.

D. S. Katz and A. M. Smith.Transitive credit and JSON-LD.Journal of Open Research Software, 3(1), 2015.

R. J. LeVeque.Top ten reasons to not share your code (and why you should anyway).SIAM News, 46(3), 2013.

Jens Saak, [email protected] RRR 4 MOR 2/4

Page 31: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

References IIG. Marcus.The crisis in social psychology that isn’t.www.newyorker.com/tech/elements/the-crisis-in-social-psychology-that-isnt,2013.

Code share.Nature, 514:536, 2014.

Ctrl alt share.Scientific Data, 2, 2015.

J. Nitsche.Uber ein Variationsprinzip zur Losung von Dirichlet-Problemen bei Verwendung vonTeilraumen, die keinen Randbedingungen unterworfen sind.Abhandlungen aus dem Mathematischen Seminar der Universitat Hamburg, 36(1):9–15,1971.

Y. Saad and M. H. Schultz.GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems.SIAM Journal on Scientific and Statistical Computing, 7(3):856–869, 1986.

Jens Saak, [email protected] RRR 4 MOR 3/4

Page 32: .975!RRR4MOR .96!Replicability, Reproducibility and ... · Replicability, Reproducibility, and Reusability We discuss what RRR looks like for CBEx, how RRR forms a hierarchy, and

References IIIJ. Vitek and T. Kalibera.Repeatability, reproducibility, and rigor in systems research.In Proceedings of the 9th ACM International Conference on Embedded Software, pages33–38, 2011.

Jens Saak, [email protected] RRR 4 MOR 4/4