975!RRR4MOR .96!Replicability, Reproducibility and ... Replicability, Reproducibility, and...

download 975!RRR4MOR .96!Replicability, Reproducibility and ... Replicability, Reproducibility, and Reusability

of 32

  • date post

    22-Jul-2020
  • Category

    Documents

  • view

    2
  • download

    0

Embed Size (px)

Transcript of 975!RRR4MOR .96!Replicability, Reproducibility and ... Replicability, Reproducibility, and...

  • RRR 4 MOR Replicability, Reproducibility and Reusability for Model Order Reduction

    Jörg Fehr, Jan Heiland, Christian Himpe, Jens Saak

    2017-01-12

    3rd Workshop on Model Reduction of Complex Dynamical Systems MODRED 2017

  • Best Practices for RRR in MOR

    An open-access review article, doi.org/bsb2

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 2/17

    http://doi.org/bsb2 mailto:saak@mpi-magdeburg.mpg.de

  • Introduction: Motivation Observation (of an inadequacy):

    Numerical experiments gain importance as part of science:

    1971 Nitsche: Über ein Variationsprinzip zur Lösung von Dirichlet-Problemen bei Verwendung von Teilräumen, die keinen Randbedingungen unterworfen sind — no numerical experiments reported

    1986 Saad & Schultz: GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems — 2/14 pages are devoted to numerical experiments

    2010 Chataranbutat & Sorensen: Nonlinear Model Reduction via Discrete Empirical Interpolation — consists to more than 30% of numerical examples or reasonings based on numerical experiments

    Still, there is no standard defining how to present, document and publish numerical experiments.

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 3/17

    mailto:saak@mpi-magdeburg.mpg.de

  • Introduction: Motivation

    Observation (of a chance):

    Science builds on previous findings

    in theory – using e.g. theorems and in practice – using established methods

    Often, the first step of a new scientific endeavor is the reproduction of previous results

    Building upon numerical investigations can be incredibly easy

    software is easy to share and to modify hardware is easy to replace

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 4/17

    mailto:saak@mpi-magdeburg.mpg.de

  • Introduction: Motivation

    Observation (of a chance):

    Science builds on previous findings

    in theory – using e.g. theorems and in practice – using established methods

    Often, the first step of a new scientific endeavor is the reproduction of previous results

    Building upon numerical investigations could be incredibly easy software is easy to share and to modify hardware is easy to replace

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 4/17

    mailto:saak@mpi-magdeburg.mpg.de

  • Introduction: Our Aim

    Improve Computer-Based Experiments (CBEx):

    Create problem-awareness and ensure scientificity

    Define terminology

    Establish best-practices

    Formulate discipline-agnostic practical guidelines

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 5/17

    mailto:saak@mpi-magdeburg.mpg.de

  • Introduction: Our Aim

    Improve Computer-Based Experiments (CBEx):

    Create problem-awareness and ensure scientificity

    Define terminology

    Establish best-practices

    Formulate discipline-agnostic practical guidelines

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 5/17

    mailto:saak@mpi-magdeburg.mpg.de

  • Outline

    1. Best Practices for RRR in MOR

    2. Introduction

    3. RRR Replicability Reproducibility Reusability The Road to Reusability

    4. Proposed Best Practices

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 6/17

    mailto:saak@mpi-magdeburg.mpg.de

  • RRR

    We argue that Good Scientific Practice is all about

    Replicability,

    Reproducibility, and

    Reusability

    We discuss

    what RRR looks like for CBEx,

    how RRR forms a hierarchy, and

    how RRR can be implemented up to a desired or feasible level.

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 7/17

    mailto:saak@mpi-magdeburg.mpg.de

  • RRR

    We argue that Good Scientific Practice is all about

    Replicability,

    Reproducibility, and

    Reusability

    We discuss

    what RRR looks like for CBEx,

    how RRR forms a hierarchy, and

    how RRR can be implemented up to a desired or feasible level.

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 7/17

    mailto:saak@mpi-magdeburg.mpg.de

  • RRR

    We argue that Good Scientific Practice is all about

    Replicability,

    Reproducibility, and

    Reusability

    We discuss

    what RRR looks like for CBEx,

    how RRR forms a hierarchy, and

    how RRR can be implemented up to a desired or feasible level.

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 7/17

    mailto:saak@mpi-magdeburg.mpg.de

  • RRR : Replicability

    Definition

    The attribute Replicability describes the ability to repeat a CBEx and to come to the same (in a numerical sense) results. Sometimes the equivalent term Repeatability is used for this experimental property.

    Replicability is a basic requirement of reliable software as well as of its result as it shows a certain robustness of the procedure against

    statistical influences

    and bias of the observer.

    Also, only replicable CBEx can serve as a benchmark to which new methods can be compared, cf. [Vitek & Kalibera ’11].

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 8/17

    mailto:saak@mpi-magdeburg.mpg.de

  • RRR : Reproducibility

    Definition

    Reproducibility of a CBEx means that it can be repeated by a different researcher in a different computer environment.

    This is an adaption of the general concept of Reproducibility

    that is key in any science that relies on experiments,

    that is a subject in the theory of science, and

    which absence in a significant fraction of publications in many research areas has shaped the term Reproducibility crisis in recent years [Marcus ’13]; cf. also [Collberg, Proebsting, & Warren ’04] on Reproducibility in computer science.

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 9/17

    mailto:saak@mpi-magdeburg.mpg.de

  • RRR : Reusability

    Definition

    In the sphere of CBEx, Reusability refers to the possibility to reuse the software or parts thereof for different purposes, in different environments, and by researchers other than the original authors.

    In particular, Reusability enables the utilization of the test setup or parts of it for other experiments or related applications.

    Although theoretically, any bit of a software can be reused for different purposes, here, Reusability applies only for reproducible parts.

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 10/17

    mailto:saak@mpi-magdeburg.mpg.de

  • The Road to Reusability

    Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

    • Replicability Required: Basic Documentation

    Recommended: Automation & Testing

    • Reproducibility Required: Extensive Documentation

    Recommended: Availability

    • Reusability Required: Accessibility

    Recommended: Software Management, Modularity & Licensing

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 11/17

    mailto:saak@mpi-magdeburg.mpg.de

  • The Road to Reusability

    Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

    Replicability

    ← This is a sanity check Reproducibility

    ← This makes it science

    Reusability

    ← This is a competitive advantage

    Replicability Reproducibility Reusability

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 12/17

    mailto:saak@mpi-magdeburg.mpg.de

  • The Road to Reusability

    Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

    Replicability ← This is a sanity check

    Reproducibility

    ← This makes it science

    Reusability

    ← This is a competitive advantage

    Replicability Reproducibility Reusability

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 12/17

    mailto:saak@mpi-magdeburg.mpg.de

  • The Road to Reusability

    Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

    Replicability ← This is a sanity check Reproducibility

    ← This makes it science Reusability

    ← This is a competitive advantage

    Replicability Reproducibility Reusability

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 12/17

    mailto:saak@mpi-magdeburg.mpg.de

  • The Road to Reusability

    Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

    Replicability ← This is a sanity check Reproducibility ← This makes it science

    Reusability

    ← This is a competitive advantage

    Replicability Reproducibility Reusability

    Jens Saak, saak@mpi-magdeburg.mpg.de RRR 4 MOR 12/17

    mailto:saak@mpi-magdeburg.mpg.de

  • The Road to Reusability

    Summary. . . . .