Open science incognitive modeling

Caspar Addymanc.addyman@bbk.ac.uk @BrainStraining

International Conference on Cognitive ModelingUniversity of Groningen

10 April 2015

To the extent possible under law, Caspar Addyman has waived all copyright and related or neighboring rights to Open science in cognitive modeling. This work is published from: United Kingdom.

Overview

• How I came to write a “Modeling Manifesto”

• What I’ve done since

• What we should all do next

Overview

• How I came to write a “Modeling Manifesto”

• What I’ve done since

• What we should all do next

• What I am going to do next

TRACX

French, Addyman & Mareschal, Psych. Rev. 2011

Truncated Recursive AutoassocativeChunk Extractor (TRACX)

(French, Addyman & Mareschal, Psych Rev, 2011)

http://leadserv.u-bourgogne.fr/~tracx/

http://github.com/YourBrain/TRACX-Web

Computational Modeling in Cognitive Science: A Manifesto for Change

Addyman & French, TOPICS, 2012

1.Let EVERYONE run your simulations (easily)

2.Let motivated users adjust parameters/ run their own simulations

3.Let other modelers use your code4. Write good documentation5. Clear mapping between model and empirical

data6. Show internal structure7. Allow import/export to Excel, JSON, etc

Yes but..

“Implementations are not specifications” Cooper and Guest, Cognitive Systems Research, 2013

1. Troubles with replicating modelling resultsGuest and Cooper (2012) ICCM 11

2. Bit rot“a consequence of continuing changes in technology code needs to be kept up to date.”

3. Formal model specification “..more attention must be devoted to model specification as an independent exercise to implementation. “

Playing ATARI with Deep Learning Google DeepMindNature 26 Feb 2015Released their code

“It is true though that we don’t publish all our code right away. In fact, some of our code gets tied up in industrial projects which make it hard to release.” – Jurgen Schmidhuber, Reddit AMA, March 2015

“Psychology PhD students should know how to code.” – Jay McClelland

http://convnetjs.comAndrej Karpathy

• The associated source code of the software described by the paper. This should, as far as possible, follow accepted community standards and be licensed under a suitable license such as BSD, LGPL, or MIT. Dependency on commercial software such as Mathematica and MATLAB does not preclude a paper from consideration, although complete open source solutions are preferred.

• Documentation for running and installing the software. For end-user applications, instructions for installing and using the software are prerequisite; for software libraries, instructions for using the application program interface are prerequisite.

• A test dataset with associated control parameter settings. Where feasible, results from standard test sets should be included. Where possible, test data should not have any dependencies — for example, a database dump.

Supporting information for submissions in which software is the central part of the paper:

Bianca Kramer & Jeroen Bosman(2015) dx.doi.org/10.6084/m9.figshare.1286826

101 Innovations in Scholarly Communication

GAMIT-Net, CogSci 2014

http://github.com/YourBrain/GAMIT Addyman & Mareschal, 2014

Caspar Addyman. (2014). GAMIT Model MATLAB Code. Zenodo. dx.doi.org/10.5281/zenodo.9895

GitHub

Caspar Addyman. (2015). Contec-Pulse-Oximeter-in-Matlab: Version 1. Zenodo. 10.5281/zenodo.16277

Caspar Addyman. (2015). Infant-Habituation-in-Matlab: Version 1. Zenodo. 10.5281/zenodo.16627

iPython / Jupyter

Beakerbeakernotebook.com

Computational Modeling in Cognitive Science: A Manifesto for Change

1.Let EVERYONE run your simulations (easily)

2.Let motivated users adjust parameters/ run their own simulations

3.Let other modelers use your code4. Write good documentation5. Clear mapping between model and empirical

data6. Show internal structure7. Allow import/export to Excel, JSON, etc

✔

?

✔

✔✔

✔✔✔✔

✔✔

✔✔

?

Computational Modeling in Cognitive Science: A Manifesto for Change

1.Let EVERYONE run your simulations (easily)

2.Let motivated users adjust parameters/ run their own simulations

3.Let other modelers use your code4. Write good documentation

5.Clear mapping between model and empirical data

6. Show internal structure7. Allow import/export to Excel, JSON, etc

✔

?

✔

✔✔

✔✔✔✔

✔✔

✔✔

?

• Try notebooks• Try Python• Try sharing

Questions for you

1. What is your workflow? Is it connected?Is it shareable?If not, why not?

2. How can we narrow the gap between models and experiments?