Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python)...

18
Workflow matan gavish stanford statistics 5/2013

Transcript of Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python)...

Page 1: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflowmatan gavish

stanford statistics5/2013

Page 2: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

What is workflow?

How we accomplish daily chores on computer

Computer routine are different

Tells apart professionals

Tells apart professional groups

Page 3: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Who cares?

Be more productive

Be less frustrated

Produce better quality work

Do impossible things before breakfast

You.

Page 4: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Things we do

Write code

Execute code

Keep track

Write papers & slides

alone. in group. in community.

Computational scientist’s workflow

Page 5: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflow elements

Workstation

Directory structure

Shell

Text editor

Knowledge base

Basics

Page 6: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflow elements

Source control and collaboration

Codebase, packages, dependencies

Style guide and lint

Tests and coverage

Docs generator

Code review

Software development

Page 7: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflow elements

Production environment

Job monitoring

Result harvest

Production

Page 8: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflow elements

Research Journal

Lab Journal

Result archive

Keeping track

Page 9: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflow elements

Typesetting system

Source control and collaboration

Citation manager

Writing papers & slides

Page 10: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

current workflow?

none.

To: advisor

From: Student

Re: Paper draft

attached draft_2_final_4_submitted_student_changes12.tex and draft_references_for_final_4_15.bib

15

Page 11: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

My Workflow

Workstation: unix only, Mac ($$$), Ubuntu ($)

Directory structure: ~/r/project-name/talks/asilomar

Shell: ZShell + ohmyzsh goodness, .zshrc, .ssh/config

Text editor: vim (filetypes{colors,templates,indentation, macros}, pathogen, fugitive, completion, keybinds) .vimrc

Knowledge base: wiki

Basics

Page 12: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflow elements

Source control and collaboration: Git! git! git! (branches, remotes, tags, submodules, hooks, github)

Codebase, packages, dependencies: Git+Phabricator, pip+virtualenv (python)

Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Tests and coverage: nose (python)Docs generator: sphinx-docCode review: Phabricator

Software development

Page 13: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflow elements

Production environment: qsub, starcluster, MRJob

Job monitoring: qsub, MRJob

Result harvest: rsync, scp, dropbox, MySQL, VCR

Production

Page 14: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflow elements

Research Journal: black notebook, tex notes in project repo

Lab Journal: iPython Notebook, VCR

Important result archive: VCR

Keeping track

Page 15: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Workflow elements

Typesetting system: vim-latex, soft links (macros, bibtex, graphics), Skim (back+forward search), beamer

Source control and collaboration: Git, github, Meld(diff)

Citation manager: Mendeley + git + soft links

Writing papers & slides

Page 16: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

Ultimate workflow?

Effortless to collaborate within group

Easy to collaborate outside group

Easy to learn

Easy to maintain

Effortless to set up

Portable across computers

Page 17: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

living workflow

Tutorial + wiki for each element

Regularly maintained

One or few installers

Page 18: Workflow - Stanford Universitygavish/documents/workflow...Git+Phabricator, pip+virtualenv (python) Style guide and lint: Google’s style guide, pylint, .pylintrc, mlint, R “lint”

get a workflow

http://software-carpentry.org

http://verifiable-research.org

https://github.com

http://vimcasts.org

http://www/~gavish/workflow.htmlhttp://jackman.stanford.edu/classes/SSMART/2011/workflow.pdf

start yours! (1 week)Opensource it!