PLOTCON NYC: Domain Specific Visualization

NOVEMBER 15-18, 2016

#plotcon

1

Domain Specific Visualization

Nathan GoldbaumMoore Data Fellow_________ @njgoldbaum @yt_astrohttps://dxl.ncsa.illinois.edu/

Generic Plotting Libraries are Powerful

John Hunter Plotting Contest Entries: http://scipy2016.scipy.org/ehome/115969/276538/

Creative Power Implies API Complexity

4

I’m picking on matplotlib here, but the same is true of any generic plotting library

5

● Declarative Visualization Libraries

○ Vega-lite, ggplot, altair

● Documentation by example

○ Users create plots by cargo-culting examples

● Provide shortcuts in high-level API

○ “Easy mode” can’t cover all use cases

Managing API Complexity

6

Alternate approach: make assumptions for users

APLpyAstronomical Data

https://aplpy.github.io

CartopyGeographic Data

http://scitools.org.uk/cartopy/

yt: 3D Data Analysis

7

Ingest Data Analysis/Visualization

yt-project.org

Ramses

Gadget

Ingestion

Ingestion Representation

Ingestion Representation Analysis

Ingestion Representation Analysis Visualization

12


Science

13


Community

Science


Community

Science

The ice cream sandwich of open research software

15

yt: A Community of Practice

>100 Contributors as of yt 3.3.2 (Oct 2016)

Tom Abel Bili Dong Suoqing Ji Sam Leitner John Regan Geoffrey SoGabriel Altay Nicholas Earl Allyson Julian Yuan Li Mark Richardson Casey W. StarkKenza Arraki Hilary Egan Christian Karch Alex Lindsay Sherwood Richers Antoine StrugarekKirk Barrow Daniel Fenn Max Katz Chris Malone Thomas Robitaille Elizabeth TaskerRicarda Beckmann John Forbes BW Keller Jonah Miller Anna Rosen Ben ThompsonElliott Biondo Enrico Garaldi Ji-hoon Kim Joshua Moloney Chuck Rozhon Robert ThompsonAlex Bogert Sam Geen Chang-Goo Kim Christopher Moody Douglas Rudd Joseph TomlinsonRobert Bradshaw Austin Gilbert Steffen Klemer Stuart Mumford Hsi-Yu Schive Stephanie TonnesenAndre Bubel Adam Ginsburg Fabian Koller Andrew Myers Anthony Scopatz Matt TurkCorentin Cadiou Nathan Goldbaum Kacper Kowalik Jill Naiman Noel Scudder Miguel de Val-BorroYi-Hao Chen William Gray Mark Krumholz Desika Narayanan Patrick Shriwise Rick WagnerPengfei Chen Markus Haider Michael Kuhlen Kaylea Nelson Devin Silvia Mike WarrenDavid Collins Eric Hallman Meagan Lang Brian O'Shea Sam Skillman Charlie WatsonBrian Crosby David Hannasch Erwin Lau J.S. Oishi Stephen Skory Andrew WetzelWeiguang Cui Cameron Hummels Eve Lee JC Passy Aaron Smith John WiseAndrew Cunningham Anni Järvenpää Doris Lee Rafael Ruggiero Britton Smith Michael Zingale

“Scaling a code in the human dimension”: https://arxiv.org/abs/1301.7064

16

yt: A Community of Practice

Tom Abel Bili Dong Suoqing Ji Sam Leitner John Regan Geoffrey SoGabriel Altay Nicholas Earl Allyson Julian Yuan Li Mark Richardson Casey W. StarkKenza Arraki Hilary Egan Christian Karch Alex Lindsay Sherwood Richers Antoine StrugarekKirk Barrow Daniel Fenn Max Katz Chris Malone Thomas Robitaille Elizabeth TaskerRicarda Beckmann John Forbes BW Keller Jonah Miller Anna Rosen Ben ThompsonElliott Biondo Enrico Garaldi Ji-hoon Kim Joshua Moloney Chuck Rozhon Robert ThompsonAlex Bogert Sam Geen Chang-Goo Kim Christopher Moody Douglas Rudd Joseph TomlinsonRobert Bradshaw Austin Gilbert Steffen Klemer Stuart Mumford Hsi-Yu Schive Stephanie TonnesenAndre Bubel Adam Ginsburg Fabian Koller Andrew Myers Anthony Scopatz Matt TurkCorentin Cadiou Nathan Goldbaum Kacper Kowalik Jill Naiman Noel Scudder Miguel de Val-BorroYi-Hao Chen William Gray Mark Krumholz Desika Narayanan Patrick Shriwise Rick WagnerPengfei Chen Markus Haider Michael Kuhlen Kaylea Nelson Devin Silvia Mike WarrenDavid Collins Eric Hallman Meagan Lang Brian O'Shea Sam Skillman Charlie WatsonBrian Crosby David Hannasch Erwin Lau J.S. Oishi Stephen Skory Andrew WetzelWeiguang Cui Cameron Hummels Eve Lee JC Passy Aaron Smith John WiseAndrew Cunningham Anni Järvenpää Doris Lee Rafael Ruggiero Britton Smith Michael Zingale

>100 Contributors as of yt 3.3.2 (Oct 2016)“Scaling a code in the human dimension”: https://arxiv.org/abs/1301.7064

What is yt?

17

18Data on disk has no physical meaning

19

With yt you can think about data using a physically motivated interface

20

Allowing you to forget about what the data look like as a file format

21And select only the data you want to select

22And only the data you want to select

23

It understands multiresolution data as a first-class citizen

PlotWindow plotting system

27

import yt

import yt.units as u

ds = yt.load('DD0600/DD0600')

prj = yt.ProjectionPlot(

ds, 'z', 'density', width=15*u.kpc)

prj.save()

http://yt-project.org/doc/visualizing/plots.html

29http://paste.yt-project.org/show/6913/

http://www.youtube.com/watch?v=bnm554et0T8

import yt





prj.save()

Short but powerful scripts

● Register namespace● Set up unit metadata for further

use later

30

import yt





prj.save()


● Automagically determine data format

● Parse parameters● Initialize IO and data selection

interface

31

import yt





prj.save()


● Set up in-memory mesh ● Determine available fields to plot ● Chunked IO● Mask out overlapping data● Convert to data CGS units● Perform projection● Pixelize from multi-resolution

projection● Initialize plot object

32

import yt





prj.save()


● Calculate positioning and metadata for matplotlib plot

● Create and customize matplotlib Figure and Axes objects

● Call imshow() with appropriate keyword arguments

● Save plot to disk

33

Generic plotting libraries can’t make assumptions

34

import matplotlib.pyplot as plt

plt.imshow(image_data)

plt.savefig('plot.png')

Customizing plots increases complexityimport matplotlib

import matplotlib.pyplot as plt

plt.imshow(image_data, norm=matplotlib.colors.LogNorm(),

origin='lower', cmap='viridis',

extent=[-7.5, 7.5, -7.5, 7.5])

cb = plt.colorbar()

cb.set_label(r'Projected Density $(\rm{g}/\rm{cm}^2)$')

plt.xlabel('x (kpc)')

plt.ylabel('y (kpc)')

plt.savefig('plot.png', bbox_inches='tight')

35

37


Community

Science

Build new libraries for your use cases

Add new plotting tasks to libraries that already exist

Design Tips for Plotting Libraries

38

Provide sane defaults

39

import yt


prj = yt.ProjectionPlot(ds, 'z', 'density')

prj.save('./frames/0000.png')

for i in range(150):

prj.zoom(1.03)

prj.save('./frames/%04d.png' % (i+1))

import yt


prj = yt.ProjectionPlot(ds, 'z', 'density')

prj.set_zlim('density', 8e-6, 3e-1)

prj.save('./frames/0000.png')

for i in range(150):

prj.zoom(1.03)

prj.save('./frames/%04d.png' % (i+1))

Customize via properties/setters, not keyword arguments

40

Don’t overcomplicate for niche use cases

import yt



fields = ['density', 'temperature',

'Velocity_divergence', 'grid_level']

slc = yt.SlicePlot(ds, 'z', fields)

slc.set_width(15*kpc)

slc.set_log('grid_level', False)

slc.set_unit('velocity_divergence', 'km/s/kpc')

slc.set_unit('density', 'Msun/pc**3')

slc.save()

41

Expose base plotting library primitives slc = yt.SlicePlot(ds, 'z', 'density')

plot = slc.plots['density']

# matplotlib.figure.Figure

plot.figure

# matplotlib.axes.Axes

plot.axes

# matlotlib.colorbar.Colorbar

plot.cb

# matplotlib.image.AxesImage

plot.image

42http://yt-project.org/docs/dev/cookbook/simple_plots.html#matplotlib-primitives

Only perform expensive actions as needed

43

def invalidate_plot(f): @functools.wraps(f) def newfunc(*args, **kwargs): rv = f(*args, **kwargs) args[0]._plot_valid = False return rv return newfunc

def validate_plot(f): @wraps(f) def newfunc(*args, **kwargs): if not args[0]._plot_valid: args[0]._setup_plots() rv = f(*args, **kwargs) return rv return newfunc

class SlicePlot:

def _setup_plots(self): # make plots using matplotlib primitives

@invalidate_plot def set_axes_unit(self, unit_name): self._axes_unit_names = (unit_name, unit_name)

@validate_plot def save(self): for field in self.plots: self.plots[field].save()

44

http://www.youtube.com/watch?v=B2tZIax5U2A

Thank you for your attention!

45

Questions? Concerns? Comments?

PLOTCON NYC: Domain Specific Visualization

Data & Analytics

Transcript of PLOTCON NYC: Domain Specific Visualization