EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen...
-
Upload
daniel-newman -
Category
Documents
-
view
221 -
download
0
Transcript of EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen...
![Page 1: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/1.jpg)
EAVL
EXTREME-SCALE ANALYSIS AND VISUALIZATION LIBRARY
Jeremy Meredith
SDAV Next-Gen Library MeetingSeptember, 2012
![Page 2: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/2.jpg)
History
• Originally ORNL LDRD– Jeremy Meredith, Sean Ahern, Dave Pugmire– plus Rob Sisneros joined as a postdoc
• Many hours sitting in conference rooms arguing over things like “what does it mean to have one of your dimensions be unstructured?”– then determine what to do that’s practical without
falling off the data modeling deep end . . . .
• Exascale focus
![Page 3: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/3.jpg)
Approaching the Exascale Problems
• Update traditional data model to handle modern simulation codes and a wider range of data.
• Investigate how an updated data and execution model can achieve the necessary computational, I/O, and memory efficiency.
• Explore methods for visualization algorithm developers to achieve these efficiency gains and better support exascale architectures.
![Page 4: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/4.jpg)
DATA MODELING CHALLENGES
![Page 5: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/5.jpg)
Connectivity3D Point CoordinatesCell FieldsPoint Fields
Dimensions3D Point CoordinatesCell FieldsPoint Fields
Dimensions3D Axis CoordinatesCell FieldsPoint Fields
A Traditional Data Set Model
Data Set
Rectilinear Structured Unstructured
![Page 6: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/6.jpg)
Challenge: Non-Physical Data Analysis
• Graph Data– topologically 0D vertices, 1D edges– non-spatial; storing X/Y/Z values is wasted space
• Pure Parameter Studies– e.g. reaction rate of combustion• FOUR “spatial” dimensions
– e.g. methane concentration vs oxygen concentration vs temperature vs pressure
• more complex reaction higher dimensionality
oxyg
en
met
hane
temperature
pressure
![Page 7: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/7.jpg)
Challenge: Molecular Data(e.g., LAMMPS, VASP)
• To represent using vtkPolyData or vtkUnstructuredGrid:– VTK_VERTEX cells for the atoms– VTK_LINE cells for the bonds
• Any field data must exist on both element types– Not only inefficient:
• dummy bond strengths on the atoms?• dummy atomic numbers on the bonds?
– But also incorrect:• e.g. average(BondStrength) uses dummy values from atoms?
H
C
H
C
H
H
BondStr11112
AtomicNum661111
![Page 8: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/8.jpg)
Challenge: Side Sets(e.g. Exodus, flux surfaces)
• The flow from A to B is defined on a set of faces• The flux variable is defined only on those faces– do you combine them into a single mesh?• waste space on dummy values, potentially introducing errors
– or create a separate mesh and lose the mapping info?• horribly expensive and error-prone to recalculate mapping
A Bflux surface lives inside the volumetric mesh
![Page 9: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/9.jpg)
Challenge: Dimensionality, Refinement(e.g. GenASiS)
• (a) seven (or eight) dimensional mesh– f(x,y,z,ϴ,ϕ,λ,F)=E, plus time
• (b) refinement occurs on a per-cell basis– can’t assume per-block refinement– sometimes referred to as “unstructured AMR”
![Page 10: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/10.jpg)
www.vacet.org
Challenge: Unique Mesh Topologies(e.g. MADNESS)
• MADNESS does not have a traditional mesh– Just a quad-tree with polynomial coefficients– Up to 30 refinement levels / tree depth
31 2 4
5
root
1
2
345
spatial structure internal tree representation
![Page 11: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/11.jpg)
www.vacet.org
Challenge: Very High Order Fields(e.g. MADNESS)
• Legendre polynomial series at each tree node– Each tree node has Kdim coefficients– K can be up to approx. 20• i.e. 400 coeffs per tree node in 2D, 8000 in 3D
(example with K=3, dim=2)
0.834 0.592 0.0030.592 0.003 0.0100.003 0.010 0.007
![Page 12: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/12.jpg)
THE EAVL DATA MODEL
![Page 13: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/13.jpg)
Connectivity3D Point CoordinatesCell FieldsPoint Fields
Dimensions3D Point CoordinatesCell FieldsPoint Fields
Dimensions3D Axis CoordinatesCell FieldsPoint Fields
A Traditional Data Set Model (again)
Data Set
Rectilinear Structured Unstructured
![Page 14: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/14.jpg)
TreeConnectivity Dimensions
FieldNameComponent
NameAssociationValues
Cells[]Points[]Fields[]
The EAVL Data Set Model
Data Set
CellSet
Explicit Structured
CoordsField
QuadTreeCellList
Subset
![Page 15: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/15.jpg)
Connectivity: (a bunch of cells)
FieldName: “c” “c” “c”Component: 0 1 2
Name: “c”Association: PointsValues[3*npts]
Cells[1]Points[1]Fields[1]
Example: An Unstructured Grid(with interleaved coordinates)
eavlDataSet
eavlExplicitCellSet
eavlCoordinates
eavlField
![Page 16: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/16.jpg)
Connectivity: (a bunch of cells)
FieldName: “x” “y” “z”Component: 0 0 0
Name: “x”Association: PointsValues[npts]
Cells[1]Points[1]Fields[3]
Example: An Unstructured Grid(with separated coordinates)
eavlDataSet
eavlExplicitCellSet
eavlCoordinates
eavlField #0Name: “y”Association: PointsValues[npts]
eavlField #1Name: “z”Association: PointsValues[npts]
eavlField #2
![Page 17: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/17.jpg)
RegularStructure: 30 40 30
FieldName: “x” “y” “z”Component: 0 0 0
Name: “x”Association: PointsValues[npts]
Cells[1]Points[1]Fields[3]
Example: A Curvilinear Grid
eavlDataSet
eavlStructuredCellSet
eavlCoordinates
eavlField #0Name: “y”Association: PointsValues[npts]
eavlField #1Name: “z”Association: PointsValues[npts]
eavlField #2
![Page 18: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/18.jpg)
RegularStructure: 30 40 30
FieldName: “x” “y” “z”Component: 0 0 0
Name: “x”Association: LogicalDim0Values[ni]
Cells[1]Points[1]Fields[3]
Example: A Rectilinear Grid
eavlDataSet
eavlStructuredCellSet
eavlCoordinates
eavlField #0Name: “y”Association: LogicalDim1Values[nj]
eavlField #1Name: “z”Association: LogicalDim2Values[nk]
eavlField #2
![Page 19: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/19.jpg)
RegularStructure: 30 40 30 4 360
FieldName: “x” “y” “z” “μ” “ϴ”Component: 0 0 0 0 0
Name: “x”Association: LogicalDim0Values[ni]
Cells[1]Points[1]Fields[5]
Example: High-Dimensional Grid
eavlDataSet
eavlStructuredCellSet
eavlCoordinates
eavlField #0Name: “y”Association: LogicalDim1Values[nj]
eavlField #1Name: “z”Association: LogicalDim2Values[nk]
eavlField #2Name: “μ”Association: LogicalDim3Values[nμ]
eavlField #3Name: “ϴ”Association: LogicalDim4Values[nϴ]
eavlField #4
![Page 20: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/20.jpg)
RegularStructure: 30 40
FieldName: “lat” “lon”Component: 0 0
Name: “lat”Association: LogicalDim0Values[ni]
Cells[1]Points[2]Fields[3]
Example: Geospatial Data
eavlDataSet
eavlStructuredCellSet
eavlCoordinates
eavlField #0Name: “lon”Association: LogicalDim1Values[nj]
eavlField #1Name: “c”Association: PointsValues[3*npts]
eavlField #2
FieldName: “c” “c” “c”Component: 0 1 2
eavlCoordinates
![Page 21: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/21.jpg)
Example: Molecular Data
Connectivity: the atoms
FieldName: “c” “c” “c”Component: 0 1 2
Name:”atomic number”Association: Cell Set #0Values[ncells #0]
Cells[2]Points[1]Fields[3]
eavlDataSet
eavlExplicitCellSet #0
eavlCoordinates
eavlField #1
Connectivity: the bonds
eavlExplicitCellSet #1
Name: “c”Association: PointsValues[3*npts]
eavlField #0Name: “bond strength”Association: Cell Set #1Values[ncells #1]
eavlField #2
![Page 22: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/22.jpg)
Example: Face-centered Data
Connectivity: volumetric
FieldName: “c” “c” “c”Component: 0 1 2
Cells[2]Points[1]Fields[2]
eavlDataSet
eavlExplicitCellSet
eavlCoordinates
Parent: ( )
eavlAllFacesOfExplicit
Name: “c”Association: PointsValues[3*npts]
eavlField #0Name: “facevariable”Association: Cell Set #2Values[nfaces]
eavlField #1
![Page 23: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/23.jpg)
FILTERING IN EAVL
![Page 24: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/24.jpg)
Data flow networks in EAVL (or not)
• A “Filter” is a stage in a data flow network– Creates a new data set from an old one
• Many operations do not change a mesh structure (assuming data model is sufficiently descriptive)– Arithmetic expressions: only modifies fields– External facelist: points and structure remain– Feature edges: just a new cell set with old points– Smooth, displace, elevate: only modify coordinates
• So: eavlMutator is an alternative to eavlFilter– Modifies a data set in-place
![Page 25: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/25.jpg)
eavlMutator• In-place data set modification• Support for destructive in-place operation– free memory as you go
• Execute multiple mutators simultaneously on the same data set (barring conflicts)– e.g. displace (coords) + threshold (cells) concurrently
• How about data flow network support?– encapsulate an eavlMutator through a
eavlFilterFromMutator facade • Of course, some operations are natively eavlFilters– can facade through eavlMutatorFromFilter (?)
![Page 26: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/26.jpg)
FieldName: “x” “y” “z”Component: 0 0 0
• Explicit cells can be combined with structured coordinates.
Example: Thresholding an RGrid (a)
eavlCoordinates
Name: “x”
Association: LogicalDim0Values[ni]
eavlField#0Name: “y”
Association: LogicalDim1Values[nj]
eavlField#1Name: “z”
Association: LogicalDim2Values[nk]
eavlField#2
RegularStructure: 30 40 30
eavlStructuredCellSet
FieldName: “x” “y” “z”Component: 0 0 0
eavlCoordinates
Name: “x”
Association: LogicalDim0Values[ni]
eavlField#0Name: “y”
Association: LogicalDim1Values[nj]
eavlField#1Name: “z”
Association: LogicalDim2Values[nk]
eavlField#2
Connectivity: (a bunch of cells)
eavlExplicitCellSet
![Page 27: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/27.jpg)
Cells: (…)
Parent: ( )
• A second Cell Set can be added which refers to the first one
Example: Thresholding an RGrid (b)
RegularStructure: 30 40 30
eavlStructuredCellSet eavlSubset
FieldName: “x” “y” “z”Component: 0 0 0
eavlCoordinates
Name: “x”
Association: LogicalDim0Values[ni]
eavlField#0Name: “y”
Association: LogicalDim1Values[nj]
eavlField#1Name: “z”
Association: LogicalDim2Values[nk]
eavlField#2
RegularStructure: 30 40 30
eavlStructuredCellSet
Name: “x”
Association: LogicalDim0Values[ni]
eavlField#0Name: “y”
Association: LogicalDim1Values[nj]
eavlField#1Name: “z”
Association: LogicalDim2Values[nk]
eavlField#2
FieldName: “x” “y” “z”Component: 0 0 0
eavlCoordinates
![Page 28: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/28.jpg)
30x30 or 30x40
eavlStructSubset
30x30 or 30x40
eavlStructSubset
• Add six new subset-cell sets to original mesh
Example: Structured External Facelist
FieldName: “x” “y” “z”Component: 0 0 0
eavlCoordinates
Name: “x”
Association: PointsValues[npts]
eavlField#0Name: “y”
Association: PointsValues[npts]
eavlField#1Name: “z”
Association: PointsValues[npts]
eavlField#2
FieldName: “x” “y” “z”Component: 0 0 0
eavlCoordinates
Name: “x”
Association: PointsValues[npts]
eavlField#0Name: “y”
Association: PointsValues[npts]
eavlField#1Name: “z”
Association: PointsValues[npts]
eavlField#2
RegularStructure: 30 40 30
eavlStructuredCellSet30 40 30
eavlStructCellSet30x30 or 30x40
eavlStructSubset
x6
![Page 29: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/29.jpg)
• No problem-sized data modifications.– Interleaved and separated coordinates can be used simultaneously.
Example: Elevating a Structured Grid
FieldName: “c” “c”Component: 0 1
eavlCoordinates
Name: “c”Association:PointsValue[2*npts]
eavlField#0Name: “val”Association:PointsValues[npts]
eavlField#1
RegularStructure: 30 40
eavlStructuredCellSet
FieldName: “c” “c” “val”Component: 0 1 0
eavlCoordinates
RegularStructure: 30 40
eavlStructuredCellSet
Name: “c”Association:PointsValue[2*npts]
eavlField#0Name: “val”Association:PointsValues[npts]
eavlField#1
![Page 30: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/30.jpg)
• No problem-sized data modifications.– Some axes on logical dims, with others on the points.
Example: Elevating a Regular Grid
FieldName: “x” “y”Component: 0 0
eavlCoordinates
Name: “x”
Association: LogicalDim0Values[ni]
eavlField#0Name: “y”
Association: LogicalDim1Values[nj]
eavlField#1Name: “val”
Association: PointsValues[npts]
eavlField#2
RegularStructure: 30 40
eavlStructuredCellSet
Name: “x”
Association: LogicalDim0Values[ni]
eavlField#0Name: “y”
Association: LogicalDim1Values[nj]
eavlField#1Name: “val”
Association: PointsValues[npts]
eavlField#2
FieldName: “x” “y” “val”Component: 0 0 0
eavlCoordinates
RegularStructure: 30 40
eavlStructuredCellSet
![Page 31: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/31.jpg)
DEALING WITH CONCURRENCY
![Page 32: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/32.jpg)
Concurrency at Multiple Levels
• Distributed Parallelism– Message passing still works well– Avoid global communication• local domain interconnectivity information
– Hybrid (e.g. spatiotemporal) parallelism• Task Parallelism– Fine-grain dependency tracking• e.g. displace (coords) + threshold (cells) concurrently
– eavlMutator helps– single eavlDataSet container class helps
• Thread Parallelism– Fine-grain data parallelism; CUDA, OpenMP
![Page 33: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/33.jpg)
Data Parallelism for Developers
• Functor + iterator paradigm
• Iteration patterns for mesh topologies
• CUDA + OpenMP execution back-ends
![Page 34: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/34.jpg)
A Simple Data-Parallel Operation
void CellToCellDivide(Field &a, Field &b, Field &c){ for_each(i) c[i] = a[i] / b[i];}
void CalculateDensity(...){ //... CellToCellDivide(mass, volume, density);}
Internal Library API Provides ThisAlgorithm Developer Writes This
![Page 35: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/35.jpg)
Functor + Iterator Approach
void CalculateDensity(...){ //... CellToCellBinaryOp(mass, volume, density, Divide());}
template <class T> void CellToCellBinaryOp<T>(Field &a, Field &b, Field &c T &f){ for_each(i) f(a[i],b[i],c[i]);}
struct Divide{ void operator()(float &a, float &b, float &c) { c = a / b; }};
Internal Library API Provides ThisAlgorithm Developer Writes This
![Page 36: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/36.jpg)
Custom Functor
void CalculateDensity(...){ //... CellToCellBinaryOp(mass, volume, density, MyFunctor());}
template <class T> void CellToCellBinaryOp<T>(Field &a, Field &b, Field &c T &f){ for_each(i) f(a[i],b[i],c[i]);}
struct MyFunctor{ void operator()(float &a, float &b, float &c) { c = a + 2*log(b); }};
Algorithm DeveloperWrites These
Internal Library API Provides This
![Page 37: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/37.jpg)
Functor Efficiency on CPU and GPU
• Data: noise.silo• Surface normal
0 μs
100 μs
200 μs
300 μs
400 μs
500 μs
600 μs
unoptimized optimized unoptimized optimized
Intel Xeon L5420 NVIDIA GeForce 8800GTS
Functor Inline
![Page 38: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/38.jpg)
Binding Values to Functorsstruct ScaleByConst{ float scale; ScaleByConst(float s) : scale(s) { } void operator()(float &a, float &b) { b = a * scale; }};
void CalculateDensity(...){ //... cell_volume = mesh_volume / mesh_numcells; CellToCellUnaryOp(mass, density, ScaleByConst(1.0/cell_volume));}
![Page 39: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/39.jpg)
DATA PARALLELISM BASICS
![Page 40: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/40.jpg)
Map with 1 input, 1 output
Simplest data-parallel operation. Each result item can be calculated from its corresponding input item alone.
x 3 7 0 1 4 0 0 4 5 3 1 0
0 1 2 3 4 5 6 7 8 9 10 11
6 14 0 2 8 0 0 8 10 6 2 0
struct f { float operator()(float x) { return x*2; }};
result
![Page 41: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/41.jpg)
Map with 2 inputs, 1 output
With two input arrays, the functor takes two inputs. You can also have multiple outputs.
x 3 7 0 1 4 0 0 4 5 3 1 0
0 1 2 3 4 5 6 7 8 9 10 11
5 11 2 2 12 3 9 9 10 4 3 1
struct f { float operator()(float a, float b) { return a+b; }};
result
y 2 4 2 1 8 3 9 5 5 1 2 1
![Page 42: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/42.jpg)
Scatter with 1 input (and thus 1 output)
Possibly inefficient, risks of race conditions and uninitialized results. (Can also scatter to larger array if desired.)Often used in a scatter_if–type construct.
x 3 7 0 1 4 0 0 4 5 3 1 0
0 1 2 3 4 5 6 7 8 9 10 11
0 1 3 0
No functor
result
indices
4
2 4 1 5 5 0 4 2 1 2 1 4
![Page 43: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/43.jpg)
Gather with 1 input (and thus 1 output)
Unlike scatter, no risk of uninitialized data or race condition. Plus, parallelization is over a shorter indices array, and caching helps more, so can be more efficient.
x 3 7 0 1 4 0 0 4 5 3 1 0
0 1 2 3 4 5 6 7 8 9 10 11
7 3 0 3 1
No functor
result
indices 1 9 6 9 3
![Page 44: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/44.jpg)
Reduction with 1 input (and thus 1 output)
Example: max-reduction. Sum is also common.Often a fat-tree-based implementation.
x 3 7 0 1 4 0 0 4 5 3 1 0
0 1 2 3 4 5 6 7 8 9 10 11
7result
struct f { float operator()(float a, float b) { return a>b ? a : b; }};
![Page 45: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/45.jpg)
Inclusive Prefix Sum (a.k.a. Scan)with 1 input/output
Value at result[i] is sum of values x[0]..x[i].Surprisingly efficient parallel implementation.Basis for many more complex algorithms.
x 3 7 0 1 4 0 0 4 5 3 1 0
0 1 2 3 4 5 6 7 8 9 10 11
3 10 10 11 15 15 15 19 24 27 28 28
No functor.
result
+ + + + + + + + + + +
![Page 46: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/46.jpg)
Exclusive Prefix Sum (a.k.a. Scan)with 1 input/output
Initialize with zero, value is sum of only up to x[i-1].May be more commonly used than inclusive scan.
x 3 7 0 1 4 0 0 4 5 3 1 0
0 1 2 3 4 5 6 7 8 9 10 11
0 10 10 11 15 15 15 19 24 27 28
No functor.
result 3
+ + + + + + + + + + +0
![Page 47: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/47.jpg)
DATA PARALLELISM ON MESHES
![Page 48: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/48.jpg)
Example: Surface Normal
• For each 2D cell(i.e. each polygon):– Get three adjacent points– Pair-wise vector subtract– Cross product
• Data-parallel:– Repeat for all cells
![Page 49: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/49.jpg)
Example: Surface Normal
• INPUT:– 3-dimensional
coordinates array on the mesh NODES
– example: length = 9
• OUTPUT:– 3-component surface
normals array onthe mesh CELLS
– example: length = 4
![Page 50: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/50.jpg)
Under the Covers: Node-to-Cell on CPUvoid NodeToCellOp3::ExecuteCPU(){#pragma omp parallel for for (int i=0; i<input->NumCells(); i++) { // get cell node indices
int nNodes, nodeIds[8]; float nodeValues[3][8]; conn.GetCellNodes(index, nNodes, nodeIds);
// get coordinates for nodes for (int i=0; i<nNodes; i++) { nodeValues[0][i] = array0[nodeIds[i]]; nodeValues[1][i] = array1[nodeIds[i]]; nodeValues[2][i] = array2[nodeIds[i]]; }
// call functor functor(nodeValues[0], nodeValues[1], nodeValues[2], &out0[i], &out1[i], &out2[i]); } }
![Page 51: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/51.jpg)
Under the Covers: Node-to-Cell on GPU
void NodeToCellOp3::ExecuteGPU(){ float *d_arr0 = (float*)array0->GetCUDAArray(); float *d_arr1 = (float*)array1->GetCUDAArray(); float *d_arr2 = (float*)array2->GetCUDAArray(); float *d_out = (float*)output0->GetCUDAArray();
// calculate CUDA thread grid num blocks & threads // based on input->NumCells()
nodeToCellKernel3<<<nb,nt>>>(d_arr0, d_arr1, d_arr2, d_out0, d_out1, d_out2, conn, functor);}
![Page 52: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/52.jpg)
Under the Covers: The CUDA Kerneltemplate <class F>__global__ void NodeToCellKernel3(float *array0, float *array1, float *array2, float *out0, float *out1, float *out2, ExplicitConnectivity conn, F functor){ const int index = blockIdx.x * blockDim.x + threadIdx.x; // get cell node indices int nNodes, nodeIds[8]; float nodeValues[3][8]; conn.GetCellNodes(index, nNodes, nodeIds); // get coordinates for nodes for (int i=0; i<nNodes; i++) { nodeValues[0][i] = array0[nodeIds[i]]; nodeValues[1][i] = array1[nodeIds[i]]; nodeValues[2][i] = array2[nodeIds[i]]; } // call functor functor(nodeValues[0], nodeValues[1], nodeValues[2], &out0[i], &out1[i], &out2[i]);}
![Page 53: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/53.jpg)
WRITING ALGORITHMS IN EAVL
![Page 54: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/54.jpg)
Face Surface Normal Functor+Iteratorstruct PolyNormalFunctor{ void operator()(int nvals, int shapetype, float x[], float y[], float z[], float *nx, float *ny, float *nz) { // get two adjacent edge vectors float ax = x[1]-x[0], ay = y[1]-y[0], az = z[1]-z[0]; float bx = x[2]-x[1], by = y[2]-y[1], bz = z[2]-z[1]; // calculate their cross product *nx = ay*bz - az*by; *ny = az*bx - ax*bz; *nz = ax*by - ay*bx; }};
void CalculateFaceNormals(...){ eavlExecutor::AddOperation( new eavlTopologyMap_3_3(inputcells, EAVL_NODES_OF_CELLS, xcoord, ycoord, zcoord, xnormal, ynormal, xnormal, PolyNormalFunctor()));}
![Page 55: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/55.jpg)
Making it Easy for Developers
• Single functor code works on either CPU or GPU• Iteration patterns have multiple back-ends• CPU vs GPU execution choice is made at runtime• EAVL array class automatically supports
heterogeneous memory spaces for GPUs• Internal APIs support CUDA if developers want to
write their own CUDA kernels
![Page 56: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/56.jpg)
Algorithms are Sequences of Operations(a) calculate face surface normals
with node-to-cell operation
(b) average to point normals via cell-to-node operation
![Page 57: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/57.jpg)
EXAMPLE: THRESHOLD
![Page 58: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/58.jpg)
47 52 63
32 49
31
Starting Mesh
We want to threshold a mesh based on its density values (shown here).
43 47 52 63
32 38 42 49
31 37 41 38
density 43 47 52 63 32 38 42 49 31 37 41 38
0 1 2 3 4 5 6 7 8 9 10 11
If we threshold 35 < density < 45, we want this result:
43
38 42
37 41 38
![Page 59: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/59.jpg)
Which Cells to Include?
Evaluate a Map operation with this functor:struct InRange { float lo, hi; InRange(float l, float h) : lo(l), hi(h) { } int operator()(float x) { return x>lo && x<hi; }}
1 0 0 0
0 1 1 0
0 1 1 1
density 43 47 52 63 32 38 42 49 31 37 41 38
0 1 2 3 4 5 6 7 8 9 10 11
1 0 0 0 0 1 1 0 0 1 1 1inrange
InRange()
![Page 60: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/60.jpg)
How Many Cells in Output?
Evaluate a Reduce operation using the Add<> functor.We can use this to create output cell length arrays.
1 0 0 0
0 1 1 0
0 1 1 1
0 1 2 3 4 5 6 7 8 9 10 11
1 0 0 0 0 1 1 0 0 1 1 1inrange
6result
plus
![Page 61: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/61.jpg)
Where Do the Output Cells Go?
Inputindices
Outputindices
0 1 2 3
4 5 6 7
8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5output cell
0
1 2
3 4 5
input cell
How do we create this mapping?
![Page 62: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/62.jpg)
Create Input-to-Output Indexing?
Exclusive Scan (exclusive prefix sum) gives us the output index positions.
0
1 2
3 4 5
0 1 2 3 4 5 6 7 8 9 10 11
1 0 0 0 0 1 1 0 0 1 1 1inrange
0 1 1 1 1 1 2 3 3 3 4 5startidx
+ + + + + + + + + + +0
![Page 63: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/63.jpg)
startidx
Create Output-to-Input Indexing?
We want to work in the shorter output-length arrays and use gathers. A specialized scatter in EAVL creates this reverse index.
0 1 2 3 4 5 6 7 8 9 10 11
0 5 6 9 10 11revindex
0 1 1 1 1 1 2 3 3 3 4 5
0
5 6
9 10 11
![Page 64: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/64.jpg)
density
Gather Input Mesh Arrays to Output?
We can now use simple gathers to pull input arrays (density, pressure) into the output mesh.
0 1 2 3 4 5 6 7 8 9 10 11
43 47 52 63 32 38 42 49 31 37 41 38
0 5 6 9 10 11revindex
43 38 42 37 41 38
43
38 42
37 41 38
output_density
![Page 65: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/65.jpg)
PLANS AND STATUS
![Page 66: EAVL EAVL E XTREME - SCALE A NALYSIS AND V ISUALIZATION L IBRARY Jeremy Meredith SDAV Next-Gen Library Meeting September, 2012.](https://reader035.fdocuments.net/reader035/viewer/2022062407/56649e175503460f94b032d3/html5/thumbnails/66.jpg)
Plans and Status• Now Open Source (BSD “2-clause”)– Project website at http://ft.ornl.gov/eavl – Source code, docs at https://github.com/jsmeredith/EAVL
• Data Model– Most infrastructure in place– Still a few areas not fully fleshed out– The client-facing API needs work
• Execution Model– Currently imperative style– Exploring pipeline paths, contract/metadata needs
• Productization Status– More iterators– More optimization– Lots of internal cleanup