Terascale Terascale Simulation Simulation Tools and Tools and TechnologiTechnologi

eses

The TSTT Interface Definition EffortLori Freitag Diachin

Lawrence Livermore National Lab

2

The TSTT Center

• Goal: To enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales – Adaptive methods– Composite or hybrid solution strategies– High-order discretization strategies

• Barrier: The lack of easy-to-use interoperable meshing, discretization, and adaptive tools requires too much software expertise by application scientists

The TSTT center recognizes this gap and is addressing the technical and human barriers preventing use of

adaptive, composite, hybrid methods

3

TSTT brings a broad set of meshing-discretization communities together

• Diverse Partners– Overture

– AOMD/Trellis

– CUBIT – NWGrid/Phys

– – Frontier

– Mesquite

– Nek5000

• Meshing Technologies– Structured/Overset– Mixed element hybrid– Unstructured hex– Tetrahedral– Variational approaches– Mesh quality improvement

• Adaptive Technologies– Mesh refinement– Front tracking

• Software Frameworks and PDE Solvers

• Significant Applications Experience

NWGrid mesh

MEGA Boundary Layer Mesh (RPI)

Overture Mesh (LLNL)

CUBIT Mesh (Sandia)

4

The Challenge

• These tools all meet particular needs, but– They do not interoperate to form hybrid, composite meshes– They cannot be easily interchanged in an application

• In general the technology requires too much software expertise from application scientists– Difficult to improve existing codes– Difficult to design and implement new codes

• Our goal is to develop interchangeable and interoperable software components for meshing and discretization

5

TSTT interoperability goals require services build on a common infrastructure

• All tools must build on a common abstract framework and data model– Provides a single reference for

PDE simulation data hierarchy

• Defining common interfaces is critical– Required for interchangeability of

similar tools– Allows vertical integration of tools

• Applications write to the TSTT specification once and can use all TSTT services

GeometryInformation(Level A)

Full GeometryMeshes(Level B)

MeshComponents(Level C)

6

TSTT data model abstracts PDE-simulation data hierarchy

• Core Data Types– Geometric Data: provides a high level description of the boundaries

of the computational domain; e.g., CAD, image, or mesh data

– Mesh Data: provides the geometric and topological information associated with the discrete representation of the domain

– Field Data: provides access to the time dependent physics variables associated with application solution. These can be scalars, vectors, tensors, and associated with any mesh entity.

• Data Relation Managers– Provides control of the relationships among the core data types

– Resolves cross references between entities in different groups

– Provides functionality that depends on multiple core data types

Each core data typehas a TSTT interface• Mesh: TSTTM• Geometry: TSTTG• Fields: TSTTF

Each core data typehas a TSTT interface• Mesh: TSTTM• Geometry: TSTTG• Fields: TSTTF

7

An overarching philosophy guidesour interface definition efforts

• Maintain data structure neutrality

• Create a small set of interfaces that existing packages can support– Small set of ‘core’ functions that must be

implemented

– Larger set of functions supported by reference implementations

• Balance performance and flexibility

• Work with a large tool provider and application community to ensure applicability

• Target both specific uses of TSTT services and broad dissemination as CCA components

Lowers the burden for adoption of the interface

Performance is critical for kernel computations involving mesh access; flexibility is critical for covering a broad usage spectrum

CCA provides infrastructure and guidance for domain-specific interface definition efforts

8

The TSTT mesh data model consists of entities, entity sets and meshes• Entity Definition

– Unique type and topology– Canonical ordering defines adjacency

relationships

• Entity Set Definition– Arbitrary grouping of TSTT mesh entities– There is a single “Root Set”– Relationships among entity sets

• Contained-in• Hierarchical

• Mesh Definition– Simple meshes are a set of connected, non-

overlapping entities related through adjacency relationships

– Meshes can contain collections of simple meshes

9

Core functions must be implemented to be TSTT compliant

• The smallest set of functions required to be considered TSTT compliant

• Provides basic access to vertex coordinates and adjacency information– Mesh loading and saving– Accessing global information such as the root set, geometric

dimension, number of entities of a given type or topology– Primitive array access for entity sets– Global entity arrays for entity sets

• Reference implementations can be used to provide advanced capabilities needed for TSTT services– Requires a copy of mesh data– Allows incremental implementation of advanced functions

10

Advanced traversal functions provide additional flexibility and performance

• Single Entity Traversal– Iterator functionality (init, next, reset, end)

– Query (topology, type, adjacency)

• Workset Traversal– Block iterators allow access of entities in a user-defined array size

– Array based query functions for topology, type, adjacency

– Increases performance by reducing the number of calls through the interface

– Interface is not as intuitive

11

Advanced functionality is supported through optional interfaces

• Entity Sets– Creation/Destruction– Boolean Operations– Relations

• Tags– Allows the incorporate user-defined

data in the TSTT data model– Associated with entities, meshes,

and entity sets• Mesh Modification

– Simple mesh geometry and topology modification– Requires simple classification mechanisms against the

geometric model– Intended to support higher-level functionality

12

The geometry interface provides access to the computational domain

• Must support – automatic mesh generation– mesh adaptation– tracking domain changes– relating information between alternative

discretizations

• Builds on boundary representations of geometry

• Used to support various underlying representations – Commercial modelers (e.g., Parasolid, ACIS)– Modelers that operate from standard files (e.g. IGES, STEP)– Models constructed from an input mesh

13

Both basic and advanced functionalities are supported in the geometry interface

• Model loading and initiation• Topological queries of entities

and adjacencies• Pointwise geometric shape

interrogation• Parametric coordinate

systems• Model topology modification• Tags

14

Relating mesh and geometry data is critical for advanced TSTT services

• Required for e.g., adaptive loops, mesh quality improvement

• Mesh/Geometry Classification Interface– Manages the relationship between the high level

geometric description and the mesh– Called by an application that knows about both

• Capabilities– For a given mesh entity, get the geometric entity

against which it is classified– Establish a classification relationship between a

mesh entity and a geometric entity

15

Preliminary interface implementations are well underway

• Working documents describing the mesh and geometry data models as well as the overall vision

• TSTT Mesh Interface (version 0.6) complete• Preliminary interfaces for geometry and mesh/geometry

classification defined• Implementations

– AOMD, FronTier, GRUMMP, MOAB, NWGrid, Overture for mesh– CGM for geometry

• C, C++, and Fortran language interoperability through SIDL/Babel (CCA)– Analyzing performance ramifications of SIDL/Babel language

interoperability tools (joint with the CCA)

16

Performance studies will provide ‘best practices’ guidelines

• Working with SIDL/Babel team to characterize the interface performance

• Focus on entity-based access– Single entity iterators– Workset iterators

• Compare– Native data structures– Native language interfaces– SIDL/Babel interfaces

Workset Iterators

Initialize Workset(entity_type, num_entities, &wrk_iter)

While (!done){

GetNextWorkset(&entity_handles[ ], &num, wrk_iter);

GetInfo(entity_handles);

for i=0:num-1 {

do_something(entity_handle[i]);

}

}

Destroy Workset(wrk_iter);

Workset Iterators

Initialize Workset(entity_type, num_entities, &wrk_iter)

While (!done){

GetNextWorkset(&entity_handles[ ], &num, wrk_iter);

GetInfo(entity_handles);

for i=0:num-1 {

do_something(entity_handle[i]);

}

}

Destroy Workset(wrk_iter);

17

Preliminary performance focus on TSTT mesh access patterns

Relative Wall Clock Time for Each Variant

0

100

200

300

400

500

1 100Work Set Size

Pe

rce

nta

ge

of

Na

tiv

e L

ink

ed

Lis

t

Native Array

Native Linked List

Native Interface

SIDL Direct

SIDL Memcpy

SIDL For-loop

• Workset Iterators– Increased workset size amortizes overhead – 15-40% compared to native data structures– SIDL a few percent more than native language– Single entity access via workset iterators 2X-

4X for native and SIDL interfaces

• Single entity iterators have similar performance to a workset size of 5-10

• Array based access for NWGrid has .5% overhead independent of mesh size

• Ongoing research to further improve performance when interoperability is not critical (w/ PERC)

Relative Wall Clock Time for Interfaces

75

100

125

150

175

200

225

0 25 50 75 100Work Set Size

Per

cen

tag

e o

f N

ativ

e In

terf

ace

Native Interface

SIDL Direct

SIDL Memcpy

SIDL For-loop

18

The TSTT mesh interface working group uniquely engages a large number of people

Internal– ANL: Fischer,

Buschelman– LLNL: Freitag, Chand,

Henshaw, Dahlgren– PNNL: Trease– RPI: Shephard, Seole– SNL: Tautges, Kim– SUNY SB: Li, Fix

Internal– ANL: Fischer,

Buschelman– LLNL: Freitag, Chand,

Henshaw, Dahlgren– PNNL: Trease– RPI: Shephard, Seole– SNL: Tautges, Kim– SUNY SB: Li, Fix

External (active)– UofC: Loy– UBC: Ollivier-Gooch

External (active)– UofC: Loy– UBC: Ollivier-Gooch

External (monitoring)– UIUC: Guoy– Boeing: Michal (UGC)– LLNL: Keasler– SNL: Pebay, Haynes,

Edwards, Jung– ElemTech: Meyers– UU: Parker– ORNL: Bernholdt

External (monitoring)– UIUC: Guoy– Boeing: Michal (UGC)– LLNL: Keasler– SNL: Pebay, Haynes,

Edwards, Jung– ElemTech: Meyers– UU: Parker– ORNL: Bernholdt

Managing the interactions across institutions• Face to face meetings (all-hands and opportunistic)• Telecons (approximately every three weeks)• Email discussion group (39 participants from 13 institutions)• Working documents and SIDL files• CVS repository

19

Defining the TSTT interfaces has been an interesting challenge

• Interface definition is harder than we anticipated• Cannot achieve the 100 percent solution, so...

– What level of functionality should be supported?– How to best support existing (different) packages?

• The devil is in the details– Initial interface completed a year ago– Lots of interesting issues

• Invariant handles

• Consistent error handling across packages

• Memory management issues

• Core interface functions

• Performance, performance, performance

20

TSTT services support SciDAC applications developers

• Domains from multiple geometry source• Use of alternative mesh generators• Use alternative equation discretization methods and

tools• Application of adaptive control of the

– Geometry - shape optimization

– Mesh - discretization error control

– Both - evolving geometry simulations

• Combining multiple analysis codes to solve multiphysics problems

21

TSTT services

• “Current” TSTT Services – Mesh services libraries (AOMD, MOAB, NWGrid)– Mesh quality improvement (Mesquite)– Discretization libraries (Overture, Trellis)– Parallel mesh generation (MOAB) – Front tracking (FronTier)– Adaptive control loops (combination of tools)

• Services Under Development– Geometry interface functions (not doing geometry)– Fields library– Mesh control for shape optimization

22

Mesquite Mesh Quality Improvement

• Stand alone library of mesh quality improvement algorithms– 2D/3D unstructured and hybrid meshes– Triangles, Quads, Tets, Hex elements– State-of-the-art optimization-based improvement schemes– Joint work with TOPS created fastest mesh quality optimizer in Mesquite

tool suite – improves 1M elements in 15 seconds• TSTT interface to access mesh and geometry information

– Vertex coordinates, element connectivity, tags– Successfully integrated with AOMD, MOAB, Overture, Cubit

• Application uses– Surface mesh smoothing on complex geometries for accelerator modeling– Mesh quality analysis for SLAC Tau3P code– Mesh deformation in design optimization problems– Anisotropic smoothing for error minimization

23

TSTT developed a new interoperable AMR/Front tracking scheme

• Combine front tracking in Frontier with adaptive techniques in Overture– Increase problem resolution using

AMR– Maintain exact interface

representation capabilities

• Applications– Richtmeyer-Meshkov instability– Diesel jet spray break up

• Using TSTT interface to combine front-tracking schemes with NWgrid capabilities

24

TSTT/SLAC collaboration provides adaptive mesh capabilities for Omega3P

• Omega3P is SLAC’s quadratic FEM eigensolver for computing normal modes in RF cavities

• Extremely accurate solutions (0.01% error) are required for requirements of accelerator design

• Using new TSTT technology developed at RPI, adaptive mesh capability has been provided for Omega3P

• Simulations achieve same accuracy with 1/3 the number of unknowns

3 levels of refinement

First Refinement Final Adaptive MeshInitial Mesh

25

Using geometry operatorsmeans alternate solid modelers can be inserted

Using geometry operatorsmeans alternate solid modelers can be inserted

Using TSTT mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted

Using TSTT mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted

Using TSTT field operators allows easy construction of alternative error estimators

Components used in construction of adaptive loop for SLAC

Projection-based error estimatorused to construct new mesh sizefield given to mesh modification

Mesh adaptation based on local modification linkeddirectly to CAD

Mesh adaptation based on local modification linkeddirectly to CAD

UnalteredSLAC code

UnalteredSLAC code

Error estimatorsfrom RPI and SLAC

Error estimatorsfrom RPI and SLAC

26

Contact Information

• TSTT Web Site: www.tstt-scidac.org

• David Brown: dlb@llnl.gov• Lori Freitag Diachin: diachin2@llnl.gov• Jim Glimm: glimm@ams.sunysb.edu