Progress on Component-Based Subsurface Simulation I: Smooth

Particle Hydrodynamics

Bruce Palmer

Pacific Northwest National Laboratory

Richland, WA

Smooth Particle Hydrodynamics

Use a discrete sampling of points to approximate continuum hydrodynamic fields

where W is a smooth, normalized weighting function with support h and ρ is

),()( hWA

mA ii

i

ii rrr

),()( hWm ii

i rrr

Smooth Particle Hydrodynamics

The equations of motion for the particles are

ij

j

j

i

i

jj

i PPm

dt

d22

v

where P is the pressure and is a stress tensor that depends only on the properties of particles at locations i and j.

Smooth Particle Hydrodynamics

• Algorithmically similar to molecular dynamics

• Explicit time integration of particle trajectories

• Finite range, pairwise interactions between particles

• For large simulations, particles are distributed on processors based on spatial location

Spatial Distribution

P0 P1 P2 P3

P4 P5 P6 P7

Spatial Decomposition Algorithm:

• Partition particles among processors

• Update coordinates at every step

• Update partitioning after fixed number of steps

Componentization

• Defining components: What functionality and data belong together in a component

• Granularity: At what level is componentization compatible with performance?

• Abstraction of Interfaces: Can interfaces be defined that support multiple implementations representing different models and/or algorithms?

• Resource Allocation: Which components allocate memory and how is this communicated to other components?

CCA Framework for SPH

• Three major components (and numerous minor ones)

• Time integration component: update coordinates and velocities at each time step

• Force evaluation component: Evaluate forces on each particle

• Communication component: manage movement of particle data between processors

CCA Framework for SPH

Communication

ForcesTime Integration

It works! (Mostly)SPH simulation of flow in porous media on MPP2

Communication layer

• Use integer handles to label each “action” specified by user

• 5 types of action supported: “shuffle”, “sort”, “update”, “gather”, “scatter-add”

• Actions support the distribution of particles to processors based on spatial location and the gathering and scattering of particles into and from a buffer region of some user-specified width surrounding the spatial domain owned by individual processors

Communication layer

• Each handle can be assigned an action. An arbitrary number of integer or double vectors pairs can also be assigned to this handle.

• When the user calls the transfer function on a handle, all data associated with that handle is moved according to the appropriate action

Communication layer

• The communication layer is also the repository of geometry information (simulation size, cell size)

• Only coordinates and particle index have any special status

Observations, Requests, etc.

• Developer environment (we need one)– Where’s Bocca?– Support for multiple files in components– Support top level setting of libraries, compiler options,

etc. These can be overwritten at component level– Some kind of facility for duplicating existing

components to new components (facilitates experimentation)

– Can a component be easily generated if a Babel interface for a software package already exists? (ITAPS)

Observations, Requests, etc.

• 32 and 64 bit platforms– Integer arrays are particularly a problem (I

think kluges exist for single variables)– sidl_integer data type?

Observations, Requests, etc.

• Interactions with workflow environments (I think some of this stuff already exists but is undocumented)– Command line arguments– Support for versioning documentation that can

be exported to output files– Need some way of extracting port names

without looking at setServices method (currently the only way to do this is to use the GUI)

Observations, Requests, etc.

• Documentation– How do you use sidl_opaque data types?– How do exceptions work?