1

FEM Framework Tutorial

Sayantan Chakravorty10/19/2004

2

Roadmap

Why use FEM? FEM Concepts FEM Program Structure FEM Basic Calls FEM Advanced Calls Extra Features

3

Why use FEM?

4

Why use the FEM Framework?

Makes parallelizing a serial code faster and easier Handles mesh partitioning Handles communication Handles load balancing (via Charm)

Allows extra features NetFEM Visualizer Collision Detection Library Parallel Refinement Library IFEM Matrix Library

5

Why not use the FEM Framework?

Does not help you write serial code But it does help with parallel

Another thing to learn But it has a manual and examples

Another thing to break But it runs on native MPI as well as

AMPI

6

Fem Framework Users CSAR

Rocflu: Fluids solver CPSD

SpaceTime meshing Frac3D

Fracture Mechanics Dendritic Growth

Metal Solidification process

7

FEM Concepts

8

FEM Basics

FEM programs manipulate elements and nodes

Element is a portion of problem domain, surrounded by nodes

Node is one point in the domain

9

Serial FEM Mesh

N5N4N2E3

N4N2N1E2

N4N3N1E1

Surrounding Nodes

Element

10

Partitioned Mesh

N3N2N1E2

N4N3N1E1

Surrounding Nodes

Element

N3N2N1E1

Surrounding Nodes

Element

Shared Nodes

N3N4

N1N2

BA

11

FEM Parallel Model: Shared Nodes

“Shared Node” model Element computations based on

values of surrounding nodes Node values are sum of

surrounding elements

Example: Mechanical Simulation Element stresses are computed

from locations of element’s nodes Sum up forces at nodes

12

FEM Mesh: Node Communication

Summing forces from other processors only takes one call:

FEM_Update_field

Adds values from shared nodes

13

FEM Parallel Model: Ghosts

“Ghost” model Element computations based only

on values of surrounding nodes and elements

Example: Fluid Dynamics Element pressures and velocities

come from neighboring element pressures and velocities, and node locations

14

FEM Mesh: Ghosts

1 2 3 4

1 2Ghostof3

Ghost of2

3 4

Serial Mesh

Left Chunk Right Chunk

Partitioning

Communication

15

FEM Program Structure

16

Serial FEM Program Structure

read mesh, connectivity, boundary conditionstime loop           element loop- Element deformation applies forces to surrounding nodes           node loop- Forces and boundary conditions change node positions end time loopwrite out mesh data for postprocessing

17

Parallelization

Partition the FEM Mesh into multiple chunks

Distribute elements, replicate shared nodes and/or add ghosts Keep track of communication

Partition so that communication is minimized

18

Parallel FEM Program

read/get chunk mesh data, connectivity, shared nodeschunk time loop           element loop- Element deformation applies forces to surrounding nodes           <update forces on shared nodes>           node loop- Forces and boundary conditions change node positions end time loopwrite chunk mesh data for postprocessing

19

FEM Framework Program

Consists of at least two user-written subroutines init driver

init is called on chunk 0 driver is called on every chunk

20

init

subroutine init           read the serial mesh and configuration data inform the framework about the mesh end subroutine

21

driver

subroutine driver           get local mesh chunk           time loop                FEM computations                communication                more FEM computations           end time loop end subroutine

22

Structure of an FEM Application

init()

Update Update Update

driver driver driver

23

FEM Mesh Access Calls

24

void FEM_Mesh_data(

int mesh,int entity,int attr, void *data, int first,

int length,int datatype,

int width);

void FEM_Mesh_data(

int mesh,int entity,int attr, void *data, int first,

int length,int datatype,

int width);

Get/Set, and multi-mesh Get/Set, and multi-mesh supportsupportNODE, ELEM, SPARSE NODE, ELEM, SPARSE

(+GHOST)(+GHOST)DATA, CONN, SYM, DATA, CONN, SYM, GLOBALNO,…GLOBALNO,…

Apply to first…first+length-1Apply to first…first+length-1

User data: width x length User data: width x length arrayarray

User data User data formattingformatting

25

FEM_Mesh_data(

mesh,FEM_NODE,FEM_DATA+23, coord, 0,nNodes, FEM_DOUBLE,3

);

Mesh Access: Example

e.g., e.g., FEM_Mesh_default_read()FEM_Mesh_default_read()We’re changing node We’re changing node valuesvaluesUser data (tag User data (tag 23)23)

Change all the Change all the nodesnodes

An array of 3 x nNodes An array of 3 x nNodes doublesdoubles

3 3 doubles/noddoubles/nodee

26

FEM Communication Calls

27

Node Fields Framework handles combining data

for shared nodes and keeps them in sync

Framework does not understand meaning of node fields, only their location and types

Framework needs to be informed of locations and types of fields

Create_field once, Update_field every timestep

28

Create a Field

integer function FEM_Create_simple_field(datatype, len)

integer, intent(in) :: datatype, len

29

Update Field: Shared Nodes

subroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes

30

FEM Ghost Layers

31

Ghost Elements: Overview Most FEM programs

communicate via shared nodes Some computations require

read-only copies of remote elements—“ghosts” Stencil-type finite volume

computation Many kinds of mesh modification

32

Ghosts: 2D Example

1 2 3 4

1 2Ghostof3

Ghost of2

3 4

Serial Mesh

Left Chunk Right Chunk

33

Building Ghosts: Add ghost elements layer-by-layer from init A chunk will include ghosts of all the

elements it is connected to by “tuples”—sets of nodes

For 2D, a tuple might be a 2-node edge For 3D, a tuple might be a 4-node face You specify a ghost layer with

FEM_Add_ghost_layer(tupleSize,ghostNodes) ghostNodes indicates whether to add ghost

nodes as well as ghost elements.

34

Building Ghosts: FEM_Add_ghost_elem(e,t,elem2tuple) e is the element type t is the number of tuples per element elem2tuple maps an element to its tuples:

A tupleSize by t array of integers Contains element-local node numbers

Repeat this call for each ghost element type

35

Ghosts: Node adjacency/* Node-adjacency: triangles have 3 nodes */

FEM_Add_ghost_layer(1,0); /* 1 node per tuple */

const static int tri2node[]={0,1,2};

FEM_Add_ghost_elem(0,3,tri2node);

0

1 2

36

Ghosts: Edge adjacency

/* Edge-adjacency: triangles have 3 edges */

FEM_Add_ghost_layer(2,0); /* 2 nodes per tuple */

const static int tri2edge[]={0,1, 1,2, 2,0};

FEM_Add_ghost_elem(0,3,tri2edge);

0

1 2

37

Extracting and Using Ghosts Ghosts are always given larger

numbers than non-ghosts—that is, ghosts are at the end

FEM_Get_node_ghost() and FEM_Get_elem_ghost(e) Return the index of the first ghost node

or element

FEM_Update_ghost_field(fid,e,data) Obtain other processor’s data (formatted

like fid) for each ghost element of type e

0 eg

38

Update Field: Ghosts

subroutine FEM_Update_ghost_field(fid,elType,elts) integer, intent(in) :: fid,elType varies, intent(inout) :: elts

39

Ghost Example: Mesh

40

Ghost Example: Ghost Elements

41

FEM Installation

42

Where to Get It ?FEM Framework is included in Charm++ distribution, available under CVSCSH:setenv CVSROOT ":pserver:checkout@charm.cs.uiuc.edu:/cvsroot"Or BASH:export CVSROOT=":pserver:checkout@charm.cs.uiuc.edu:/cvsroot"

You should now be able to do a> cvs login(no password needed, just type [Enter] at prompt)

and then> cvs co -P charmto get the entire Charm++ source: FEM is in charm/src/libs/ck-libs

43

How to Build It ?

> cd charm

and do

> ./build FEM net-linux -O

This will make a net-linux directory, with bin, include, lib etc subdirectories.

Platforms: net-sol, mpi-origin, mpi-linux etc.

44

How to Compile & Link ? Use “charmc”: available under bin

a multi-lingual compiler driver, understands f90

Knows where modules and libraries are Portable across machines and compilers

Linking use “-language femf” : for F90 Use “–language fem” : for C/C++

See example Makefiles charm/examples/fem/…

45

How to Run ?

Charmrun A portable parallel job execution

script Specify number of processors: +pN Specify number of chunks: +vpN Special “nodelist” file for net-*

versions

46

Example

Nodelist File: $(HOME)/.nodelist

group main host tur0001.cs.uiuc.edu host tur0002.cs.uiuc.edu host tur0003.cs.uiuc.eduetc…

./charmrun pgm +p4

47

Advanced FEM Calls

48

Advanced: FEM Migration

49

Advanced API: Migration Chunks may not be computationally

equal: Results in load imbalance Multiple chunks per processor Chunks cannot have writable global

data Automatic load balancing

Migrate chunks to balance load How to migrate allocated data for

chunks ? Embed it in a user-defined type

50

Chunk Data Example

MODULE my_block_mod        TYPE my_block          INTEGER :: n1,n2x,n2y          REAL*8, POINTER, DIMENSION(:,:) :: arr        END TYPE END MODULE

51

Pack/Unpack (PUP) RoutineSUBROUTINE pup_my_block(p,m)        USE my_block_mod        USE pupmod        INTEGER :: p        TYPE(my_block) :: m        call fpup_int(p,m%n1)        call fpup_int(p,m%n2x)        call fpup_int(p,m%n2y)        IF (fpup_isUnpacking(p)) THEN          ALLOCATE(m%arr(m%n2x,m%n2y))        END IF        call fpup_doubles(p,m%arr,m%n2x*m%n2y)        IF (fpup_isDeleting(p)) THEN          DEALLOCATE(m%arr)        END IF END SUBROUTINE

52

Registering Chunk Data

!- Fortran driver subroutine           use my_block_mod interface       subroutine pup_my_block(p,m)        use my_block_mod         INTEGER :: p           TYPE(my_block) :: m       end subroutine end interface

TYPE(my_block) :: m

CALL FEM_Register(m,pup_my_block)

53

Migration Every chunk driver calls

FEM_Migrate Framework calls PUP

for getting the size of packed data For packing data

Chunk migrates to new processor

Framework calls PUP for unpacking

Driver returns from FEM_Migrate

54

Advanced: Complicated Fields

55

Node Fields

56

FEM_Create_Fieldfunction integer :: FEM_Create_Field( base_type, vec_len, offset, dist) integer, intent(in) :: base_type, vec_len, offset, dist

Base_type

•FEM_BYTE- INTEGER*1, or CHARACTER*1 •FEM_INT- INTEGER*4 •FEM_REAL- REAL*4 •FEM_DOUBLE- DOUBLE PRECISION, or REAL*8

57

Create_field Example

! 3D Force for each node! stored as 3*n real*8 array

REAL*8 ALLOCATABLE, DIMENTION(:) :: nodeForce INTEGER :: fid

... allocate nodeForce as 3*n_nodes...

fid = FEM_Create_Field(FEM_DOUBLE,3, foffsetof(nodeForce(1),nodeForce(1)), foffsetof(nodeForce(1),nodeForce(4))

58

Create_field Example

! 3D force is contained as fXYZ variable! in a user-defined type node_type

TYPE(node_type), ALLOCATABLE, DIMENTION(:) :: nodes INTEGER :: fid

...allocate nodes array as n_nodes...

fid = FEM_Create_Field(FEM_DOUBLE,3,          foffsetof(nodes(1), nodes(1)%fXYZ),          foffsetof(nodes(1), nodes(2)) )

59

Advanced: FEM on MPI

60

FEM on MPI

FEM routines perform their communication using MPI calls

This means you can call FEM routines from an MPI program; or call MPI routines from an FEM program

61

Chunk Data Example

PROGRAM main include ‘femf.h’ include ‘mpif.h’ CALL MPI_Init() CALL FEM_Init(MPI_COMM_WORLD) ... MPI or FEM routines ...END PROGRAM

62

Extra FEM Features

63

Collision Detection

64

Charm++ Collision Detection Detect collisions (intersections) between objects

scattered across processors

Built on Charm++ Arrays Overlay regular 3D sparse grid of voxels (boxes) Send objects to all voxels they touch Collect collisions from each voxel

Collision response is left to caller

65

Mesh Adaptation

66

Parallel Mesh Refinement 2D refinement 2D coarsening 3D refinement Integration with FEM

67

Triangular Mesh Refinement To refine, split the longest edge:

But if split neighbor has a longer edge, split his edge first

Refinement propagates across mesh, but preserves mesh quality

Initial 2D parallel implementation built on Charm++ Integrated with FEM Framework

FEM_Refine2D_Newmesh FEM_Refine2D_Split

• Input: Nodes, coordinates, elements and desired areas for elements

• Optional: Boundary values for edges• Effect: Updates FEM mesh, interpolates user

data and sets boundary values for new nodes

68

TMR : Results

Initial Mesh: coarse

Refined Mesh: later in the simulation

69

2D coarsening Collapse the smallest edge of an element Makes the adjacent elements larger Being integrated into FEM

70

Parallel 3D refinement Longest Edge based refinement Longest Face based refinement Internal refinement

71

NetFEM

72

NetFEM Client: pretty pictures

Wave dispersion off a crack (simplified frac3d)

73

NetFEM: Easy Visualization Can interact with a running FEM

application Uses Charm++ CCS (Client-server) library

Easy to “publish” attributes

NetFEM n=NetFEM_Begin(2,t,NetFEM_POINTAT);

NetFEM_Nodes(n,nnodes,(double *)g.coord,"Position (m)"); NetFEM_Vector(n,(double *)g.d,"Displacement (m)"); NetFEM_Vector(n,(double *)g.v,"Velocity (m/s)");

NetFEM_Elements(n,nelems,3,(int *)g.conn,"Triangles"); NetFEM_Scalar(n,g.S11,1,"X Stress (pure)"); NetFEM_Scalar(n,g.S22,1,"Y Stress (pure)"); NetFEM_Scalar(n,g.S12,1,"Shear Stress (pure)");

NetFEM_End(n);

Author:

Orion Lawlor

74

NetFEM: Easy visualization

75

NetFEM: Zoom in

76

NetFEM: Outline Elements

77

NetFEM: Point Nodes

78

NetFEM Server Side: Overview To allow the NetFEM client to connect,

you add NetFEM registration calls to your server Register nodes and element types Register data items: scalars or spatial

vectors associated with each node or element

You provide the display name and units for each data item

Link your program with “-module netfem”

Run with “++server”, and connect!

79

NetFEM Server Side: Setup n=NetFEM_Begin(FEM_My_partition(),ti

mestep, dim,NetFEM_POINTAT) Call this each time through your

timeloop; or skip timestep identifies this data update dim is the spatial dimension—must be

2 or 3 Returns a NetFEM handle n used by

everything else NetFEM_End(n)

Finishes update n

80

NetFEM Server Side: Nodes NetFEM_Nodes(n,nnodes,coord,”Posi

tion (m)”) Registers node locations with NetFEM—future

vectors and scalars will be associated with nodes n is the handle returned by NetFEM_Begin nnodes is the number of nodes coord is a dim by nnodes array of doubles The string describes the coordinate system and

meaning of nodes

Currently, there can only be one call to nodes

81

NetFEM: Node Displacement

82

NetFEM Server Side: Elements NetFEM_Elements(n,nelem,nodeper,

conn,”Triangles”) Registers elements with NetFEM—future vectors

and scalars will be associated with these elements

n is the handle returned by NetFEM_Begin nelem is the number of elements nodeper is the number of nodes per element conn is a nodeper by nelem array of node indices The string describes the kind of element

Repeat to register several kinds of element Perhaps: Triangles, squares, pentagons, …

83

NetFEM: Element Stress

84

NetFEM Server Side: Vectors NetFEM_Vector(n,val,”Displacement (m)”)

Registers a spatial vector with each node or element

• Whichever kind was registered last n is the handle returned by NetFEM_Begin val is a dim by nitems array of doubles

• There’s also a more general NetFEM_Vector_field in the manual

The string describes the meaning and units of the vectors

Repeat to register multiple sets of vectors Perhaps: Displacement, velocity,

acceleration, rotation, …

85

NetFEM: Element Velocity

86

NetFEM Server Side: Scalars NetFEM_Scalar(n,val,s,”Displacement (m)”)

Registers s scalars with each node or element• Whichever kind was registered last

n is the handle returned by NetFEM_Begin val is an s by nitems array of doubles

• There’s also a more general NetFEM_Scalar_field in the manual

s is the number of doubles for each node or element

The string describes the meaning and units of the scalars

Repeat to register multiple sets of scalars Perhaps: Stress, plasticity, node type,

damage, …

87

NetFEM Server Side: 2D Exampleinteger :: t,n, numnp, numel

real*8, dimension(2,numnp) :: coor,d,v,a

integer, dimension(3,numel) :: conn

n=NetFEM_Begin(FEM_My_partition(),t,2,NetFEM_POINTAT)

CALL NetFEM_Nodes(n,numnp,coor,'Position (m)')

CALL NetFEM_Vector(n,d,'Displacement (m)')

CALL NetFEM_Vector(n,v,'Velocity (m/s)')

CALL NetFEM_Vector(n,a,'Acceleration (m/s^2)')

CALL NetFEM_Elements(n,numel,3,conn,'Triangles')

CALL NetFEM_Scalar(n,stress,1,'Stress (pure)')

CALL NetFEM_End(n)

88

NetFEM: Conclusion Easy, general way to get output from an

FEM computation Client configures itself based on server Client can be run anywhere (from

home!) Server performance impact minimal

(1s!) Future work:

Support multiple chunks per processor Movie mode

89

Conclusions: Charm++ FEM

Easy to parallelize existing codes

Flexible: shared nodes or ghosts High performance Extra features

Visualization with NetFEM Collision Matrix methods with IFEM

98

Additional features IFEM: Iterative FEM Linear Solver

Interface FEM supports Symmetries Provides support for data

transfer between meshes