PDE Discretization and Analysis with libMeshscisoftdays.org/pdf/2016_slides/stogner.pdf · Git...
Transcript of PDE Discretization and Analysis with libMeshscisoftdays.org/pdf/2016_slides/stogner.pdf · Git...
PECOSPredictive Engineering and Computational Sciences
PDE Discretization and Analysis with libMesh
Roy H. Stogner1
1The University of Texas at Austin
Feb 26, 2016
Roy H. Stogner libMesh Feb 26, 2016 1 / 22
Introduction
libMesh Finite Element Library
Scope
• Free, Open source
LGPL2 for core
• 35 Ph.D. theses, 393
papers (58 in 2015)
• ∼ 10 current developers
• 160− 240 current users?
Jan 2010 Jul 2011 Oct 2013 Feb 201650
100
150
200
250
300libmesh-devellibmesh-users
LibMesh Mailing List Membership Size
Challenges
• Radically different application types
• Widely dispersed core developers
INL, UT-Austin, U.Buffalo, JSC, MIT,
Harvard, Argonne
• OSS, commercial, private applications
Roy H. Stogner libMesh Feb 26, 2016 2 / 22
Introduction
PetscMetis
Mpich
Laspack
LibMesh
STL
RBM
TumorAngiogenesis
CompressibleNS
DD
• Foundational
(typically optional)
library access via
LibMesh’s “roots”.
• Application
“branches” built off
the library “trunk”.
• Additional
middleware layers
(e.g. Akselos,
GRINS, MOOSE) for
more complex
applications
Roy H. Stogner libMesh Feb 26, 2016 3 / 22
Introduction
Typical Boundary Value Problem
• Common BVP components:
M∂u
∂t= F (u) ∈ Ω ⊂ R
n
G(u) = 0 ∈ Ω
u = uD ∈ ∂ΩD
N(u) = 0 ∈ ∂ΩN
u(x, 0) = u0(x)
• Less common components:
Moving domain Ω(t), Ω(u, t) Multi-dimensional manifolds Self-overlapping, contact Acceleration ∂2u/∂t2
Integro-differential equations
Ω
∂ΩD
∂ΩN
Roy H. Stogner libMesh Feb 26, 2016 4 / 22
Introduction
Mesh Classes
MeshBase
#_dim: unsigned int
#_n_parts: unsigned int
+boundary_info: AutoPtr<BoundaryInfo>
+elements_begin()
+read()
+<<empty>> allgather()
+<<empty>> delete_remote_elements()
Paral lelMesh
+_elements: mapvector<Elem*>
+elements_begin()
+allgather()
+delete_remote_elements()
UnstructuredMesh
+read()
SerialMesh
_elements: std::vector<Elem*>
+elements_begin()
• MeshBase gives node or
element iterators, all or
active, global or local
• SerialMesh or
ParallelMesh manages
synchronized or
distributed data
Roy H. Stogner libMesh Feb 26, 2016 5 / 22
Introduction
Geometric Element Classes
Elem
#_nodes: Node **
#_neighbors: Elem **
#_parent: Elem *
#_children: Elem **
#_*flag: RefinementState
#_p_level: unsigned char
#_subdomain_id: unsigned char
+n_faces,sides,vertices,edges,children(): unsigned int
+centroid(): Point
+hmin,hmax(): Real
NodeElem FaceEdge Cell InfQuad InfCell
Prism Hex Pyramid Tet
Hex8 Hex20 Hex27
DofObject
-_n_systems: unsigned char
-_n_vars: unsigned char *
-_n_comp: unsigned char **
-_dof_ids: unsigned int **
-_id: unsigned int
-_processor_id: unsigned short int
Node
• Abstract interface gives
mesh topology
• Concrete instantiations of
mesh geometry
• Hides element type from
most applications
• Base class data arrays allow
more optimization, inlining
Similar trees:
• Shape functions, quadrature
• Periodic boundary maps
• Reference counts, comms
• Linear algebra, solvers
• Integrators, strategies
Roy H. Stogner libMesh Feb 26, 2016 6 / 22
Introduction
Poisson Example
• Consider the weak form arising from the Poisson equation,
R(u, v) :=∫
Ωh
[∇u · ∇v − fv] dx = 0 ∀v ∈ V
Roy H. Stogner libMesh Feb 26, 2016 7 / 22
Introduction
Poisson Example
• The LibMesh representation of the matrix and rhs assembly is similar
to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
Roy H. Stogner libMesh Feb 26, 2016 8 / 22
Introduction
Poisson Example
• The LibMesh representation of the matrix and rhs assembly is similar
to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
F ei =
Nq∑
q=1
f(x(ξq))φi(ξq)|J(ξq)|wq
Roy H. Stogner libMesh Feb 26, 2016 8 / 22
Introduction
Poisson Example
• The LibMesh representation of the matrix and rhs assembly is similar
to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i)
Fe(i) += JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
Keij =
Nq∑
q=1
∇φj(ξq) · ∇φi(ξq)|J(ξq)|wq
Roy H. Stogner libMesh Feb 26, 2016 8 / 22
Introduction
Scalability: Hybrid MPI + Threads
Parallel:: interfaces wrap MPI
Library data is distributed, synchronized:
• Simpler, STL-compatible, inlined API
• Works with complex serializable classes
Threads:: interfaces TBB/pthreads/OpenMP
• Sparsity, AMR constraint calculation
• FEMSystem assembly routines
• Asynchronous I/O
• Mesh projections, utility functions
Number of Processor Cores
Spe
edup
100 101 102 103100
101
102
103
IdealScaled-Size (Weak) ScalingFixed-Size (Strong) Scaling
0 10 20 30 40 50 60
# of Threads
0
10
20
30
40
50
60
Speedup
Facto
r
MIC Speedup
2D, no mutex
2D, mutex
3D, no mutex
3D, mutex
Ideal
• Hybrid parallel apps with no threading or message passing code
• API-independence via libMesh wrappers
Roy H. Stogner libMesh Feb 26, 2016 9 / 22
Adaptivity
Adaptivity: Error Estimators, Error Indicators
Error decompositions
Subterms on each element K:
• ||u− uh||2H=
∑
K ||u− uh||2H(K) ≤
∑
K |ηK |2
• Q(u)−Q(uh) ≈∑
K ηK
• |Q(u)−Q(uh)| ≤∑
K |ηK |
Refinement heuristics
• Refinement/coarsening of elements with:
Worst/best fraction sorted by error Error over/under fraction of tolerance Error over/under target mesh size average
• h-vs-p refinement:
a priori singularity identification Behavior vs. cost when h vs. p coarsening?
Roy H. Stogner libMesh Feb 26, 2016 10 / 22
Adaptivity
Goal-oriented Adaptivity
Refine to reduce solution error when
it influences QoI error:
2 2.5 3 3.5 4 4.5−6.5
−6
−5.5
−5
−4.5
−4
−3.5
−3
−2.5
Log10
(Dofs), Degrees of Freedom
Lo
g1
0(A
bso
lute
Err
or)
Convergence of the interior QoI, ε = 10−8
Uniform Refinements
Flux Jump Error Estimator
Adjoint Error Estimator
• Global error indicator targets
layer alone; QoI temporarily
plateaus
• Rapid convergence from
adjoint-based refinement
Roy H. Stogner libMesh Feb 26, 2016 11 / 22
Adaptivity
Adjoint-based Error Indicators, Sensitivities
Q(uh)−Q(u) = R(uh, z − zh) +H.O.T.
q′ = Qξ(u; ξ)−Rξ(u, z −Ψ(u; ξ); ξ)
Adjoint Refinement
R(uh, z − zh) ≈∑
E
RE(uh, zH − zh)
Adjoint Residual∣
∣
∣R(uh, z − zh)
∣
∣
∣≤
∑
E
∣
∣
∣
∣REu
∣
∣
∣
∣
B(UE ,VE∗)
∣
∣
∣
∣
∣
∣u− uh
∣
∣
∣
∣
∣
∣
UE
∣
∣
∣
∣
∣
∣z − zh
∣
∣
∣
∣
∣
∣
VE
Generalized Adjoint Residual∣
∣
∣R(uh, z − zh)
∣
∣
∣≤
∑
E
∣
∣
∣
∣
∣
∣zi − zh
i
∣
∣
∣
∣
∣
∣
iMij
∣
∣
∣
∣
∣
∣uj − uh
j
∣
∣
∣
∣
∣
∣
j
Roy H. Stogner libMesh Feb 26, 2016 12 / 22
Collaboration Strategies
Collaboration Strategies
Communication
• Face to face, instant messaging, teleconference
• Email lists
• Trac/Redmine/Sourceforge issue tracking
• GitHub issues
Code
• Email attachments
• Ticket attachments
• Repository forks!
• Pull requests!
Roy H. Stogner libMesh Feb 26, 2016 13 / 22
Source Code Control
Git Guidelines
• Strive for “useful nonlinearity.”
• Develop separate feature sets on
separate branches; merge them
back to master when complete.
• Minimize or eliminate periodic or
unnecessary merge commits.
• rebase feature branches on top of
master before merge + push
• Rebasing public branches is badTM.
Complete the shared branch,
branch from the shared branch
locally, rebase from target, then
merge
• http://nvie.com/posts/a-successful-git-branching-model
Roy H. Stogner libMesh Feb 26, 2016 14 / 22
Development and Testing
Issue, Pull Request Tracking
Roy H. Stogner libMesh Feb 26, 2016 15 / 22
Development and Testing
Tracking API Changes
API versions easily proliferate...
#if PETSC_VERSION_LESS_THAN(3,1,0)
ierr = MatGetSubMatrix(matrix->mat(),
_restrict_to_is, _restrict_to_is_complement,
PETSC_DECIDE, MAT_INITIAL_MATRIX, &submat1);
#else
ierr = MatGetSubMatrix(matrix->mat(),
_restrict_to_is, _restrict_to_is_complement,
MAT_INITIAL_MATRIX, &submat1);
#endif
• Maintain a wide range of external compatibility
Dropped PETSc 2.3.3 (2007) support for libMesh 1.0 (2016) C++11 shims
• Limit libMesh API changes
Roy H. Stogner libMesh Feb 26, 2016 16 / 22
Development and Testing
Signaling API Changes
Development practices
• Old, new APIs overlap
• Easier with C++ function overloading, default arguments
Adding f(a,b) does not preclude keeping f(a) Adding f(a,b=default) can replace f(a)
Runtime warnings
• libmesh experimental() (in-flux APIs)
• libmesh deprecated() (∼1 year, 1-2 releases)
Examples
• OStringStream workaround class
• Parallel:: global functions
Roy H. Stogner libMesh Feb 26, 2016 17 / 22
Development and Testing
Regression Testing
• ∼ 7000 internal assertions in debug mode
• ∼ 60 core example applications
• ∼ 400 unit tests
• Configurations: optional features, index width,
ParallelMesh, solver package, scalar precision,
-np, –n threads
• Middleware, application test suites
Roy H. Stogner libMesh Feb 26, 2016 18 / 22
Development and Testing
Assertions
libmesh_assert(c < _variables.size());
libmesh_assert(s < elem->n_sides());
libmesh_assert((ig >= Ug.first_local_index()) &&
(ig < Ug.last_local_index()));
libmesh_assert(requested_ids[p].size() == ghost_objects_from_proc[p]);
libmesh_assert(obj_procid != DofObject::invalid_processor_id);
libmesh_assert(mesh.is_prepared());
MeshTools::libmesh_assert_valid_node_procids(mesh);
libmesh_assert(neigh->has_children());
libmesh_assert(error_estimator.error_norm.type(var) == H1_SEMINORM ||
error_estimator.error_norm.type(var) == W1_INF_SEMINORM)
libmesh_assert(number_h_refinements > 0 || number_p_refinements > 0);
Roy H. Stogner libMesh Feb 26, 2016 19 / 22
Development and Testing
Assertions
libmesh_assert(c < _variables.size());
libmesh_assert(s < elem->n_sides());
libmesh_assert((ig >= Ug.first_local_index()) &&
(ig < Ug.last_local_index()));
libmesh_assert(requested_ids[p].size() == ghost_objects_from_proc[p]);
libmesh_assert(obj_procid != DofObject::invalid_processor_id);
libmesh_assert(mesh.is_prepared());
MeshTools::libmesh_assert_valid_node_procids(mesh);
libmesh_assert(neigh->has_children());
libmesh_assert(error_estimator.error_norm.type(var) == H1_SEMINORM ||
error_estimator.error_norm.type(var) == W1_INF_SEMINORM)
libmesh_assert(number_h_refinements > 0 || number_p_refinements > 0);
• Trust preconditions/postconditions, not any of your users
Roy H. Stogner libMesh Feb 26, 2016 19 / 22
Development and Testing
Assertions
libmesh_assert(c < _variables.size());
libmesh_assert(s < elem->n_sides());
libmesh_assert((ig >= Ug.first_local_index()) &&
(ig < Ug.last_local_index()));
libmesh_assert(requested_ids[p].size() == ghost_objects_from_proc[p]);
libmesh_assert(obj_procid != DofObject::invalid_processor_id);
libmesh_assert(mesh.is_prepared());
MeshTools::libmesh_assert_valid_node_procids(mesh);
libmesh_assert(neigh->has_children());
libmesh_assert(error_estimator.error_norm.type(var) == H1_SEMINORM ||
error_estimator.error_norm.type(var) == W1_INF_SEMINORM)
libmesh_assert(number_h_refinements > 0 || number_p_refinements > 0);
• Trust preconditions/postconditions, not any of your users
• Earlier errors are easier errors
Roy H. Stogner libMesh Feb 26, 2016 19 / 22
Development and Testing
Assertions
libmesh_assert(c < _variables.size());
libmesh_assert(s < elem->n_sides());
libmesh_assert((ig >= Ug.first_local_index()) &&
(ig < Ug.last_local_index()));
libmesh_assert(requested_ids[p].size() == ghost_objects_from_proc[p]);
libmesh_assert(obj_procid != DofObject::invalid_processor_id);
libmesh_assert(mesh.is_prepared());
MeshTools::libmesh_assert_valid_node_procids(mesh);
libmesh_assert(neigh->has_children());
libmesh_assert(error_estimator.error_norm.type(var) == H1_SEMINORM ||
error_estimator.error_norm.type(var) == W1_INF_SEMINORM)
libmesh_assert(number_h_refinements > 0 || number_p_refinements > 0);
• Trust preconditions/postconditions, not any of your users
• Earlier errors are easier errors
• Exceptions! Stack traces! Line numbers!
Roy H. Stogner libMesh Feb 26, 2016 19 / 22
Development and Testing
Manufactured Solution Testing
(Finding unknown unknowns)
Verification of FIN-S hypersonics
code
• FANS, Spalart-Allmaras
• Derivative:
d(sa)
dx=
1
ρ∗
(
d(ρ ∗ sa)
dx− sa
dρ
dx
)
• In code:
d(sa)
dx=
1
ρ∗d(ρ ∗ sa)
dx− sa
dρ
dx10-5
10-4
10-3
10-2
4 5 6 7 8
||rn
- rn
h|| 2
Nref
Before1567
1567 & 1568O(h2)
Manufactured Analytical Solution Abstraction library:
https://manufactured-solutions.github.io/MASA/
Roy H. Stogner libMesh Feb 26, 2016 20 / 22
Build Systems
Autotools, Pros and Cons
Autoconf
• Manages feature selection
50+ --enable-foo options
• Portability tests, workarounds
• POSIX shell dependence
Libtool
• DLL management in install
• Broader shared library support
• Easily used via automake
• Trickier in-build debugging
Automake
• dist, check, install
targets
• Out-of-source builds
• Standardized
conventions
• More difficult METHODS
support
• “bootstrap” process
Do users have
autotools? Custom scripts for
libMesh
Roy H. Stogner libMesh Feb 26, 2016 21 / 22
Build Systems
AcknowledgementsRecent contributors:
• David Andrs
• Paul Bauman
• Vikram Garg
• Derek Gaston
• Dmitry Karpeev
• Benjamin Kirk
• David Knezevic
• Cody Permann
• John Peterson
• Sylvain Vallaghe
Useful resources:
• libMesh: https://libmesh.github.io/
• GRINS: https://grinsfem.github.io/
• MOOSE: https://mooseframework.org/
• MASA: https://manufactured-solutions.github.io/MASA/
Questions?
Roy H. Stogner libMesh Feb 26, 2016 22 / 22