Scalable Mesh Generation for HPC...
1
Conclusion Scalable Mesh Generation for HPC Applications Rajeev Jain, Navamita Ray, Iulian Grindeanu, Danqing Wu, Vijay Mahadevan Acknowledgements References • Computational solvers simulating physical phenomenon on complex domain geometries need well resolved, high-quality meshes to tackle the discrete problem efficiently. • Mesh generation for HPC applications is a complex process requiring access to geometry data and efficient mesh data-structures in a parallel setting as well as optimization techniques for quality preservation. • A key component of any such parallel mesh infrastructure is the parallel I/O file system whose performance is vital to any complex application workflow. • MeshKit, developed as a component of SIGMA tool chain, supports variety of meshing algorithms leveraging the scalable interfaces in SIGMA to geometry data (CGM) and unified parallel data-structures (MOAB) that can be used for HPC applications. Scalable Interfaces for Geometric and Mesh based Applications(SIGMA) An open-source toolchain to simplify computational modeling workflow[4] Common Geometry Module(CGM) Libraries for querying and modifying solid geometry model[1] Problem Description Mesh Generation Toolkit(MeshKit) Flexible algorithms to generate high-quality meshes[3] Mesh Oriented datABase(MOAB) Efficient array-based data structures to handle mesh queries[2] • Provides a collection of meshing algorithms to support mesh generation; a platform to develop mesh generation algorithms. • Coordination of BREP-based meshing process, mesh smoothing, optimization • Notable meshing algorithms include embedded boundary meshing, watertight models, mesh based geometry generator • Exposes mesh manipulation and generation features such as Copy, Move, Rotate and Extrude Mesh. • Uses SIGMA tools CGM and MOAB for accessing geometry model and mesh representation in a parallel setting. • Common API and topological model to access geometry representations for a variety of solid modeling engines such as ACIS, Open-CASCADE, Facet-based surfaces, etc. • Provides parallel geometry model querying and handling; virtual geometry, non-manifold topology. • Represent unstructured and structured mesh and field data on a mesh efficiently. • Parallel mesh capabilities: - IO(parallel HDF5 library based) - link to existing state of the art mesh partitioners(ParMetis, Zoltan) - algorithms for resolving entities on shared processor interfaces(geometric proximity based, global id based) - exchange ghost layers, mesh migration, field data exchange • Hierarchical mesh generation for unstructured meshes through uniform refinement and quality metrics. • Multimesh intersection and transfer algorithms. • Automatic mesh generation describing complex nuclear reactor assemblies and core geometries(templates) using MeshKit • Reactor Geometry Generator a.k.a., RGG GUI, developed in collaboration with Kitware Inc. MeshKit for Nuclear Engineering MOAB Weak Scaling Studies • A mini-app GenLargeMesh for generating large meshes to explore the capability of MOAB’s parallel infrastructure and of the parallel IO file system. • It creates 3D hexahedral meshes for a rectangular domain in-memory and writes out the mesh to a file in parallel. • The partitioned mesh is generated on each task locally, and all tasks write to the same file. Approximately 21.6K hexes per task; largest mesh sizes: 221M (Blues) and 1.8B (Vesta). • Weak scalability studies on Blues for assembly generation along with two levels of degree 2 uniform refinement. Each core 7500 hexes; 60K and 480K after 2 refinement levels; largest mesh size 1.1B hexes. • Entities on the shared processor interface are resolved using a geometric proximity based vertex-merge algorithm for both core assemblies and after refinement; maintains scalability upto 1K cores. Machine Details: Blues: 310 nodes, 16 cores/node (Intel Sandy Bridge), 64 GB of RAM per node Vesta: 2,048 nodes, 16 cores/node(1600 MHz PowerPC A2), 16 GB RAM per node • Parallel I/O (based on hdf5 library) reads from or writes to a single file which involves indirect referencing to access entities on each partition. This effects the weak scalability of the I/O negatively as seen below. • Shared entities at the interfaces between partitions is resolved using vertex global ids and crystal router, an efficient gather-scatter algorithm for sparse communication. The interface resolution is highly scalable and maintains efficiency to thousands of processors. [1] T. J. Tautges, R. Meyers, K. Merkley, C. Stimpson, and C. Ernst, “MOAB: A Mesh- Oriented Database,” Sandia National Laboratories, SAND2004-1592, Apr. 2004. [2] T. J. Tautges, CGM: A geometry interface for mesh generation, analysis and other applications, Engineering with Computers, 17 (2001), pp. 299–314. [3] Rajeev Jain, T.J. Tautges, “Generating Unstructured Nuclear Reactor Core Meshes in Parallel”, In Proceedings of 23rd International Meshing Roundtable, Oct 2014. [4] http://sigma.mcs.anl.gov/ • SIGMA tools provide necessary components and interfaces for developing advanced mesh generation capabilities. • The parallel infrastructure in MOAB for resolving shared entities using geometrical proximity and global ids for vertices are highly scalable onto thousands of processors. • The parallel IO deteriorates in performance and needs further investigation and optimization. • Parallel write deteriorates significantly around 1K cores. This work was supported by UChicago Argonne, LLC, a U.S. Department of Energy Office of Science laboratory, operated under Contract No. DE-AC02-06CH11357.
Transcript of Scalable Mesh Generation for HPC...
Conclusion
Scalable Mesh Generation for HPC Applications Rajeev Jain, Navamita Ray, Iulian Grindeanu, Danqing Wu, Vijay Mahadevan
Acknowledgements
References
• Computational solvers simulating physical phenomenon on complex domain geometries need well resolved, high-quality meshes to tackle the discrete problem efficiently. !• Mesh generation for HPC applications is a complex process requiring access to geometry data and efficient mesh data-structures in a parallel setting as well as optimization techniques for quality preservation. !• A key component of any such parallel mesh infrastructure is the parallel I/O file system whose performance is vital to any complex application workflow. !• MeshKit, developed as a component of SIGMA tool chain, supports variety of meshing algorithms leveraging the scalable interfaces in SIGMA to geometry data (CGM) and unified parallel data-structures (MOAB) that can be used for HPC applications.
Scalable Interfaces for Geometric and Mesh based Applications(SIGMA)
An open-source toolchain to simplify computational modeling workflow[4]
Common Geometry Module(CGM) Libraries for querying and modifying solid geometry model[1]
Problem Description Mesh Generation Toolkit(MeshKit) Flexible algorithms to generate high-quality meshes[3]
Mesh Oriented datABase(MOAB) Efficient array-based data structures to handle mesh queries[2]
• Provides a collection of meshing algorithms to support mesh generation; a platform to develop mesh generation algorithms.
• Coordination of BREP-based meshing process, mesh smoothing, optimization
• Notable meshing algorithms include embedded boundary meshing, watertight models, mesh based geometry generator
• Exposes mesh manipulation and generation features such as Copy, Move, Rotate and Extrude Mesh. !!!• Uses SIGMA tools CGM and MOAB for accessing geometry model and mesh representation in a parallel setting. !
• Common API and topological model to access geometry representations for a variety of solid modeling engines such as ACIS, Open-CASCADE, Facet-based surfaces, etc.
• Provides parallel geometry model querying and handling; virtual geometry, non-manifold topology.
• Represent unstructured and structured mesh and field data on a mesh efficiently. • Parallel mesh capabilities:
- IO(parallel HDF5 library based) - link to existing state of the art mesh partitioners(ParMetis, Zoltan) - algorithms for resolving entities on shared processor
interfaces(geometric proximity based, global id based) - exchange ghost layers, mesh migration, field data exchange
• Hierarchical mesh generation for unstructured meshes through uniform refinement and quality metrics.
• Multimesh intersection and transfer algorithms.
• Automatic mesh generation describing complex nuclear reactor assemblies and core geometries(templates) using MeshKit
• Reactor Geometry Generator a.k.a., RGG GUI, developed in collaboration with Kitware Inc. !!
MeshKit for Nuclear Engineering
MOAB Weak Scaling Studies • A mini-app GenLargeMesh for generating large meshes to
explore the capability of MOAB’s parallel infrastructure and of the parallel IO file system.
• It creates 3D hexahedral meshes for a rectangular domain in-memory and writes out the mesh to a file in parallel.
• The partitioned mesh is generated on each task locally, and all tasks write to the same file. Approximately 21.6K hexes per task; largest mesh sizes: 221M (Blues) and 1.8B (Vesta).
• Weak scalability studies on Blues for assembly generation along with two levels of degree 2 uniform refinement. Each core 7500 hexes; 60K and 480K after 2 refinement levels; largest mesh size 1.1B hexes.
• Entities on the shared processor interface are resolved using a geometric proximity based vertex-merge algorithm for both core assemblies and after refinement; maintains scalability upto 1K cores.
Machine Details: !Blues: 310 nodes, 16 cores/node (Intel Sandy Bridge), 64 GB of RAM per node !!!Vesta: 2,048 nodes, 16 cores/node(1600 MHz PowerPC A2), 16 GB RAM per node
• Parallel I/O (based on hdf5 library) reads from or writes to a single file which involves indirect referencing to access entities on each partition. This effects the weak scalability of the I/O negatively as seen below.
• Shared entities at the interfaces between partitions is resolved using vertex global ids and crystal router, an efficient gather-scatter algorithm for sparse communication. The interface resolution is highly scalable and maintains efficiency to thousands of processors.
[1] T. J. Tautges, R. Meyers, K. Merkley, C. Stimpson, and C. Ernst, “MOAB: A Mesh-Oriented Database,” Sandia National Laboratories, SAND2004-1592, Apr. 2004. [2] T. J. Tautges, CGM: A geometry interface for mesh generation, analysis and other applications, Engineering with Computers, 17 (2001), pp. 299–314. [3] Rajeev Jain, T.J. Tautges, “Generating Unstructured Nuclear Reactor Core Meshes in Parallel”, In Proceedings of 23rd International Meshing Roundtable, Oct 2014. ![4] http://sigma.mcs.anl.gov/
• SIGMA tools provide necessary components and interfaces for developing advanced mesh generation capabilities.
• The parallel infrastructure in MOAB for resolving shared entities using geometrical proximity and global ids for vertices are highly scalable onto thousands of processors.
• The parallel IO deteriorates in performance and needs further investigation and optimization.
SIGMA&Components/Enabling(strong'application(support(through(loosely(connected(software5
Vijay(Mahadevan5(((((((Navamita(Ray5Iulian(Grindeanu(((((((((Danqing(Wu5Rajeev(Jain5((((((((((((((((((((Evan(Vanderzee55
Paul(Wilson5Patrick(Shriwise5Andy(Davis5
SIGMA:&Simplifying&traditional&computational&modeling&workflow/An(openIsource(simulation(toolchain5
! Computational(solvers(simulating(physical(phenomena(on(complex'domain'geometries(need(well(resolved,(high'quality'meshes(to(tackle(the(discrete(problem(efficiently.5
! Mesh(generation(is(a(complex(problem;(SIGMA(tools(simplify(the(process.5
! SIGMA(provides(interfaces'and'components(to(access(geometry(data,(unified(dataIstructures(to(load(and(manipulate(parallel(unstructured(computational(meshes(for(various(applications.5
! Leverage(demonstrated(scalability(of(SIGMA(tools(on(petascale(systems((research(for(exascale).5
Unstructured&Mesh&Oriented&Database&(MOAB)/Efficient(arrayIbased(datastructures(to(handle(mesh(queries(in(memory5
Define(Geometry5
Generate(discrete(mesh5
Solve(nonlinear(
PDE(systems5Serialize(and(Checkpoint5
Visualize(and(PostIprocess5
! CGM(–(Common(Geometry(Module5! MOAB(–(Unstructured'Mesh(Oriented(datABase5! Lasso&–(Relation(between(geometry(and(mesh(representations5
! Meshkit(–(Library(of(advanced(mesh;generation(algorithms5
! CouPE(–(Coupled(multi;Physics(Environment5! PySIGMA(–(Python(interfaces(to(SIGMA(tools5! DMMoab(–(MeshISolver(interfaces(in(PETSc5! Notable&applications:(RGG,(Nek5000,(PROTEUS,(Diablo,(MBCSLAM,(MoFEM,(SpaFEDTe5
" API(provided(for(querying(faceted'geometry(models5" Field(descriptors(and(scalable(solver'hooks((PETSc)5
# MOAB(handles(unstructured(mesh(natively(while(PETSc(DM(interfaces(provide(DoF(mapping,(operator(assembly5
# Utilize(uniform(refinement(to(drive(geometry(multigrid((KSP(or(PC)5" Efficient(discretization'kernels(for(P(1,2)/Q(1,2)(elements5" Ongoing(research:(understanding(portable(performance(of(unstructured(meshing(and(handling(algorithms5
hbp://sigma/mcs.anl.gov5
Geometry&and&Solver&aware&tools/Exposure(to(geometry,(mesh(and(discretization5
Common&Geometry&Module&(CGM)/Libraries(for(querying(and(modifying(solid(geometry(models5
$ Implements(the(ITAPS(iGeom(interface(completely5$ Provides(geometry(infrastructure(for(the(CUBIT(mesh(generation(toolkit5$ Parallel(geometry'model'querying(and(handling5
! Access(geometry(representations(for(a(variety(of(solid(modeling(engines5% ACIS(–(Geometry(backbone(for(CUBIT((upto(v14.0)(5% Open;CASCADE((OCC/OCE)(–(Supported(natively5% Mesh(based(representation((facetIbased(surfaces)5% Other(BREP(based(CAD(models5
$ Non;manifold(topology(representation(and(detection5$ Support(ray'tracing,'surface'crossing(queries(for(MonteICarlo((MCNP)5
Faceted&geometry&(ITER)/
Common(API(and(topological(model(5
$ Implements(the(ITAPS(iMesh(and(iMeshP'interfaces5$ Represent(unstructured(and(structured(mesh,(and(field(data(on(a(mesh(
efficiently5
% Represent(most(kinds(of(metadata(often(accompanying(the(mesh((e.g.(material(data,(boundary(conditions,(processor(partitions,(geometric(topology,(solution(data)5
$ Scalable(parallel(mesh(capabilities((verified(upto(512K(processors)5$ Robust(point(location(and(interpolation(in(parallel5$ Consistent(solution(transfer((C0/P1/spectral(basis)(between(computational(
meshes(to(support(accurate(multiIphysics(simulations5$ Uniform(unstructured(mesh(refinement(and(quality(metrics5$ Future(extensions(to(support(AMR(and(surface'reconstructionsL
Data(model(is(simple(yet(powerful5
5e−0
51e−0
42e−0
45e−0
41e−0
32e−0
35e−0
3
Tet −> Hex RMS Element Coupling Error
Number of Processes
RM
S Er
ror i
n El
emen
t Val
ues
32 256 2048 16384 131072
1 tet : 1000 hex1 tet : 100 hex1 tet : 10 hex1 tet : 1 hex
MeshNgeneration&toolkit&(MeshKit)/Flexible(algorithms(to(generate(highIquality(meshes5
Accurate5
Reactor&Geometry&Generator&(RGG)/
100
200
300
400
500
600
Strong Scaling
Number of Processes
Coup
ling
Tim
e (s
)
32768 65536 131072 262144 524288
1024^3 gridPerfect scaling Nek5000(CFD(Spectral(element(code5PROTEUS(Neutron(transport(code5
Scalable(and(efficient5
$ Provides(a(collection(of(meshing(algorithms(to(support(mesh(generation((coordination(of(BREPIbased(meshing(process,(mesh(smoothing,(optimization)5
$ Exposes(mesh(manipulation(and(generation(features(such(as(Copy,(Move,(Rotate(and(Extrude(mesh.5
$ Notable(meshing(algorithms(include(embedded(boundary(meshing,(watertight(models,(mesh(based(geometry(generator5
$ Automatic(computational(mesh(generation(describing(complex(nuclear(reactor(assembly(and(core(geometries((templates).5
$ The(RGG(GUI(is(being(developed(in(collaboration(with(Kitware;(Motivated(to(improve(scientific(research(productivity((simplify(reactor(mesh(generation)5
Diablo(thermoImechanics(code5MoFEM(hIp(adaptive(FEA(code5
• Parallel write deteriorates significantly around 1K cores.
This work was supported by UChicago Argonne, LLC, a U.S. Department of Energy Office of Science laboratory, operated under Contract No. DE-AC02-06CH11357.
