HOPR - High Order Preprocessor

Florian Hindenlang1,

Thomas Bolemann2, Malte Hoffmann2, Claus-Dieter Munz2

1Max-Planck Institute for Plasma Physics, Garching2Institute for Aero- and Gasdynamics, University of Stuttgart

High-Order Computational Fluid Dynamics Technologies WebinarSeptember 21, 2015

1 / 22

Curved Meshes: HOPR

I DG is high order accurate on unstructured grids

I Curved elements to maintain accuracy at curved wall boundaries

I DG for industrial applications:I Wall-bounded flowsI 3D curved geometries, intersecting surfaces, sharp cornersI Unstructured and coarse grids

⇒ HOPR (High Order Preprocessor)released Open-Source: www.hopr-project.org

2 / 22

Facts on HOPR

I Open-source under GPL v2: www.hopr-project.org

I Mesh preprocessor for parallel high order simulations(DG, FR, SEM, pFEM...)

I Written in FORTRAN

I Generate simple meshes for testing, directly in HOPR

I HOPR Input:I Unstructured hybrid meshes from any grid generatorI Several formats supported (CGNS mostly used)

⇒ Different strategies to produce high order meshes from the initiallinear mesh

I HOPR Output: HDF5 mesh file

3 / 22

HOPR output: HDF5 mesh file

I Idea: Direct parallel readinI HOPR sorts elements along a Space-Filling Curve

⇒ 1D element list, can be split into arbitrary number of domainsI Element-per-element datablocks:

high order node coordinates, side info (BC, connectivity)

⇒ Split global arrays into non-overlapping blocks during parallel readin!

HDF5 mesh file

globalattributes

BCNames

BCTypes

ElemInfo SideInfoNodeCoordselem1elem2elem3

elem4elem5elem6

globalattributes

BCNames

BCTypes

globalattributes

BCNames

BCTypes

globalattributes

BCNames

BCTypes

computenLocElemsoffsetElem

computenLocElemsoffsetElem

computenLocElemsoffsetElem

elem1elem2elem3

elem4elem5elem6

elem7elem8

elem7elem8

offset/sizeSidesNodes

offset/SizeSidesNodes

offset/SizeSidesNodes

buildelement

connectivity

buildelement

connectivity

buildelement

connectivity

rank

#0

rank

#1

rank

#2

4 / 22

HOPR Mesh Curving Flowchart

commercial mesher: ICEM, ANSA,... (cgns unstruct. format) linear unstructured hybrid mesh

curved boundary faces

Normal vector(s) at surface nodes

point-normalsICEM

spectral elementssurface

subdivision

ICEM Cheb.-Lobattonodes on edges

curved boundary edges

blend curved edges to face

blend curved edgesto inner faces

blend curved facesto volumes

subdividedsurface mesh

curved high order mesh

block-structured mesh(cgns struct. format)

Agglomerationof subblocks to

curved hexahedra

5 / 22

HOPR Mesh Curving Flowchart

commercial mesher: ICEM, ANSA,... (cgns unstruct. format) linear unstructured hybrid mesh

curved boundary faces

Normal vector(s) at surface nodes

point-normalsICEM

spectral elementssurface

subdivision

ICEM Cheb.-Lobattonodes on edges

curved boundary edges

blend curved edges to face

blend curved edgesto inner faces

blend curved facesto volumes

subdividedsurface mesh

curved high order mesh

block-structured mesh(cgns struct. format)

Agglomerationof subblocks to

curved hexahedra

5 / 22

Curved edges from normal vectors

Get normal vectors at linear surface mesh points:

1. In HOPR without additional data:a) Reconstruction via linear surface meshb) Analytic functions (sphere ,cylinder ...)

2. Additional point-normal file(from CAD,...)

⇒ Generate curved edges

6 / 22

Curved edges from normal vectors

Get normal vectors at linear surface mesh points:

1. In HOPR without additional data:a) Reconstruction via linear surface meshb) Analytic functions (sphere ,cylinder ...)

2. Additional point-normal file(from CAD,...)

⇒ Generate curved edges

6 / 22

Multiple normal vectors to recover sharp edges & corners

p1

p2

n1

t 1

e p1

p2

n1

t 1

en2

(a) (b)

(a) 1 normal vector, tangential vector by projection

(b) 2 normal vectors, tangential vector by cross product

7 / 22

Blending curved edges to curved faces

=12 { X

a X

b X

c− X

t }

= X a X b − X t

Superposition of linear blended pairs of curved edges (Coons patches)

8 / 22

HOPR Mesh Curving Flowchart

commercial mesher: ICEM, ANSA,... (cgns unstruct. format) linear unstructured hybrid mesh

curved boundary faces

Normal vector(s) at surface nodes

point-normalsICEM

spectral elementssurface

subdivision

ICEM Cheb.-Lobattonodes on edges

curved boundary edges

blend curved edges to face

blend curved edgesto inner faces

blend curved facesto volumes

subdividedsurface mesh

curved high order mesh

block-structured mesh(cgns struct. format)

Agglomerationof subblocks to

curved hexahedra

9 / 22

High Order Surface by Subdivision

linear surface mesh

I Mesh generator: subdivision of the initial mesh while keeping newpoints on the CAD surfaces

⇒ Polynomial surface patches by interpolation

10 / 22

High Order Surface by Subdivision

after one subdivision

I Mesh generator: subdivision of the initial mesh while keeping newpoints on the CAD surfaces

⇒ Polynomial surface patches by interpolation

10 / 22

High Order Surface by Subdivision

after two subdivisions

I Mesh generator: subdivision of the initial mesh while keeping newpoints on the CAD surfaces

⇒ Polynomial surface patches by interpolation

10 / 22

Blending curved faces to curved volume

1. Curved face on boundary

2. Blend adjacent faces with linear edges

3. Blending of all element faces → volume mapping

4. Generalized for all element types (tets,pyramids,prisms,hexas)

11 / 22

Blending curved faces to curved volume

1. Curved face on boundary

2. Blend adjacent faces with linear edges

3. Blending of all element faces → volume mapping

4. Generalized for all element types (tets,pyramids,prisms,hexas)

11 / 22

Blending curved faces to curved volume

1. Curved face on boundary

2. Blend adjacent faces with linear edges

3. Blending of all element faces → volume mapping

4. Generalized for all element types (tets,pyramids,prisms,hexas)

11 / 22

Blending curved faces to curved volume

1. Curved face on boundary

2. Blend adjacent faces with linear edges

3. Blending of all element faces → volume mapping

4. Generalized for all element types (tets,pyramids,prisms,hexas)

11 / 22

Example: DLR-F6 Wing-Body-Nacelle Configuration

I Test case of an idealized passenger jetliner

I CAD geometry provided as STEP file

I Element layer on geometry ∼ 3600 hexa∼ 400 prism

I Volume grid : ∼ 225000 tetrahedra∼ 2900 pyramids

12 / 22

Example: DLR-F6 Wing-Body-Nacelle Configuration

13 / 22

Example: DLR-F6 Wing-Body-Nacelle Configuration

⇒ Visualization of polynomial patches shows smooth boundary surfaceelements

14 / 22

HOPR Mesh Curving Flowchart

commercial mesher: ICEM, ANSA,... (cgns unstruct. format) linear unstructured hybrid mesh

curved boundary faces

Normal vector(s) at surface nodes

point-normalsICEM

spectral elementssurface

subdivision

ICEM Cheb.-Lobattonodes on edges

curved boundary edges

blend curved edges to face

blend curved edgesto inner faces

blend curved facesto volumes

subdividedsurface mesh

curved high order mesh

block-structured mesh(cgns struct. format)

Agglomerationof subblocks to

curved hexahedra

15 / 22

Hexa-meshes: Agglomeration

I Start from fine block-structured meshes

⇒ Coarsen by a fixed factor

⇒ Use inner points to generate high order element mapping

16 / 22

Agglomeration Example: Serrated nozzle I

I Test case provided by Andrei Cimpoeru, Imperial College

I 85 blocks, 20.6M linear elements, 21.8M points, 0.5 GB CGNS file

I Agglomeration of 4x4x4 elements ⇒ 322,110 P4-elements

I HOPR used ≈ 6 GB RAM, run for ≈ 2 minutes(1 min eliminate duplicate nodes, 40s Jacobian checker)

17 / 22

Agglomeration Example: Serrated nozzle II

all elements curvedinvalid elements inside the volume

only curved at wall BCinvalid elements at the boundary

#elems with Jscaled < 0 0− 0.1 0.1− 0.2 0.2− 0.3 ≥ 0.3

all linear 0 0 65 204 321,841all curved 10 737 896 1,684 318,783curved BC 61 6 70 216 321,757

18 / 22

Agglomeration Example: Serrated nozzle II

1st layer curvedinvalid elements

2 layers curved⇒ all elements valid

#elems with Jscaled < 0 0− 0.1 0.1− 0.2 0.2− 0.3 ≥ 0.3

all linear 0 0 65 204 321,841all curved 10 737 896 1,684 318,783curved BC 61 6 70 216 321,757

1st layer 10 50 104 376 321,5702 layers 0 0 197 520 321,393

18 / 22

Agglomeration Example: Serrated nozzle II

3 layers curved⇒ all elements valid

4 layers curved⇒ all elements valid

#elems with Jscaled < 0 0− 0.1 0.1− 0.2 0.2− 0.3 ≥ 0.3

all linear 0 0 65 204 321,841all curved 10 737 896 1,684 318,783curved BC 61 6 70 216 321,757

1st layer 10 50 104 376 321,5702 layers 0 0 197 520 321,393

3 layers 0 0 178 667 321,2654 layers 0 0 160 746 321,204

18 / 22

Ongoing Work on Boundary Layer Meshes

I Structured grids:

⇒ Agglomeration (works in current HOPR release)

I Unstructured grids: Only curving the first element at the surfaceleads to inverted elements

⇒ For prismatic layers of linear elements:Blending of element stack with curved boundary face (not released)

⇒ Insert boundary layers afterwards (not released)

19 / 22

Boundary layer Insertion

I Valid curved mesh with prismatic elements (hex, prism)I Split the first element layer

⇒ into a given number boundary layer elements (input parameter)

⇒ with a given stretching (input parameter)

20 / 22

Boundary layer Insertion

I Valid curved mesh with prismatic elements (hex, prism)I Split the first element layer

⇒ into a given number boundary layer elements (input parameter)

⇒ with a given stretching (input parameter)

20 / 22

Conclusion

I HOPR provides unstructured high order meshes

I Different curving strategies, depending on provided data

I HDF5 mesh file, designed for parallel IO

I Software is open-source

⇒ Announcement for next ICOSAHOM:David Kopriva wants to organize a mini-symposium todiscuss a standard for a high order mesh format

21 / 22

Thanks to E. Sonnendrucker, C.-D. Munz, G. Gassner, T. Bolemann, H.Frank, and

Thank you for your attention!

www.hopr-project.org

HOPR Homepage (with Mailing List, Tutorials, Documentation...)

Contact: florian.hindenlang@ipp.mpg.de, bolemann@iag.uni-stuttgart.de

22 / 22

Appendix:

23 / 22

Mesh Format: Element Node Connectivity

I All element mappings in the mesh have the same order (watertight)I Element node connectivity defined for all orders (i , j , k loops)I Full order elements only

XI, i

ETA, j

ZETA, k

1 3

6

13 15

18

2

14

4

16

5

17

7 9

12

8

10 11

1 3

6

10

2

4 5

7 8

9

1 3

97

19 21

2725

2

8

20

26

4 6

22 24

5

23

10 12

1816

11

17

13 1514

1 3

97

14

2

8

4 65

10 11

12 13

24 / 22

SFC Domain Decomposition

I 3D unstructured simulation (N = 4) of a turbulent flow past asphere (Re = 1000)

I Domain decomposition based on a space filling curve element sorting

25 / 22

SFC Domain Decomposition

I SFC-DD allows arbitrary number of domains, given a-posteriori

#Domains #Elements/core #MPIneighbors (mean)128 165-166 5-23 (11.75)

25 / 22

SFC Domain Decomposition

I SFC-DD allows arbitrary number of domains, given a-posteriori

#Domains #Elements/core #MPIneighbors (mean)128 165-166 5-23 (11.75)

1024 20-21 5-21 (11.99)

25 / 22

SFC Domain Decomposition

I SFC-DD allows arbitrary number of domains, given a-posteriori

#Domains #Elements/core #MPIneighbors (mean)128 165-166 5-23 (11.75)

1024 20-21 5-21 (11.99)4096 5-6 5-20 (10.37)

25 / 22