Introduction Turbulence Simulations

Direct Numerical Simulation of Turbulent Flows

Michael Wilczek

Institute for Theoretical Physics, University of Munster

22.07.09

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Is turbulence a solved problem?

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Is turbulence a solved problem?

Engineering: conceptually yes

(depends on the complexity of your problem)

relevant scenarios are computationallyaccessible

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Is turbulence a solved problem?

Engineering: conceptually yes

(depends on the complexity of your problem)

relevant scenarios are computationallyaccessible

Physics: no

statistical description from first principles ismissing

→ non-equilibrium thermodynamics?

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Is turbulence a solved problem?

Engineering: conceptually yes

(depends on the complexity of your problem)

relevant scenarios are computationallyaccessible

Physics: no

statistical description from first principles ismissing

→ non-equilibrium thermodynamics?

Mathematics: no

boundedness of solutions with smooth initalconditions?

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Introduction

Problem: turbulence . . .

is described by nonlinear equations

exhibits spatio-temporal chaos

involves large space- and time-scales

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Introduction

Problem: turbulence . . .

is described by nonlinear equations

exhibits spatio-temporal chaos

involves large space- and time-scales

Possible solutions:

understanding of structures

formulating a statistical theory

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Introduction

Problem: turbulence . . .

is described by nonlinear equations

exhibits spatio-temporal chaos

involves large space- and time-scales

Possible solutions:

understanding of structures

formulating a statistical theory

Tools:

any kind of mathematics, that will do

computer simulations

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: Visualization

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: equations

Navier-Stokes equations:

∂u

∂t(x, t) + u(x, t) · ∇u(x, t) = −∇p(x, t) + ν∆u(x, t) + f(x, t)

∇ · u(x, t) = 0

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: equations

Navier-Stokes equations:

∂u

∂t(x, t) + u(x, t) · ∇u(x, t) = −∇p(x, t) + ν∆u(x, t) + f(x, t)

∇ · u(x, t) = 0

Vorticity: ω(x, t) = ∇× u(x, t)Vorticity equation:

∂ω

∂t(x, t) = ∇×

(

u(x, t) × ω(x, t))

+ ν∆ω(x, t) + f(x, t)

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: numerics I

aim: forced (stationary) homogeneous, isotropic turbulence

temporal discretization: RK3 TVD

spatial discretization: box-length 2π, dim grid points, periodicboundary conditions

pseudospectral code

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: numerics II

∂ω

∂t(k, t) + ν k2 ω(k, t) = ik ×F{u(x, t) × ω(x, t)} + f(k, t)

adaptive time-stepping (Courant-Friedrichs-Levy criterion)

pseudospectral: forward/backward FFT is computationallycheaper than convolution (N log N vs. N2)

aliasing: smooth Fourier filter

viscosity is treated exactly (integrating factor)

forcing: freezing of low modes

code is MPI parallel

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: computational costs IE

(k)

∼ k−5/3

inertial range

extends with Re

integral scale

dissipative scale

k

forcing

dissipation

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: computational costs II

forcing scale and dissipative scale should be well seperated

inertial range extends with increasing Re

size of smallest structures decreases with Re

smallest structures should be well-resolved by the grid

turbulent field should be accurately advanced in time

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: computational costs III

to be more precisely . . .

η =

(

uL

ν

)

−3/4

L = Re−3/4L

∆x ∼ η

Nx ∼

(

2π

∆x

)3

∼

(

2π

L

)3

Re9/4 −→ Re ∼

(

L

2π

)4/3

N4/9x

Nt ∼T

∆t∼

T

∆x/u∼

T

l/uRe3/4

$$$ ∼ NxNt ∼

(

T

l/u

)(

2π

l

)3

Re3

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: computational costs IV

10243512325631283

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: computational costs V$$

$

serial

OpenMP

MPI

2563

Re

10243

and beyond

5123

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: computational costs V$$

$

serial

OpenMP

MPI

2563

Re

10243

and beyond

5123

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: computational costs V$$

$

serial

OpenMP

MPI

2563

Re

10243

and beyond

5123

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: computational costs V$$

$

serial

OpenMP

MPI

2563

Re

10243

and beyond

5123

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: OpenMP vs. MPI

OpenMP

“fork and join” principle

easy

incrementalparallelization

medium number of cores

limited degree ofscalability

MPI

decomposition ofcomputational domain

“communicate whennecessary” principle

broad range ofapplication

high scalability

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Amdahl‘s Law

consider serial code with runtime 1

P denotes the parallelizable fraction of code

runtime on n cores: (1 − P ) + PN

maximum expectable speed up S(N) = 1(1−P )+ P

N

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Amdahl‘s Law

consider serial code with runtime 1

P denotes the parallelizable fraction of code

runtime on n cores: (1 − P ) + PN

maximum expectable speed up S(N) = 1(1−P )+ P

N

2

4

10

20

100

2 8 32 128 512 2048 8192 32768

spee

d up

number of cores

P=0.50P=0.75P=0.90P=0.95P=0.99

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: parallelization via MPI

DD

core 3

core 2

core1

core 0

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS: scaling

32

16

8

4

2

1

1024 512 256 128 64 32

spee

dup

number of cores

scaling performance of VOrTEx on HLRB II, 10243 grid points, single precision

reference point

two partitions in use

three partitions in use

high density blades

ideal scaling

5.28

9.19

18.5

34.55

60.15

119.17

1024 512 256 128 64 32

time

per

step

(se

c)

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

Demos

Hello OpenMP/ MPI world

HLRBII @ LRZ

remote visualization

Michael Wilczek Direct Numerical Simulation of Turbulent Flows