Development of Development of CCoastal oastal OOcean cean MModeling odeling IInfrastructure nfrastructure

(COMI) at LSU(COMI) at LSU

Q. Jim Chen

Department of Civil and Environmental Engineering& Center for Computation and Technology

Louisiana State University

Acknowledgements

COMI Team:Drs. Gabrielle Allen, Mayank Tyagi (CCT)Drs. Claes Eskilsson, Kelin Hu (CEE/CCT)Yaakoub El-Khamra, Lei Jiang (CS)Qi Fan, Ranjit Jadhav, Qian Zhang, Haihong Zhao (CEE)

Funding:NSF, ONR, NOAA

Gulf Shores, Alabama

Motivation

Combined waves and surge

Depth-integrated coastal models

Higher-order finite element methods

Modeling framework and results

Conclusion and future work

Outline

Courtesy of Lake Pontchartrain Basin Foundation

Animation of Wave-Structure-Seabed Interaction

Louisiana Coastal Protection Assessment of Multiple-lines of Defense

Chen et al. (2006)

Wave overtopping

Bridge damaged by storm waves Combined

Waves & Surge

news.bbc.co.uk/.../uk_enl_1194609737/img/1.jpg

Kevork Djansezian/ AP

Gulfport, Mississippi during Gustav

US -90 bridge over Biloxi Bay, MS

Impact of Katrina on WetlandsUK

Challenges

Hurricane protection and coastal restoration require a suite of numerical models for different physical processes.

Different models use different numerical methods to solve different governing equations, e.g. FDM, FEM, FVM, SEM

There is a need for developing a common modeling framework to enable the modeling of multi-physics, multi-scale coastal processes using HPC.

CFD Toolkit

ComposingFluid-flow solver

Solvers

SuperLU

Trilinos

PETSc

Numerics

Finite difference

Finite Volume

Spectral Element

Infrastructure

Cartesian

M-B curvilinear

Unstructured

Physics

Navier-Stokes

RANS

Boussinesq

Other coastal models

To develop the capability of modeling coastal circulation and nearshoresurface waves in deltaic environments using high-order numerical methods

To integrate application-oriented coastal modeling systems with massive-processor, high-performance computing facilities and technologies available at LSU

Objectives

Depth-Integrated Coastal Models

3-DimensionalEuler Equations

Boussinesq-typeEquations

Nonlinear ShallowWater Equations

Linear Long Wave Equations

Mild SlopeEquations

Wind Stress

Storm Surge

Wave Growth Storm Surge

Modeling Surge/Wave Attenuation by Porous Media or Vegetation

Porous Media

Cruz and Chen (2007)

Why higher-ordernumerical methods?

Unstructured Spectral/hpDG Methods

Solution approximated inside an elementby a pth order polynomial expansion

(note: can be non-uniform).

Elements coupled throughso-called numerical fluxes

(computed by approximate Riemann solvers)

Supports unstructured meshes (conforming or non-conforming) consistingof elements of size h. Note that the solution

is allowed to be discontinuous over theelement boundaries.

CACTUS Framework

CACTUS is a freely available, modular, portable and manageable environment

for collaboratively developing parallel, high-performance multi-dimensional simulations

http://www.cactuscode.org/Successful Applications: Astrophysics,Petroleum Engineering,

http://www.cactuscode.org/

COMI Architecture

UMDriver provides the underlying parallel layer. At present it uses the Zoltan library to provide mesh partitioning, load balancing and mesh migration

LocalToGlobal provides local re-indexing of elements, edges and vertices The Nektar++ thorn initializes and populates the data structures of the

Nektar++ library MeshReader provides a simple ASCII mesh file reader, and also allows users to

register their own mesh readers (e.g. the mesh reader from Nektar++) The core thorn for the coastal modeling toolkit in Cactus is CoastalWave. This thorn

defines the generic variables, parameters, and methods for coastal models Thorn SWE contains the actual SWE solver

Nektar++ library

Nektar++ is an open-source spectral/hp element library presently in the last stages of development (www.nektar.info)

As the name suggests Nektar++ is written in C++

Nektar++ is developed and maintained by Prof. Sherwin (Imperial College London) and Prof. Kirby (University of Utah)

COMI's shallow water codes are based upon Nektar++ and will be distributed as part of the Nektar++ solver library

http://www.nektar.info/

Discontinuous GalerkinSpectral Element Model

Flood Wall

Flood wal

Breach

Domain distributed over 8 cores

Sixth order triangular elements

COMI Result Eskilsson et al. (ICCS 2009)

Parallel DG Model

Weak Scaling 900 quadrilateral elements per core 100 time steps Times without I/O, initializing and partitioning of the mesh 128 cores, p = 8 roughly 28 million DoF

Hurricane Gustav Animation of the combined H*wind and background winds (NCEP/NCAR Reanalysis)

Wind Comparison

0

10

20

30

40Buoy 42040

Win

d sp

eed

(m/s

)

0

10

20

30

40Buoy 42007

Win

d sp

eed

(m/s

)

0

10

20

30

40Dauphin Island

Win

d sp

eed

(m/s

)

Aug.31 00 12 Sep.1 00 12 Sep.2 000

10

20

30

40PSTL1 SW Pass

Win

d sp

eed

(m/s

)

Time (days, Aug.30 ~ Sep.2, 2008)

0

90

180

270

360Buoy 42040

Win

d di

rect

ion

(deg

rees

)

0

90

180

270

360Buoy 42007

Win

d di

rect

ion

(deg

rees

)0

90

180

270

360Dauphin Island

Win

d di

rect

ion

(deg

rees

)

Aug.31 00 12 Sep.1 00 12 Sep.2 000

90

180

270

360PSTL1 SW Pass

Win

d di

rect

ion

(deg

rees

)

Time (days, Aug.30 ~ Sep.2, 2008)

90.5 90.0 89.5 89.0 88.5 88.0 87.5 87.0 86.5 86.0 85.5 85.028.5

29.0

29.5

30.0

30.5

31.0

Longitude (degrees)

Latit

ude

(deg

rees

)

Buoy 42040

Buoy 42039

Buoy 42007

SWP BURL1

CSBF1

Grand Isle

Dauphin Island Perdido PassFort Walton

Panama Beach

Apalachicola

PSTL1 SW Pass

State Docks

Mobile Airport Pensacola AirportDauphin Island Sea Lab StationMeteorology Stations

Tide Gages

C-MAN StationsBuoy Stations

42007

42040

Total triangle elements: 63,757 Total nodes: 32,544. Time step: 4 s

The basinThe basin--scale mesh provides the scale mesh provides the open boundary conditions.open boundary conditions.

Both ADCIRC and SWAN are coupled Both ADCIRC and SWAN are coupled using the same regional mesh.using the same regional mesh.

Fine spatial resolution is needed to Fine spatial resolution is needed to resolve topographic and hydrodynamic resolve topographic and hydrodynamic features of flooding and coastal waves.features of flooding and coastal waves.

Nested Domain for Wave-Surge Coupling

Lake Pontchartrain

New Orleans

ADCIRC Mesh

Animation of Modeled Significant Wave Heights (m) during Gustav

Modeled Maximum Significant Wave Heights (m) during Gustav

Wave Comparison

90.5 90.0 89.5 89.0 88.5 88.0 87.5 87.0 86.5 86.0 85.5 85.028.5

29.0

29.5

30.0

30.5

31.0

Longitude (degrees)

Latit

ude

(deg

rees

)

Buoy 42040

Buoy 42039

Buoy 42007

SWP BURL1

CSBF1

Grand Isle

Dauphin Island Perdido PassFort Walton

Panama Beach

Apalachicola

PSTL1 SW Pass

State Docks

Mobile Airport Pensacola AirportDauphin Island Sea Lab StationMeteorology Stations

Tide Gages

C-MAN StationsBuoy Stations

25 30 35 40 45 50 55 60 65 700

2

4

6

8

10

12Buoy 42040 (Red for the observed;Blue for the unstructured modeled;

Hs

(m)

25 30 35 40 45 50 55 60 65 700

2

4

6

8

10

12Black for the rectangular modeled; Green for the unstructured modeled with no elevation change)

Wav

e pe

riod

(s)

25 30 35 40 45 50 55 60 65 700

2

4

6

8

10

12Buoy 42007

Hs

(m)

Hours from 2008-08-30 00h25 30 35 40 45 50 55 60 65 70

0

2

4

6

8

10

12

Wav

e pe

riod

(s)

Hours from 2008-08-30 00h

42007

42040

Contribution of Storm Surge to Maximum Significant Wave Heights during Gustav

Conclusion and Future Work

A coastal modeling framework has been developed at LSU using spectral/hp DG methods and HPC technologies.

Effects of vegetation and fluid mud are being incorporated into the solvers.

Recent hurricanes provide an excellent testbed for skill assessment of coupled, unstructured surge, wave, salinity and sediment transport models

Surge Comparison

20 30 40 50 60 70 800

0.5

1

1.5

2PSTL1 (red for the observed; blue for the calculated)

Sur

ge (m

)

20 30 40 50 60 70 800

0.5

1

1.5

2

Sur

ge (m

)

Dauphin Island (red for the observed; blue for the calculated)

20 30 40 50 60 70 800

0.5

1

1.5

2

Sur

ge (m

)

Panama City

20 30 40 50 60 70 800

0.5

1

1.5

2

Sur

ge (m

)

Pensacola

Hours from 08-30-2008 00:00

20 30 40 50 60 70 800

0.5

1

1.5

2

Sur

ge (m

)

Mobile State Docks

Hours from 08-30-2008 00:0020 30 40 50 60 70 80

0

0.5

1

1.5

2S

urge

(m)

GDIL1

Hours from 08-30-2008 00:00

90.5 90.0 89.5 89.0 88.5 88.0 87.5 87.0 86.5 86.0 85.5 85.028.5

29.0

29.5

30.0

30.5

31.0

Longitude (degrees)

Latit

ude

(deg

rees

)

Buoy 42040

Buoy 42039

Buoy 42007

SWP BURL1

CSBF1

Grand Isle

Dauphin Island Perdido PassFort Walton

Panama Beach

Apalachicola

PSTL1 SW Pass

State Docks

Mobile Airport Pensacola AirportDauphin Island Sea Lab StationMeteorology Stations

Tide Gages

C-MAN StationsBuoy Stations

Modeled Maximum SurgeHeights (m, MSL) during Gustav

What is the cause of Katrinas record high surge?

250 200 150 100 50-500

-400

-300

-200

-100

0

Ave

rage

dep

th b

elow

MS

L (m

)

Distance from shoreline along the track of hurricane (km)

KatrinaIvanFrederic

50 40 30 20 10 00

2

4

6

8

10

Sur

ge H

eigh

t (m

,MS

L)

Distance from shoreline (km)

Sloping BottomFlat Bottom

Chen et al. (2008)

Structure of Cactus

Core Flesh

Plug-In Thorns(modules)

driverdriver

input/outputinput/output

interpolationinterpolation

SOR solverSOR solver

coordinatescoordinates

boundaryboundary conditionsconditions

black holesblack holes

equations of stateequations of state

remote steeringremote steering

wave evolverswave evolvers multigridmultigrid

parametersparameters

gridgrid variablesvariables

errorerror handlinghandling

schedulingscheduling

extensibleextensible APIsAPIs

makemake systemsystem

ANSI CANSI C

Fortran/C/C++Fortran/C/C++

Reservoir simulatorsReservoir simulators

Coastal modellingCoastal modelling

Molecular dynamicsMolecular dynamics

A Fully Nonlinear Boussinesq Model for Waves and Currents

Continuity Equation

Momentum Equation

)( 04 ghOM

t =+

)(

)()(

0

04

3212

lghORRR

VVVguut

u

wfi

=+

+++++

Wei et al. (1995) and Chen et al. (2003, 2004, 2006)

= free surface elevationM = volume rate of flow

Boussinesq dispersive terms

Momentummixing term

Wind forcing

Conservation ofpotential vorticity

Sea-Dependent Wind Drag Coefficients

A new formulation

Wu (1980)

10213 |)|(10 bUaaC xd ++=

103 065.08.010 UCd +=

a1= 0.2a2 = 18b = 0.065

Surface slope

(Sea-independent)

0 5 10 15 20 25 300

0.5

1

1.5

2

2.5

3

U10(m/s)

Cd

x 10

3

ka0.2

0.15

0.1

0.05

Large and Pond (81)Smith (80)Smith and Banke (75)Geernaert et al.(87)Geernaert et al.(86)Sheppard et al. (72)Donelan (82)Denman and Miyake (73)Pond et al. (71)Graf et al. (84)

Ka=2

(Chen et al, 2004)

Wind Stress = Skin Friction + Form Drag

Challenges Depth-Integrated Coastal Models Modeling Surge/Wave Attenuation by Porous Media or VegetationWhy higher-ordernumerical methods? Unstructured Spectral/hp DG MethodsCOMI ArchitectureNektar++ library Discontinuous Galerkin Spectral Element Model Parallel DG Model Weak ScalingHurricane GustavWind ComparisonNested Domain for Wave-Surge CouplingAnimation of Modeled Significant Wave Heights (m) during GustavModeled Maximum Significant Wave Heights (m) during GustavWave ComparisonContribution of Storm Surge to Maximum Significant Wave Heights during GustavConclusion and Future WorkSurge ComparisonModeled Maximum SurgeHeights (m, MSL) during Gustav What is the cause of Katrinas record high surge?Structure of CactusA Fully Nonlinear Boussinesq Model for Waves and CurrentsSea-Dependent Wind Drag Coefficients