Application of a Subgrid Orography Scheme to CAM and CLM

Steven Ghan and Tim ShippertPacific Northwest National Lab

Ghan, S. J., X. Bian, A. G. Hunt, and A. Coleman, 2002: The thermodynamic influence of subgrid orography in a global climate model, Climate Dynamics, 20, 31-44.

Subgrid scheme

Implementation

),min(

)()(

)()()(

*

*

N

Uzzh

hzXzX

XXXQXDXA

t

X

snn

nn

nnnn

n

• Applies to all column physics– clouds– radiative transfer– vertical mixing– surface physics

Elevation ClassificationElevation Classifi cation

-10000

100020003000400050006000700080009000

Su

rfac

e E

leva

tio

n (

m)

Progress

• Scheme applied to CAM and CLM – Euler dycore– Finite-volume dycore (1-D and 2-D domain decomposition)

• Developmental branch updated to cam2_0_2_dev55 (IPCC physics)

• Bit-for-bit agreement between SP, SMP, SPMD• Restarts bit-for-bit agreement• Bit-for-bit agreement with dev trunk if subgrid scheme turned

off• Energy conservation demonstrated*• Load balancing within nodes for both dycores• Load balancing between nodes for both dycores (FV 1-D only)• Runoff distributed according to elevation of river transport

model surface elevation

Experiments

• AMIP SST • Initialize September 1977• T42

– TOPOG run through 1983– No TOPOG data taken from cam2_0_2_dev61_climo

• 2x2.5– TOPOG run through 1989– No TOPOG run through 1989

Evaluation

temperature

March snow

Does the Scheme Impact the Grid Cell Mean Climate?

The subgrid scheme changes the large-scale climate significantly for the FV dycore, but not the

EUL dycore.

Planetary Energy Balance

Global Annual Mean

T42TOPOG

T42 NOTOP

T42 2X2.5 TOPOG

2X2.5 NOTOP

2X2.5

FSNT 235.2 234.0 1.2 234.5 232.2 2.3

FSNS 160.6 159.2 1.5 161.5 157.3 4.2

FLNT 233.6 233.5 0.1 228.9 233.2 -4.4

FLNS 57.5 56.5 1.1 63.9 57.9 6.0

RAD -101.5 -102.1 0.6 -91.9 -100.4 8.5

SHFLX 20.2 19.6 0.6 24.0 17.8 6.2

LHFLX 80.7 82.0 -1.3 93.8 81.9 11.9

TFLX 100.9 101.5 -0.6 117.7 99.8 17.9

RAD+TFLX

-0.6 -0.6 0.0 25.8 -0.6 26.4

PREC 2.786 2.831 -0.046 3.237 2.829 0.408

QFLX 2.785 2.831 -0.046 3.239 2.829 0.410

What is Causing the FV Problem?

Divergence of T and ST

Computational Burden

Zonal Mean Burden

Global Burden

Spectral Resolution

T42 T85 T170

Average classes per cell 2.28 1.85 1.58

Maximum classes in a cell 11 11 10

Maximum zonal mean classes per cell

3.8 3.2 2.7

Balancing the Load•Distribute elevation classes across physics chunks

–Rank grid cells according to number of classes–Starting with cells with most classes, assign all classes in cell to the chunk with the least number of columns–Continue distribution until classes for all cells have been assigned to chunks–Save cell index and area corresponding to each chunk column

•Distribute chunks across nodes

-Minimize dynamic-physics transpose cost by assigning chunks to node with most dynamics grid cells from chunk

-fraction of columns assigned to same node as dynamics: 40% for T42 11 classes on 8 nodes

Distribution of Physics Load in Chunks

1 2 3 4 5 6 7 8 9 10 12 13 14 15 16

Columns In Chunk

Num

ber o

f Chu

nks

Distribution of Chunks Across Nodes

0

1

2

3

4

5

6

7

8

1 150Chunks on Node

Nu

mb

er o

f No

des

Load Balancing Performance

10

100

1000

1 10 100

MPI Processes

T4

2 W

allc

loc

k R

un

Tim

e

(se

co

nd

s/d

ay

)

NCLASS=1

NCLASS=11 No load balance

NCLASS=11 Load balance

Code Changes

• Unlike any other parameterization– All column physics applied to each elevation class

• Mostly manifest at higher levels– phys_grid

phys_grid_initcreate_chunksscatter_field_to_chunkgather_chunk_to_field

– phys_types– dp_coupling– history

Future Work

• Extend the T42 subgrid run and run control for comparison.• Correct the FV energy imbalance and rerun FV 2x2.5.• Apply load balancing to the finite-volume implementation of

the scheme, for 2D domain decomposition.• Present to CCSM LMWG for approval.• Submit proposal to CAM code review board.• Check in as permanent CAM and CLM option.• Perform simulations at higher resolution with subgrid

scheme.• Perform IPCC simulations with CAM using the subgrid

scheme and ocean surface conditions from CCSM simulations without the subgrid scheme.

• Adapt the scheme to CCSM.• Continue to update as CCSM changes.