Developments in the dynamical core of the global semi-Lagrangian SL-AV model
description
Transcript of Developments in the dynamical core of the global semi-Lagrangian SL-AV model
Developments in the dynamical core of the global semi-Lagrangian SL-AV modelMikhail Tolstykh, Vladimir Shashkin
Institute of Numerical Mathematics, Russian Academy of Sciences
andHydrometcentre of Russia
With contribution by Vassili Mizyak
Outline
• SL-AV model• Mass-conservative shallow-water version on
the reduced lat-lon grid - results of standard SW tests• New solvers for U-V reconstruction and
Helmholtz problem
SL-AV model. General features
• Global finite-difference semi-Lagrangian semi-implicit dynamical core of own development + parameterizations of subgrid-scale processes from French model ARPEGE/ALADIN.
• Main Russian global operational model for medium-range and seasonal forecasts.
• So far, regular lat-lon grid is used. Reduced lat-lon grid in the shallow water version
SL-AV model.
Distinct features of dynamical core
• Vorticity-divergence formulation on the unstaggered grid, wide use of 4th order finite differences, usual and compact (Tolstykh, JCP 2002).
• Semi-implicit solver, reconstruction of U and V, horizontal diffusion are carried out in Fourier space in longitude.
• Version with variable resolution in latitude (quasioperational)
• Scales up to 100-300 cores, MPI+OpenMP
Reduced lat-lon grid
Reduced grid in the SL-AV model
• A part of calculations is carried out in space of Fourier coefficients in longitude.
• The semi-Lagrangian advection is used (no nonlinear advective terms)
=> Necessary latitudinal derivatives (i.e. geopotential gradient) can
be calculated in Fourier space. Number of gridpoints at each latitudinal circle should be suitable
for FFT.
Currently, implemented in the shallow-water version (standard and mass-conservative versions).
Motivation for a new version of dynamical core
• Future use for climate simulations• Future horizontal resolution of O(1-10km)=>• Mass conservation• More scalability => new algorithms
Next version of the SL-AV dynamical core
• Mass conservative. More uniform resolution on the sphere – reduced lat-lon grid.
• More scalable- abandon data transpositions- new parallel algorithms to solve block-
tridiagonal linear systems.
Mass-conservative shallow-water version of SL-AV
• Mass-conservative semi-Lagrangian advection scheme on the sphere on the reduced lat-lon grid.
• Mass-conservative semi-implicit time integration scheme. This implies a conservative Helmholtz problem solver on the sphere.
Mass-conservative semi-Lagrangian advection scheme on the sphere on the reduced grid
• Global conservative cascade scheme (CCS) was implemented for SL advection following “Efficient Conservative Global Transport Schemes for Climate and Atmospheric Chemistry Models” R.D. Nair et al. MWR 2002. (Meridional CFL number ≤ 1)
• Some changes were introduced for using CCS on the reduced grid.
Example of the reduced grid (region)Stars – grid pointsRegions bounded by dashed lines – grid cells
Departure volume calculation
Red lines – Lagrangian (departure) latitudes; Red stars – corners of the departure volume; Green pentagons – points where Lagrangian latitudes cross longitudes of the arrival grid (points of the intermediate grid); green dotted lines – midpoint values for latitudes of the intermediate grid defining north and south boundaries of the intermediate grid cells (grey rectangles). Orange lines – approximation for west (east) boundaries of departure volume
CCS on the reduced grid (1)
MM - mass of the advected quantity bounded in the cell of arrival (computational) grid. M * - mass bounded in the cell of departure grid.
Direct conservative cascade remapping of the mass from the reduced grid to any kind of departure grid is impossible due to the absence of well defined grid longitudes matching poles of the sphere in the reduced grid. So we implemented slightly different scheme:
Mn Mn* = Mn+1
Conservative cascade remapping
Mn+1*
Conservative cascade remapping
Mn
Conservative remapping
Mnreg Mn
* = Mn+1
Conservative cascade remapping
TThe reduced grid is used as computational and arrival grid. Mnreg - masses of
the advected quantity bounded in the cells of regular grid.
The structure of the departure grid corresponding to the reduced arrival grid is suitable for CCS since such departure grid forms well-defined intermediate cells (see Nair et al. for details)
CCS on the reduced grid (2)
We have chosen regular grid such that latitude grid lines of regular and reduced grid coincide and the number of grid points in latitude line of regular grid is equal to the maximum number of grid points in the latitude lines of the reduced grid.
1D remapping technique we use for remapping between reduced and regular grids is the same as the one used in 2D conservative cascade remapping
The first 1D remapping in the CCS is in latitudinal direction. The application of conservative scheme using first remapping in longitudinal direction (ex. SLICE by Zerroukat et al.) on the reduced grid is much more complicated.
Mass-conservative Helmholtz problem solver Semi-implicit time discretization of the SW equations leads to the
Helmholtz problem to be solved at each time step:
Φ – height, Φref – reference height, F – some known function, D – divergence of the wind field, A – some known vector, μ is a constant.
If Si, j – the volume of i, j-th cell of computational grid then the mass conservation is expressed by
The mass-conservation can be achieved if discrete Laplacian and divergence operators satisfy
refref
nnnnn AdivDF
;
),,(1212
jiji
n
jiji
n SS,
,,
,1
0;0,
,,,
,,2
jijiji
jijiji SAdivS
These operators are constructed with a finite-volume approach
(in Zerroukat et al 2009, FDs are used)
Results of standard tests - preliminaries
Tests were carried out using the regular grids of 1.5, 1, 0.75, 0.5 degrees resolution in latitude and longitude and the reduced grid of 1.5 and 0.75 degrees resolution in latitude and the same maximum resolution in longitude. Test cases ## 1, 7 were carried out only on the regular and reduced grids of 1.5 degree resolution.
Reduced grid for the tests was constructed by R. Fadeev (Fadeev R.Yu., Construction of a reduced latitude-longitude grid for a global numerical weather prediction problem, Russian Meteorology and Hydrology, 2006, N 9). Reduction rate was 10%.
For comparison, the same tests were carried out with the original non-conservative SW model.
In the tables, the abbreviation SL denotes the original non-conservative model and CM denotes its mass conservative modification
Resolution 1.50 1.00 0.750 0.50
Timestep(sec) 2700 1800 1350 900
Test case 1 - advection over the poleResolution in latitude and longitude (maximum resolution for reduced grid) - 1,50.
Meridional Courant number 0.5
Resolution l1 x10-2 l2x10-2 l∞x10-2 MAX x 10-2
CM regular grid 1.50 1.97 1.34 1.34 -0.4
CM reduced grid 1.50 1.97 1.34 1.34 -0.4
SL regular grid 1.50 6.03 3.78 3.10 -1.8
Height error measures after 1 revolution
The initial distribution (cosine bell) (A). Error fields after one revolution – CM on regular grid (B), CM on reduced grid (C), SL on regular grid (D)
CA B DC
Test case 2 – Global steady state geostrophic flow Angle α = π/4.
Resolution 1.50 1.50 1.00 .750 .750
Grid reduced regular regular reduced regular
CM 0.50E-04 0.18E-04 0.86E-05 0.11E-04 0.45E-05
SL 0.29E-04 0.22E-04 0.90E-05 0.67E-05 0.50E-05
l2 height error after 5 days of modeling
l2 height error for CM on the regular grid with resolution 1.50 (red line), 1.00 (green line), .750 (blue line)
Test case 6 – Rossby-Haurwitz wave #4
l2 height error after 14 days of modeling
Height field after 14 days of modeling with CM (regular grid 1.50 of resolution). Reference solution (A), CM error (B), SL error (C) (all on regular grid).
Resolution 1.50 1.50 1.00 .750 .750 .50
Grid reduced regular regular reduced regular regular
CM 0.47E-04 0.47E-04 0.30E-04 0.23E-04 0.22E-04 0.18E-04
SL 0.48E-04 0.48E-04 0.30E-04 0.23E-04 0.23E-04 0.18E-04
A B C
Test cases 7 – Real data
Tests were carried out with the fixed resolution of 1.50 .
l2 height error after 5 days of modeling
Test case 7a 7a 7b 7b 7c 7c
Grid reduced regular reduced regular reduced regular
CM 0.17E-02 0.17E-02 0.21E-02 0.21E-02 0.22E-02 0.22E-02
SL 0.17E-02 0.17E-02 0.22E-02 0.22E-02 0.23E-02 0.23E-02
A B C
Galewsky case4th order diffusion was applied to divergence field only. CM solution on the regular grid with
the resolution of 0.3250 is a reference.
Vorticity field after 144h. CM (regular grid): 1.0 – A, .750 – B, .50 – C, .3250 – D. Panel E - vorticity errors with respect to the reference solution
A B C
A B
C
D
E
New solver for U-V reconstruction (1)
• Old solver: solution of 2 Poisson equations, + differentiation using
Helmholtz relations
In a parallel environment, requires given Fourier coefficient at all latitudinal points. So the need for data transposition that limits scalability.
D 2
cos
11
aau
aa
v1
cos
1
2
New solver for U-V reconstruction (2)
New solver: direct solution of the system
=>
• Possibility to apply a parallel solver. No need for data transposition.
cos
cos
1
cos
cos
1
u
a
Du
a
kre
k
rek
im
kim
k
imk
re
u
u
aak
Daa
k
cos
cos1
cos
cos
cos1
cos
New solver for U-V reconstruction (3)
k
iki
ki
ki
ki
ki
i
k
k
i
k
k
i
k
k
aaa
Da
Da
Da
uC
uB
uA
11
11
11
63
2
6
63
2
6
62
cos2
cos
61
1
k
h
h
k
Ai
i
,
3
20
03
2
k
k
B,
62
cos2
cos
61
1
k
h
h
k
Ci
i
.
0,00E+00
2,00E-04
4,00E-04
6,00E-04
8,00E-04
1,00E-03
1,20E-03
1,40E-03
1,60E-03
1.5 2.0 2.5
U, original solver
V, original solver
U, new solver
V, new solver
Normalized L2-error for U in 3D steady-state test 1 by Jablonowski et al: old solver (left), new
solver (right)
0,000000000000000E+00
5,000000000000000E-03
1,000000000000000E-02
1,500000000000000E-02
2,000000000000000E-02
2,500000000000000E-02
3,000000000000000E-02
3,500000000000000E-02
4,000000000000000E-02
4,500000000000000E-02
0 100 200 300 400 500 600 700
Ряд1
0,000000000000000E+00
1,000000000000000E-02
2,000000000000000E-02
3,000000000000000E-02
4,000000000000000E-02
5,000000000000000E-02
6,000000000000000E-02
0 500 1000 1500 2000 2500
New solver also can provide better parallelization using parallel dichotomy algorithm (Terekhov, Par. Computing 2010)
Conclusions• Mass-conservative shallow-water version of SL-AV model
was created. • The error measures in standard SW tests are the same for
conservative and non-conservative versions, except for test #1 where conservative scheme gives smaller errors, and test #2 where conservative scheme gives larger errors.
• The impact of reduced grid on errors is negligible except for test #2.
• The new solver implemented for U-V reconstruction is cheaper, more accurate and more scalable.
• Further work – to implement mass-conservative version of the 3D model.
• There is also ongoing work on NH dynamical core development
Thank you for attention!
Mass-conservative Helmholtz problem solver (2) To obtain discrete divergence operator, the cell-averaged divergence is
represented as a flux of vector field through the cell-boundaries. If Ci,j is an arbitrary non-polar grid cell, and (u,v) is an vector field then:
I east , I west, I north, I south designate flux of (u,v) through one of the cell boundaries. Since cell boundaries are parallel to the coordinate axes, flux operators I are defined by straightforward expressions.
)(),(cos),(1
),( ,,,,2,
,,
jisouth
jinorth
jiwest
jieast
ijCijCijji IIII
S
andvu
S
addavudiv
Svudiv
jiji
If C0 is the south polar cap, and Ci,1 , i=1,M are the cells around, then polar cap averaged divergence is defined as follows
M
i
isouthI
S
avudiv
1
1,
00),(
Flux operators and so cell-averaged divergence operator are discretized with fourth-order accuracy. Laplacian is discretized as cell-averaged divergence of gradient with second order.
Divergence and Laplacian discretized in this way satisfy the last equality from the previous slide and produce conservative Helmholtz problem solver implying tridiagonal matrix inversion.
Another way to achieve mass-conservation is the use of conservative finite-differences for discretization of Helmholtz equation (as in Zerroukat et al. 2009)
Non-conservative model: relative total mass errors
Regular grid with the resolution 1.50 – red lines, 10 – green lines, 0.750 – blue lines; tests #1 (A), #2 (B), #6 (C), #7a (D)
A B
C
D
B
C
A
D