Current EMC global model dynamics and beyond...2016/05/24 · 3 June 2016 TWPGFS2016, H. Juang 1...
Transcript of Current EMC global model dynamics and beyond...2016/05/24 · 3 June 2016 TWPGFS2016, H. Juang 1...
3 June 2016 TWPGFS2016, H. Juang 1
Current EMC global model dynamics and beyond
Hann-Ming Henry Juang
Environment Modeling Center, NOAA/NWS/NCEP, Washington, DC
Concerns
• Current EMC global model dynamics is a spectral model.
• Two dynamics versions in the code
– Operational hybrid coordinate SL
– Generalized coordinate enthalpy NDSL
• Beyond (in-house GSM and NGGPS)
– Nonhydrostatic system
– Deep atmosphere
3 June 2016 TWPGFS2016, H. Juang 2
Contents
• Advantages of traditional global spectral model
• Developments has been done – Improve grid structure
– Improve thermodynamics
– Improve semi-Lagrangian
– Improve spectral transform
• Under testing and further plan– enhance parallelization
– different spectral truncation
– nonhydrostatic system
– deep atmosphere system
3 June 2016 TWPGFS2016, H. Juang 3
Traditional GSM
• The problem on sphere => Spherical coordinate
• Spherical harmonic (spectral transform)
– > leads to no pole problem
• Highest accuracy in horizontal discretization
• No bias due to computing direction
• Linear computation has no phase error
• easy to do semi-implicit time scheme
• Only vertical grid discretization needed
3 June 2016 TWPGFS2016, H. Juang 4
Contemporary GSM
• Reduced Spherical Gaussian grid
• Generalized vertical coordinates
• Considering gases into thermodynamics
• Semi-Lagrangian advection with semi-implicit
• Refine Legendre coefficient for high resolution
• Expand full dimensional MPP
• Exam cubic truncation with less diffusion
• Consider deep atmospheric dynamics
3 June 2016 TWPGFS2016, H. Juang 5
6
Juang 2004
Juang 2004
Based accuracy of associated Legendrepolynomial function, higher latitudes havefew significant waves leads to smaller gridnumber for spectral transform
3 June 2016 TWPGFS2016, H. Juang 11
Table 2: Details of model configurations.
NH-GFS (Baseline) * FV3 MPAS NIM NMMB-UJ NEPTUNE IFS (RAPS13) *
Resolution 13 km (TL1534) ~12 km (C768)* 12km * 13.4 * 13 km 12.71 km * 12 5 km (Tc799)
Grid Points3072x1536 (unreduced)
3,126,128 (reduced)
6x768x768
3,538,9444,096,002 ** 3,317,762
6x768x768
3,538,944 *3,110,402 **
3,336,946
(reduced)
Vertical Layers * 128 127 ** 127 *** 128 128 127 *** 137
Time Step TBD
600s (slow phys)
150s (vertical, fast
phys)
150/10 (horiz.
acoustic)
72 s (RK3 dynamics)
12 s (acoustic)
72 s (RK3 scalar
transport)
72 s 24 s **75 s (advective),
15 s (sound) ****450
Resolution 3 km (TL6718) ~3 km (C3072) * 3km 3.3 km ** 3 km 3.13 km * 3.125 km (Tc3199)
Grid Points13440x6720 (unred.)
59,609,088 (reduced)
**
6x3072x3072
56,623,10465,536,002 53,084,162
6x3072x3072
56,623,104 *61,440,000 **
51,572,436
(reduced)
Vertical Layers * 128 127 ** 127 *** 128 128 128 137
Time Step TBD
150 s (slow phys)
37 5 s (vertical,
fast phys)
37.5/10 s (horiz.
acoustic)
18 s (RK3 dynamics)
3 s (acoustic)
18 s (RK3 scalar
transport)
18 s 6 s **15 s (slow RK3 dyn.)
2 5 s (fast dyn.)120
Notes
* Unl es s noted,
la yers refers to the
number of la yers ,
not the number of
i nterfa ces between
la yers + top +
bottom
* Baseline configuration is
tentative, pending test
evaluation.
** Rough estimate for
reduced Gaussian grid
based on reduction factor
(0.66) of 13 km grid. This will
likely be revised after
further testing of accuracy
of spectral transform at
TL6718.
* True resolution is
average over equator
and/or from south to
north pole. For 13km,
max cell size (edge of
finite volume): 14.44
km, min: 10.21 km,
global avg: 12.05 km.
For 3.25 km, divide by
4.
** Favorable OpenMP
Performance
* Resolution refers to
mean cell-center
spacing on the mesh
** Subdivision of 60 km
mesh by factor of 5.
*** Following the FV3
configuration, we will
use 127 levels where
density,
theta and horizontal
momentum are defined
(on our Lorenz-grid
vertical
discretization) and 128
levels for w (that
includes both the lower
boundary and the
model top "lid").
* Generated by 6
bisections followed
by 2 trisections.
Distances between
neighbors: 13.367
average, 12.245
min., 14.397 max..
Maximum ratio of
neighboring grid
point distances:
1.17577
** Generated by 8
bisections followed
by 2 trisections.
Distances between
neighbors: 3.3417
average, 3.060 min.,
3.601 max..
Maximum ratio of
neighboring grid
point distances:
1.1765.
* B-grid mass points
** For fast modes and
advection of basic
model variables. Time
step for tracers is
longer by 2x.
* Resolution refers to the
representative nodal
spacing in the element
measured as the
midpoint between the
minimum and mean
nodal spacing and
averaged over the globe.
** Horizontal grid points
is six faces of cube times
number of elements per
face times polynomial
order squared.
* Hydrostatic
The Tc799 cubic grid has
the same number of grid
columns as a TL1599
linear grid.
While the Tc3199 cubic
grid has the same
number of grid columns
as a TL6399 linear grid.
No
min
ally
13
kmN
om
inal
ly 3
km
13
AVEC report: NGGPS Level-1 Benchmark and software evaluation
Generalized vertical coordinate
• First version of GSM used sigma coordinate, then move to hybrid sigma-pressure coordinate
• A dynamics system can be used for sigma, sigma-pressure, and sigma-theta, sigma-theta-pressure etc (Juang 2011), instead of only sigma coordinate or hybrid sigma-p coordinates
• The same model dynamics to be easily compared for different coordinates
3 June 2016 TWPGFS2016, H. Juang 12
June 3, 2016 Hann-Ming Henry Juang 13
From the ideal-gas law for individual gas as
p = rRT
pi = riRiT
p = riRii=1
N
å T = rrirRi
i=1
N
å T = rRT we havethrough
R = qiRii=1
N
å
qi =rir
let
rdCPT
dt-dp
dt= rQ
Then put them into internal equations, we have
we let
h =CPT as a prognostic variable, enthalpy.
we have
CP = qiCP ii=1
N
åso
14Black s: operational GFS Red t: sigma-theta GFS
15Black s: operational GFS Red t: sigma-theta GFS
16Black s: operational GFS Red t: sigma-theta GFS
17Black s: operational GFS Red t: sigma-theta GFS
18Black s: operational GFS Red t: sigma-theta GFS
19Black s: operational GFS Red t: sigma-theta GFS
Case Results - AC
PROW: gc s-pPROEW: gc s-p enthalpy
Root Mean Square error
Opr GFS vs WAM
22
64 layers150 layers
3 June 2016 TWPGFS2016, H. Juang
3 June 2016 TWPGFS2016, H. Juang 23
Example of T profileof 150 layers
WAM uses generalized hybrid coordinatewith enthalpy CpT as thermodynamicsvariables , where Cp is summation ofeach gases.
R Cp
O 519.674 1299.18
O2 259.837 918.096
O3 173.225 820.239
Dry air 296.803 1039.64
H2O 461.50 1846.00
3 June 2016 H. Juang
in do_dynamics_two_loop for spdmx at kdt= 40825spdmx(001:010)= 19. 20. 21. 23. 25. 26. 27. 28. 28. 28.spdmx(011:020)= 28. 27. 27. 27. 27. 28. 28. 28. 29. 30.spdmx(021:030)= 31. 33. 35. 37. 40. 42. 44. 46. 49. 53.spdmx(031:040)= 58. 61. 63. 63. 62. 60. 55. 47. 45. 44.spdmx(041:050)= 45. 45. 47. 49. 52. 55. 59. 62. 65. 68.spdmx(051:060)= 72. 76. 80. 84. 87. 90. 93. 95. 97. 98.spdmx(061:070)= 102. 110. 118. 127. 135. 143. 149. 153. 155. 152.spdmx(071:080)= 147. 145. 142. 138. 135. 132. 130. 126. 121. 119.spdmx(081:090)= 114. 112. 110. 106. 100. 95. 94. 90. 89. 89.spdmx(091:100)= 87. 82. 91. 95. 99. 97. 104. 100. 111. 120.spdmx(101:110)= 125. 133. 148. 167. 172. 164. 159. 160. 147. 124.spdmx(111:120)= 117. 125. 133. 138. 137. 157. 183. 202. 220. 243.spdmx(121:130)= 269. 297. 319. 338. 355. 368. 378. 386. 392. 396.spdmx(131:140)= 399. 402. 404. 405. 406. 407. 408. 409. 410. 410.spdmx(141:150)= 411. 412. 412. 413. 413. 414. 414. 415. 415. 418.
Maxima wind (m/s) at NCEP GFS 150 layers WAM
24
3 June 2016 TWPGFS2016, H. Juang 25
WAM Euleriandt=180sec
T62L150
3 June 2016 TWPGFS2016, H. Juang 26
WAM NDSLdt=400sec
Non-iteration and Dimensional-split Semi-Lagrangian advection method
• No need to iteratively find departure points • Dimensional split with MPI transpose, no need of
halo, so avoid the decision of halo side • Positive defined in interpolation, so no q<0• No polar grid needed • Option to have mass conservation• Based on generalized vertical coordinate enthalpy
system
June 3, 2016 Henry Juang 27
June 3, 2016 Hann-Ming Henry Juang 28
72h fcst specific humidity
at model layer 40
control
ndsl
dt=600
dt=1800
June 3, 2016 Hann-Ming Henry Juang 29
24hr fcst cloud water
at model layer 30
control
ndslmcpd
June 3, 2016 Hann-Ming Henry Juang 30
T574 GC Eulerian
T574 Opr EulerianTL878 NDSL
TL878 NDSL
dt=120dt=600
June 3, 2016 Hann-Ming Henry Juang 31
Day 1
Day 5
Opr T574 EulerianTL878 NDSL
dt=120dt=600
June 3, 2016 Hann-Ming Henry Juang 32
Day 1
Day 5
Opr T574 EulerianNDSL TL878
dt=120dt=600
Anomaly correlation- Day 5 HGT
Leads to higher and higher resolutionfor semi-Lagrangian vs Eulerian comparison
Examine Legendre coefficient
• We found Legendre polynomial function in sp lib and model have problem of spherical harmonic transform while using high truncation, say T1000 and above
• X-number technique fixes the problem• Furthermore, increase precision only in
computing Gaussian weight helps accuracy by order of 3
• These steps provide necessities to implement TL1534 with enhanced and corrected spectral transform
June 3, 2016 Henry Juang 34
June 3, 2016 Henry Juang 35
0.0
underflow overflow
2-960 2960
Machine representable real number
X number takes care of under and over flow by xB**I with B=2**960, soxB**0 is machine representable real number but xB**(-1) represents underflowAnd xB**(+1) represents overflow. The details in Fukushima 2011
Implement X-number into SP lib and GSM model dynamics
xB**ix time yB**iy, should be equal to x*yB**(ix+iy), but we should take care underflow by x*y, to do so, consider normalize of any X-number into the rangeof 2**(-480) and 2**(480). If x and y are normalized, then x*y will not be over-or under-flow.
Normalized range
June 3, 2016 Henry Juang 36
June 3, 2016 Henry Juang 37
June 3, 2016 Henry Juang 38
June 3, 2016 Henry Juang 39
June 3, 2016 Henry Juang 40
3 June 2016 TWPGFS2016, H. Juang 41
AVEC Report:NGGPS Level-1 Benchmarks and Software Evaluation
3 June 2016 TWPGFS2016, H. Juang 42
3 June 2016 TWPGFS2016, H. Juang 43
Full Dimensional MPP
• Current NCEP GSM has multi-threading by OpenMP and one dimensional MPI
• The limitation of OpenMP may be 12 and 1DMPI based on half of spectral truncation
• For 2D MPI, it is limited to spectral truncation and vertical layer number.
• Extend to 3DMPI with MPI-FFT and MPI-Legendre transform
• Invest halo or transpose for dimensional dependency computation
3 June 2016 TWPGFS2016, H. Juang 44
1D MPIGlobal communication
2D MPIGroup communication
June 3, 2016 13th on the use of hpc in meteorology 47
0 360
Halo Exchange for opr SL
Extra memory is required,which may be as huge
as computing grid while number of MPP cpu increases.
1D
2D
halo
June 3, 2016 13th on the use of hpc in meteorology 48
np
sp
np
0 360
np
sp
.
.
.
0 . . . . 180
<=>
Transpose for NDSL
No halo is requiredNo increasing memory with
Increasing number of PE(cpu)
3 June 2016 TWPGFS2016, H. Juang 49
Black grouped with red ones forMPI FFT without transpose
Black grouped with yellow ones forMPI Legendre w/o transpose
Black grouped with blue ones forregrouping into vertical formodel physics and vertical dependentmodel dynamics computations
So spectral transform will be no transpose, only vertical needs….One side MPI ....
3D MPI
Comparison
• Using TL1534 L64 as example
• 1DMPI+12 threads
=> We have 767 MPI 12 thread = 9,204 cpus
• 2DMPI + 12 threads
=> We have 767*64 MPI 12 thread = 589,056 cpus
• 3DMPI + 12 threads
=> 767*64*1534 MPI 12 threads= 903,611,904 cpus
3 June 2016 TWPGFS2016, H. Juang 50
Experimental cubic truncation
• Based on ECMWF IFS recent experiments, they have implemented cubic truncation octahedral reduced Gaussian grid for IFS Tco1279 ( 9 km )
• For semi-Lagrangian, we used to have linear truncation, for Eulerian, we have quadratic truncation to avoid aliasing
• Linear truncation resolves up to 2dx wave, quadratic to 3dx, and cubic to 4dx.
• Use cubic truncation with much less diffusion and without de-aliasing filter
• Octahedral grid is for future spectral/grid hybrid system and requires different FFT package (FFTW)
3 June 2016 TWPGFS2016, H. Juang 51
3 June 2016 TWPGFS2016, H. Juang 52
ECMWF Newsletter#147 spring2016
Implement Cycle 41r2Move from linear truncation reduced gridTo cubic truncation octahedral reduced grid
3 June 2016 TWPGFS2016, H. Juang 53
IFS octahedral reduced grid, more regular than spherical reduced grid, they arepreparing for next generation of hybrid spectral/grid-point configuration.
3 June 2016 TWPGFS2016, H. Juang 54
More resolving small scales
More efficient in computing time
ECMWF Newsletter#147 spring2016
3 June 2016 TWPGFS2016, H. Juang 55
IFS experimental results
ECMWF Newsletter #146 winter2015/2016
More small scale waves in TcBest mass conservation in Tc
3 June 2016 TWPGFS2016, H. Juang 56
ECMWF Newsletter#147 spring2016
Deep Atmos vs non-Hydro• From Deep atmosphere, we require r changes with time,
thus we need dw/dt equation• And we need full curvature and Coriolis force terms to
satisfy conservation• Thus, based on conservation requirement, a deep
atmospheric dynamic is a non-hydrostatic dynamic. A non-hydrostatic dynamics can be shallow or deep atmospheric dynamics.
• Both r and vertical components of curvature and Coriolisforce should be considered in deep atmosphere; and should not be considered in shallow atmosphere. (see Juang 2014 NCEP ON#477)
• Derive deep atmospheric system as the same form as a shallow atmospheric system with options for easy implementation
3 June 2016 TWPGFS2016, H. Juang 57
du*
dt+u*w
r - fsv
* + fc*w +
kh
p
1
r
¶ p
¶l-
¶ p
¶z
¶z
¶r
¶r
¶l
æ
èç
ö
ø÷ = Fu
dv*
dt +v*w
r + fsu
* +m2 s*2
rsinf +
kh
p
1
r
¶ p
¶j-
¶ p
¶z
¶z
¶r
¶r
¶j
æ
èç
ö
ø÷ = Fv
dw
dt -m2 s
*2
r -m2 fc
*u* +kh
p
¶p
¶z
¶z
¶r +g = Fw
dh
dt-
kh
p
dp
dt= Fh
¶r*
¶t+m2
¶r* u*
r¶l
+m2
¶r* v*
r¶j
+¶r* z
·
¶z= Fr
*
dqi
dt= Fqi
p = rkh
d()
dt=
¶()
¶t+ l
· ¶()
¶l+j
· ¶()
¶j+z
· ¶()
¶z=
¶()
¶t+m2u* ¶()
r¶l+m2v* ¶()
r¶j+z
· ¶()
¶z ; r* = r
r2
a2
¶r
¶z
fs = 2Wsinf ; fc* = 2Wcos2 f ; g = g(r) ; k =
R
CP ; g =
CP
CV ; s*2
= u*2
+ v*2
where
3 June 2016 TWPGFS2016, H. Juang 58
Deep Atmos Dyn
in spherical mapping
& generalized coordinates
Staniforth and Wood (2003)Juang (2014) NCEP Office Note
To derive deep-atmosphere continuity equation into the same
form as shallow atmosphere
We define a coordinate pressure as hydrostatic one based on
the coordinate density as
Put it into deep-atmosphere continuity equation
We have
¶r*
¶t+m2
¶r* u*
r¶l
+m2
¶r* v*
r¶j
+¶r* z
·
¶z= 0
3 June 2016 TWPGFS2016, H. Juang 59
where g0 is constant
In deep atmosphere, horizontal wind with Gaussian weighting
Modify continuity equation
We have
3 June 2016 TWPGFS2016, H. Juang 60
u* = rcos2 fdl
dt= ucosf
We can define, height-weighted horizontal wind as
v* = rcosfdf
dt= vcosf
3 June 2016 TWPGFS2016, H. Juang 61
Since and r* > 0
Thus is monotone with vertical coordinate
We can use it for coordinate definition
For opr compatibility, we use
and relation with height as
because r* = rr2
a2
¶r
¶z
let
3 June 2016 TWPGFS2016, H. Juang 62
w =r2
a2w
so rk2wk =
1
2r2
k+1wk+1 +1
2r2
kwk
we have wk =1
2wk+1 +
1
2wk
rk3 =
1
2r3
k+1 +1
2rk
3
From angular momentum principle, we have relationbetween model layer and model level as
so vertical momentum eq can be written as
with g =a2
r2g0
¶ p
¶1
r
= rg0a2
Then the hydrostatic means
3 June 2016 TWPGFS2016, H. Juang 63
In summary, we have following
So prognostic variables are
dh
dt-
kh
p
dp
dt= 0 and
where e =r
a
Summary• NCEP GSM may be an “old” model, but we are using
GSM2010s not GSM1980s.
• Current operational GSM is TL1534 (13km) semi-Lagrangian two-time level scheme with reduced spherical Gaussian grid and refined spectral transform.
• Current experimental GSM has options to be as Whole Atmospheric Model (WAM) with enthalpy and NDSL.
• Further speed up in computational wall time by multi-threading and full dimension MPI, maybe 2D MPI is more than enough.
• Experimental cubic and/or quadratic truncations with spherical reduced Gaussian grid instead of linear truncation for semi-Lagrangian, and with less diffusion.
• Developing GSM Deep-Atmospheric model beyond nonhydrostatic system with shallowness options for NGGPS.
3 June 2016 TWPGFS2016, H. Juang 64