Improving ECMWF’s IFS model - · PDF fileEUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER...

© ECMWF July 27, 2016

Improving ECMWF’s IFS modelby Nils Wedi

European Centre for Medium Range Weather Forecasts

[email protected]

Anna Agusti-Panareda, Gianpaolo Balsamo, Peter Bauer, Peter Bechtold, Willem Deconinck, Mikhail Diamantakis, Mats Hamrud, Christian Kuehnlein, Martin Leutbecher, Sarah-Jane Lock,Sylvie Malardel, Kristian Mogensen, George Mozdzynski, Pirkka Ollinaho, Irina Sandu, Piotr Smolarkiewicz, and many other colleagues …

EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014

Fit for the future …

• Research and development into

– physics & dynamics coupling

– quantifying uncertainty

– Earth-System complexity

– novel and adaptive numerical methods

• A flexible support infrastructure for European NWP science and services to

– maximise energy-efficiency

– assure the best possible use of the available computing resources

2


Affordability – the art and cost of computing

3

50 member ensemble

1 member

2015/6 2025

M€ electricity/year

Technology advances may move the energy scale by one order of magnitude, so we need to be ready! (Aurora 180PF, HPCwire April 2015)

(Bauer et al, 2015)

Hard limit for entire HPC ~20MW ?


Big Data challenge: 3Vs

4

Variety: Increase in products

Velocity: Exponentially growing data archive: 1995: 14 Tbytes / year 2015: 100 Tbytes / day

Volume:Satellite observations 1996 – 2018 (projected)

• high Volume, high Velocity, high Variety


Is the global spectral transform model dead ?

5

Technology applied at ECMWF for 30+ years …


SISL-semi-implicit semi-Lagrangian

6

Grid-point space-semi-Lagrangian advection-physical parametrizations-products of terms

Fourier space

Spectral space-horizontal gradients-semi-implicit calculations -horizontal diffusion

FFT

LT

Inverse FFT

Inverse LT

Fourier space

FFT: Fast Fourier Transform, LT: Legendre Transform

No grid-staggering of prognostic variables


MPI communication cost at large core counts …

230K cores


Application performance XC30

8

1.14

0.51

0.95 0.91

1.86

0.650.58

1.41

SEMI-LAGRANGIAN FFT (G2L, L2G) LEGENDRE (L2M, M2L) INSIDE SPECTRAL (S2M,M2S)

IFS data communication rates

rate 1279 (Tb/s) rate 1999 (Tb/s)

TCo1279 on 360 nodes; TCo1999 on 720 nodes ; dt=450s; 48h forecast

Alltoallv

Alltoallv

MPI_send/recv

MPI_send/recv


Time-to-solution at 13km

9

(Michalakes et al, NGGPS AVEC report, 2015)

IFS

use FV discretization


Scaling efficiency at 3km

10

IFS

(Michalakes et al, NGGPS AVEC report, 2015)


Time-to-solution 3km

11

(adapted from Michalakes et al, NGGPS AVEC report, 2015)

IFS

Operational need!


RAPS14 performance

12

68358729

4433

1539

564 490328

661

190

4

8

16

64 64

128 128 128

256

1

2

4

8

16

32

64

128

256

1

2

4

8

16

32

64

128

256

512

1024

2048

4096

8192

16384

TL95 TL159 TL255 TL511 TL799 TL1023 TL1279 TCo639 TCo1279

IFS model performance on CRAY XC‐30

Forecast Days per Day CRAY XC‐30 Nodes

ECMWF‐EN

SEMBLE

EC-E

arth

, AR

6

EC-E

arth

, AR

5

ECM

WF-

HR

ES

EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 13

Equal area (MPI) parallel decomposition (1600 tasks)

6,599,680 points x 137 levels at ~9km just below 1 billion points

A new grid for ECMWF


Compressible equations provide the most efficient solution as well as flexibility on the solution procedure in time

Finite-volume module (FVM)

O640 - Held-Suarez with real orography:

surface pressure after 50 days of simulation

[Courtesy Piotr Smolarkiewicz, Christian Kühnlein]

Equations beyond the hydrostatic system

EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 15

Further reading:

[Schulthess 2015]

Traditional science workflow

[Schulthess 2015]

Future science workflow

Energy efficient SCalable Algorithms for weather Prediction at Exascale www.hpc-escape.eu

Improving ECMWF’s IFS model - · PDF fileEUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER...

Documents

Transcript of Improving ECMWF’s IFS model - · PDF fileEUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER...