New Approach to Calculation of Atmospheric Model Physics ...

New Approach to Calculation of Atmospheric Model Physics: Accurate and FastNeural Network Emulation of Longwave Radiation in a Climate Model

VLADIMIR M. KRASNOPOLSKY

NOAA/NCEP/SAIC, Camp Springs, and Earth System Science Interdisciplinary Center, University of Maryland, College Park,College Park, Maryland

MICHAEL S. FOX-RABINOVITZ AND DMITRY V. CHALIKOV

Earth System Science Interdisciplinary Center, University of Maryland, College Park, College Park, Maryland

(Manuscript received 26 April 2004, in final form 25 October 2004)

ABSTRACT

A new approach based on a synergetic combination of statistical/machine learning and deterministicmodeling within atmospheric models is presented. The approach uses neural networks as a statistical ormachine learning technique for an accurate and fast emulation or statistical approximation of model physicsparameterizations. It is applied to development of an accurate and fast approximation of an atmosphericlongwave radiation parameterization for the NCAR Community Atmospheric Model, which is the mosttime consuming component of model physics. The developed neural network emulation is two orders ofmagnitude, 50–80 times, faster than the original parameterization. A comparison of the parallel 10-yrclimate simulations performed with the original parameterization and its neural network emulations con-firmed that these simulations produce almost identical results. The obtained results show the conceptual andpractical possibility of an efficient synergetic combination of deterministic and statistical learning compo-nents within an atmospheric climate or forecast model. A developmental framework and practical valida-tion criteria for neural network emulations of model physics components are outlined.

1. Introduction

Tremendous developments in numerical modelingand in computing capabilities during the last decadescontributed dramatically to the scientific and practicalsignificance of interdisciplinary climate, climate change,and weather prediction. One of the main problems ofimplementing the best atmospheric, ocean, and chem-istry models is the complexity of physical, chemical, andbiological processes involved. Parameterizations ofmodel physics are approximate subgrid-scale schemesbased on simplified 1D physical process equations andobservational data. Still, these parameterizations are sotime consuming, even for most powerful modern super-computers, that different kinds of additional simplifica-tions are usually applied. For example, some of theseparameterizations are calculated less frequently thanmodel dynamics (based on solving 3D geophysical fluiddynamics equations). This may negatively affect the ac-curacy of model physics calculations and its temporalconsistency with model dynamics and may lead to a

significant reduction of the accuracy of climate simula-tions and especially weather predictions.

Calculation of model physics in a typical moderate-resolution general circulation model (GCM), like theNational Center for Atmospheric Research (NCAR)Community Atmospheric Model (CAM-2) with T42(�3°) resolution and 26 vertical levels, takes about 70%of the total model computations. Higher uniform andvariable model resolutions (e.g., Fox-Rabinovitz et al.2001, 2002; Duffy et al. 2003) and more frequent modelphysics calculations, desirable for temporal consistencywith model dynamics, would increase the percentage tomore than 90%.

Such a situation is an important motivation for look-ing for new alternative numerical algorithms that pro-vide faster and, most importantly, very accurate ways ofcalculating model physics and chemistry. For example,a traditional statistical technique based on a represen-tation of the input/output relationship as an expansionof hierarchical correlated functions has been investi-gated in some atmospheric chemistry applications (seeSchoendorf et al. 2003, and references therein). How-ever, current climate and weather prediction modelsare complex and nonlinear and they require a muchhigher accuracy and better flexibility of approximation

Corresponding author address: Vladimir Krasnopolsky, 5200Auth Rd., Camp Springs, MD 20746-4304.E-mail: [email protected]

1370 M O N T H L Y W E A T H E R R E V I E W VOLUME 133

© 2005 American Meteorological Society

MWR2923

than those provided by traditional statistical tech-niques, which are appropriate for simpler applications.

During the last decade, neural network (NN) tech-niques have found a variety of applications in differentfields and, more specifically, in the accurate and fastmodeling of atmospheric radiative processes (Krasno-polsky 1997; Chevallier et al. 1998) and in satellite re-trieval procedures (Krasnopolsky et al. 1995; Krasno-polsky and Schiller 2003).

Two different NN-based approaches have been de-veloped to speed up calculations of model physics. Thefirst approach developed by Chevallier et al. (1998,2000) introduces NNs as a convenient tool in the tradi-tional framework of developing new, improved long-wave radiation parameterizations. Within his approach,NNs were applied to develop “a new generation of ra-diative transfer models” (Chevallier et al. 1998). A newNN-based longwave (LW) radiation parameterization(“NeuroFlux”) has been successfully developed for theEuropean Centre for Medium-Range Weather Fore-casts (ECMWF) model (Chevallier et al. 1998, 2000). Inthe NeuroFlux parameterization, the artificial neuralnetwork technique was used in conjunction with a clas-sical cloud approximation (the multilayer graybodymodel). As a result, in this new LW radiation param-eterization two NNs were used to compute upward anddownward clear-sky parts of LW fluxes, and 2 � K NNswere used to calculate upward and downward fluxes ateach of K cloud layers. Thus, the developed NeuroFluxis a battery of (2 � K � 2) NNs (40 NNs for K � 19)(Chevallier et al. 1998). Each NN has different size andis trained separately. NeuroFlux is 8 times faster thanthe previous parameterization (Chevallier et al. 1998,2000). NeuroFlux has been used operationally withinthe ECMWF four-dimensional variational data assimi-lation (4DVAR) system since October 2003.

A different new approach based on application ofNNs has been introduced to ocean models (Krasnopol-sky et al. 2000, 2002; Tolman et al. 2004) and as a pre-liminary study to an atmospheric (Krasnopolsky et al.2004) model, the NCAR single-column model with thephysics identical to that of NCAR CAM-2. This ap-proach introduces an accurate and fast method of cal-culating the atmospheric physics parameterizations bydeveloping NN emulations for existing model physicsparameterizations. In this approach, the entire param-eterization, as a single object (i.e., a continuous or al-most continuous mapping), is emulated by an NN. Inthis case, data used for the NN training are obtainedthrough a GCM simulation with the original parameter-ization. An NN emulation of a model physics param-eterization is a functional imitation of this parameter-ization, so that the results of model calculations withthe original parameterization and with its NN emula-tion are physically identical. The high quality of NNemulations is achieved because of the high accuracy ofapproximation of the original components. We prefer

to use the term NN emulation, not NN approximation,to avoid any possible confusion. The term parameter-ization already means a simplified approximation ofphysical processes. So, in the context of our approach,the term emulation means a complete functional imita-tion based on a precise mathematical/statistical ap-proximation (in a classic mathematical sense) of themodel physics parameterizations.

The key point is that NN emulation is developed herefor the existing parameterizations of atmospheric phys-ics. This allows us to preserve the integrity and level ofsophistication of the state-of-the-art physical param-eterizations of atmospheric processes. Because of thecapability of modern machine learning techniques toprovide an unprecedented accuracy in the approxima-tion of complex systems like model physics, our NNemulations of model physics parameterizations arepractically identical to original physical parameteriza-tions. In other words, the underlying idea of the ap-proach is not developing a new parameterization butrather emulating a parameterization already very care-fully tested and validated by its developers offline andthen online through experimentation with the entiremodel. It is achieved by using data for NN training thatare simulated by an atmospheric model run with theoriginal parameterization. Using model-simulated datafor NN training allows us to achieve an unprecedentedaccuracy in approximation because simulated data arefree of the problems typical in empirical data (problemslike high level of observational noise, sparse spatial andtemporal coverage, poor representation of extremeevents, etc.). In the context of our approach, the accu-racy and improved computational performance of NNemulations are always measured against the originalparameterization. It is noteworthy that the developedNN emulation has the same inputs and outputs as theoriginal parameterization and is used as its functionalsubstitute in the model.

We would like to clarify that the term “existing” pa-rameterizations means not only the ones currently usedin a model but also new, more sophisticated parameter-izations that are computationally prohibited in theiroriginal form, but will become computationally “afford-able” when using their accurate and computationallymore efficient NN emulations. Also, by “existing” wemean advanced parameterizations currently under de-velopment, like those of cloud physics in cloud-resolving models (also called superparameterizations).

The key objectives or questions of this study are thefollowing: (i) Are these emulations accurate or closeenough to the original physical parameterizations sothat their use (instead of the original parameterization)allows us to preserve all the richness, integrity, anddetailed features of atmospheric physical processes?In other words, is the NN emulation a precise emula-tion of the original physical parameterization? (ii) Arethese emulations fast enough to significantly accelerate

MAY 2005 K R A S N O P O L S K Y E T A L . 1371

model physics calculations? (iii) Are these statistical/machine learning components able to successfully co-exist or be compatible with the deterministic compo-nents of climate models, so that their combination canbe efficiently used for accurate and fast climate simu-lations without any negative impacts on their quality?(iv) Is there a real/productive synergy here? In otherwords, does this new combination of deterministic andstatistical learning approaches lead to new opportuni-ties in climate simulation and weather prediction? Anadditional objective of this study is to outline a devel-opmental framework and practical criteria for the de-velopment and validation of NN emulations for modelphysics and chemistry components, which could beused as guidelines in the follow-up developments of NNemulations for other components of model physics andchemistry.

More specifically, in this study we apply the NN ap-proach to approximating the LW radiation parameter-ization in the NCAR CAM-2 (e.g., see the special issueof Journal of Climate, 1998, vol. 11, no. 6). Longwaveradiation parameterization has been selected for thisfirst study because calculating the LW [and shortwave(SW)] radiation is the most time-consuming part of theatmospheric physics computations. For example, in theNCAR CAM-2 T42, total (LW and SW) radiation takesalmost 60% of the time required for the model physicscalculations.

The most efficient and convenient way of developingNN emulations for model physics components is devel-oping a single NN for a model physics parameteriza-tion. Such an approach was introduced in our prelimi-nary study (Krasnopolsky et al. 2004) wherein we de-veloped NN emulation for the NCAR LW radiationscheme in the framework of the NCAR single-columnmodel. This study showed the feasibility of the ap-proach. In the current study, we extend the developedapproach to build an NN emulation for the NCARCAM-2 LW radiation parameterization. We also per-formed and analyzed past climate simulations using theNN emulation. A more detailed discussion of its impacton climate simulation, with a comprehensive analysis ofclimate characteristics and the adjustment of NN emu-lations to account for climate change, goes beyond thescope of this paper and will be presented later.

It is noteworthy that the initial motivation for thisstudy came from the authors’ discussion of the possi-bility of an effective calculation of model physics on aglobal uniform fine resolution grid for stretched-gridGCMs, instead of calculating model physics on an in-termediate, global uniform resolution grid or directlyon a stretched grid (e.g., Fox-Rabinovitz et al. 2001,2002). Calculation of model physics on a global uniformfine resolution grid is feasible only when using themuch more computationally efficient NN emulations.These issues will be explored in a separate study.

Neural network emulations of model physics arebased on the fact that any parameterization of model

physics can be considered as a continuous or almostcontinuous (with finite discontinuities) mapping (inputvector versus output vector dependence), and NNs(multilayer perceptrons in our case) are a generic toolfor approximating such mappings (Cybenko 1989;Funahashi 1989; Hornik 1991; Chen and Chen 1995a,b;Attali and Pagès 1997). Neural networking is an ana-lytical approximation that uses a family of functionslike

yq � aq0 � �j�1

k

aqj��bj0 � �i�1

n

bjixi�; q � 1, 2, . . . , m,

�1�

where xi and yq are components of the input and outputvectors respectively, a and b are fitting parameters, and� is a so-called activation function (usually a hyperbolictangent), n and m are the numbers of inputs and out-puts, respectively, and k is the number of neurons in thehidden layer (see Ripley 1997 for more details).

In section 2, NN emulation of the NCAR CAM-2LW radiation parameterization is developed and ana-lyzed in terms of its accuracy and computational per-formance. In section 3, the use of the developed accu-rate and fast NN emulation within CAM-2 is validatedin terms of its impact on climate simulation. Section 4contains discussion and conclusions.

2. NN emulation of the NCAR CAM longwaveatmospheric radiation parameterization

a. General or background information

The major requirement for developing NN emula-tions for model physics is obtaining an extremely highaccuracy of NN emulations with practically zero biasesor systematic approximation errors (i.e., the systematicerrors that NN emulation introduces in addition to abias of the original parameterization). Providing verysmall additional bias is a necessary condition for assur-ing that additional errors are not accumulating duringlong-term climate simulations when using developedNN emulations. The choice of an optimal version of NNemulation is based on the accuracy, not the amount ofspeedup. All the obtained NN emulations guarantee avery significant speedup anyway.

The T42/26-level NCAR CAM-2 is used in this study.The function of the LW radiation parameterization inatmospheric GCMs is to calculate the heating fluxesand rates produced by LW radiation processes in theatmosphere. The complete description of the NCARCAM LW radiation parameterization is presented byCollins (2001) and Collins et al. (2002).

The input vectors for the NCAR CAM-2 LW radia-tion parameterization include 10 profiles [atmospheric


temperature, humidity, ozone, CO2, N2O, CH4, twochlorofluorocarbon (CFC) mixing ratios (the annualmean atmospheric mole fractions for halocarbons),pressure, and cloudiness] and one relevant surfacecharacteristic (upward LW flux at the surface). TheCAM-2 LW radiation parameterization output vectorsconsist of the profile of heating rates (HRs), and severalradiation fluxes including the outgoing LW radiationflux from the top layer of the model atmosphere [theoutgoing LW radiation (OLR)].

The NN emulation of the NCAR CAM-2 LW radia-tion parameterization has the same inputs [total 220;n � 220 in Eq. (1)] and the same outputs [total 33; m �33 in Eq. (1)] as the original NCAR CAM-2 LW radia-tion parameterization. We have developed severalNNs, all of which have one hidden layer with 20, 90,150, 200, 250, or 300 neurons [k � 20, 90, 150, 200, 250,300 in Eq. (1)]. Varying the number of hidden neuronsallows us to demonstrate the accuracy of approximationdependence on this parameter as well as its conver-gence, and as a result to provide the sufficient accuracyof approximation for the climate model.

The NCAR CAM-2 was run for 2 yr to generaterepresentative datasets. The first year of simulation wasdivided into two independent parts, each containing in-put/output vector combinations. The first part was usedfor training and the second was used for tests (controlof overfitting, control of a NN architecture, etc.). Thesecond year of simulation was used to create a valida-tion dataset, completely independent from both train-ing and test datasets. This third dataset was used forvalidations only. All approximation statistics presentedin the rest of this section are calculated using this inde-pendent validation dataset.

b. Bulk approximation error statistics

To ensure a high quality of representation of long-wave radiation processes, the accuracy of the NN emu-lations have been carefully investigated. Our NN emu-lations have been validated against the original NCARLW parameterization. For calculating the error statis-tics presented in Table 1 and the figures of this section,the original parameterization and its NN emulationhave been applied to the validation data. Two sets of

corresponding HR profiles have been generated. Thetotal and level bias (or mean error), total and levelRMSE, profile RMSE (PRMSE), and PRMSE pre-sented in Table 1 have been calculated as follows. Theoutputs of the original parameterization and NN emu-lation can be represented as Y(i, j) and YNN(i, j), cor-respondingly, where i � (lat, lon), i � 1, . . . , N is thehorizontal location of a vertical profile, N is the numberof horizontal grid points, and j � 1, . . . , L is the verticalindex, where L is the number of the vertical levels.

The mean difference, B (bias or a systematic error ofapproximation), and the root-mean-square difference(a root-mean-square error of approximation), RMSE,between the original parameterization and its NN emu-lation, are calculated as follows:

B �1

NL �i�1

N

�j�1

L

Y�i, j� � YNN�i, j��

RMSE � ��i�1

N

�j�1

L

Y�i, j� � YNN�i, j��2

NL. �2�

These two characteristics [Eqs. (2)] describe the accu-racy of the NN emulation integrated over the entire 4D(latitude, longitude, height, and time) dataset. Using aslight modification of Eqs. (2), bias and RMSE for mthvertical level can be calculated:

Bm �1N �

i�1

N

Y�i, m� � YNN�i, m��

RMSEm � ��i�1

N

Y�i, m� � YNN�i, m��2

N. �3�

The root-mean-square error can also be calculated foreach ith profile:

prmse�i� ��1L �

j�1

L

Y�i, j� � YNN�i, j��2. �4�

This error is a function of a horizontal location of theprofile. It can be used to calculate mean profile root-

TABLE 1. Statistics estimating the accuracy of HR (K day�1) calculations and computational performance for NCAR CAM-2 LWRusing NN emulation vs the original parameterization. Bias26 and RMSE26 (K day�1) correspond to the lowest layer. Total mean valuefor HRs � �1.36 K day�1 and standard deviation HR � 1.93 K day�1. For the lowest level (26th), mean value for HRs � �2.22 Kday�1 and HR � 5.57 K day�1. Corresponding statistics (K day�1) for the ECMWF model are shown just as a point of reference.

Model Bias RMSE PRMSE PRMSE Bias26 RMSE26 Performance

NCAR NN90 �4.0 � 10�4 0.33 0.27 0.19 �6.0 � 10�4 0.85 �80 times fasterNCAR NN150 1.0 � 10�4 0.28 0.23 0.16 �4.0 � 10�3 0.79 �50 times fasterNCAR NN200 5.0 � 10�5 0.26 0.21 0.15 2.0 � 10�3 0.71 �35 times fasterECMWF 0.05 0.1 0.3* 1.4* �8 times faster

* These statistics are for the lowest (31st) level in ECMWF model.


mean-square error, PRMSE, and its standard deviation,PRMSE, which are location independent:

PRMSE �1N �

i�1

N

prmse�i�

�PRMSE �� 1N � 1 �

i�1

N

prmse�i� � PRMSE�2 . �5�

Table 1 shows that the profile (PRMSE) error and theRMSE are close but not equal.

Figure 1 illustrates improvements (convergence) infour error statistics when the number of hidden neu-rons, k, increases from 20 to 300. These statistics areRMSE (2), RMSE26 [m � 26 in Eq.(3)], PRMSE, andPRMSE (5). The 26th, or lowest model level errors, arepresented because they are the maximum errors for theentire vertical profile. After a sizeable improvement fork increasing from 20 to 90, the errors practically reachconvergence for k � 150, and even for k � 90. All ourNN emulations have almost zero or negligible system-atic errors (biases), which (see Fig. 2) practically do notdepend on height and are indistinguishable from eachother for the figure scale. The rest of Fig. 2 shows thevertical profiles of RMSEs (2) for six developed NNs.For all NNs with the number of hidden neurons startingat 90, the RMSE (which is a purely random error in thecase of a zero bias) for the 10 upper levels does notexceed 0.2 K day�1, reaching 0.4 K day�1 at the 22dlevel. For the two lowest levels, RMSE is about 0.6–0.8

K day�1. Statistics (biases and RMSEs) for the lowest(26th) level are also included in Table 1. The naturalvariability () of the HRs is significantly higher (see inthe caption of Table 1) at the lowest levels than at thehigher ones. Hence, relative errors in the HRs calcu-lated with respect to the natural variability () are ap-proximately the same as errors at the higher levels.

Hereafter, NN20, NN90, NN150, NN200, NN250, andNN300 stand for NNs with 20, 90, 150, 200, 250, and 300hidden neurons. Because NN20 is less accurate, andNN250 and NN300 do not provide a significantly betteraccuracy than NN200, only three NNs, namely NN90,NN150, and NN200, are included in Table 1. Table 1shows bulk validation statistics for the accuracy of ap-proximation and computational performance for thethree best (in terms of accuracy and performance) de-veloped NN emulations. Here, statistics for comparisonof the ECMWF operational and NeuroFlux LW radia-tion parameterizations are also shown just as a refer-ence point. Mean values and standard deviations (HR)

FIG. 1. Convergence of four error statistics when the number ofhidden neurons increases from 20 to 300. RMSE [(2)]—solid;PRMSE [(5)]—dashed; PRMSE [(5)]—dotted; and RMSE26[(3)]—dotted–dashed lines.

FIG. 2. The vertical profiles of mean approximation errors ateach of 26 levels, level biases (the left vertical line), and levelRMSEs [(3)], for six developed NNs (NN20—thin solid; NN90—thick solid; NN150—dashed; NN200—dotted; NN250—dashed–dotted; NN300—dashed–double-dotted lines) all in K day�1.


of HRs are presented in the title of Table 1 for a betterunderstanding of relative errors.

In addition to this high approximation accuracy, ourNN emulation performs about 80–35 times faster (forNN90, NN150, and NN200, correspondingly) than theoriginal NCAR LW radiation parameterization. Table1 and Figs. 1 and 2 clearly demonstrate a systematicimprovement in approximation accuracy with increas-ing NN hidden layer size. Table 1 also demonstrates areciprocal reduction in performance gain, from 80 to 35times faster, than the original parameterization. Thisoffers an opportunity for the accuracy versus perfor-mance trade-off; however, as we mentioned earlier, inthis trade-off the key requirement, which allows thesuccessful, synergetic functioning of NN emulationwithin the model, is to preserve the accuracy and integ-rity of the description of the corresponding physicalprocess.

All three NNs (including NN90) provide an accurateapproximation of the original parameterization. Actu-ally, there is no difference between 1%, 2%, or 3% costof the original parameterization calculations withNN90, NN150, or NN200 (which are 80, 50, or 35 timesfaster). Therefore, we would recommend and use theaccurate and reliably converged version. Obviously, thefinal decision on the optimal version of the NN, whichhas to be implemented into the model, should be madebased on testing these NNs in a climate model (seesection 3). We would like to stress that the speedup isachieved for the NN emulation of the entire LW radia-tion scheme that includes calculations of optical prop-erties (emissivity and absorptivity), HRs, and radiativefluxes. Our speedup for the entire LW radiationscheme provides the opportunity of calculating ithourly, that is, as frequently as HRs and radiative fluxcalculations, at a very limited computational cost.

c. Detailed evaluation of approximation errors

Both the original parameterization and its NN emu-lation are complicated multidimensional objects (map-pings). In this case, calculating bulk statistics is not suf-ficient for evaluating the accuracy of the approxima-tion. We evaluated many different statistical metrics ofthe approximation accuracy, the most important ofwhich are shown in Figs. 1–6. Figure 3 shows the typicallevel statistics (for level number 20 from the top) basedon NN150 emulation. It contains the scatterplot Y ver-sus YNN (upper-left panel), the distribution of errors ordifferences Y(i, j) � YNN(i, j) (lower-left panel), andbias and RMSE as functions of HRs (the upper- andlower-right panels, correspondingly). The scatterplotcloud contains hundreds of thousands of points, whichare tightly concentrated along the diagonal, with onlyseveral points at the upper and lover ends located out-side the one sigma interval (marked by dotted lines).The error distribution is strongly picked about 0 Kday�1, and it is narrower than the normal distribution

with the same mean and standard deviation (dottedline), which indicates a lesser amount of larger errorsthan in the case of the normal error distribution. Similarbehavior takes place in the distribution of the prmseprofile rms errors [see Eq. (4)] shown in Fig. 4. Thisdistribution is also significantly sharper than the normalone with the same mean value and standard deviation.

The two right panels in Fig. 3 show both systematic(bias) and random (RMSE) approximation errors asfunctions of HRs. An increase in errors at the tails ofthe HR distribution (shown with the dashed line),where NN was exposed to an insufficient amount oftraining data, can be clearly seen. More training datashould be simulated in this part of the domain to im-prove the approximation accuracy there.

Figure 5 shows the absolute zonal mean bias (leftcolumn) and zonal mean RMSE (right column). Threerows correspond to three different NNs, namely NN90,NN150, and NN200, from top to bottom, correspond-ingly. When the accuracy of approximation increases(with increasing the number of hidden neurons in theNN), both zonal mean bias and RMSE decrease signifi-cantly. Comparing the top, middle, and bottom panels,we see that small areas with bias 0.01 K day�1 (leftcolumn) in the lower part of the atmosphere disappearcompletely. Also, the small areas of RMSE 0.25 Kday�1 (right column) disappear at the upper levels. Inthe lower part of the atmosphere, small areas withRMSE 1 K day�1 (right column) begin to disappearfor NN150, and even more so for NN200, and the areaswith RMSE 0.5 K day�1 are confined to just two smallspots located in the polar areas.

Figures 6a–c show three typical individual profileswith profile rms errors [prmse; Eq. (4)] close to theirmean [prmse; Eq. (5)]. Each of these profiles demon-strates a complicated vertical variability for original andNN emulation profiles that are very close to each other.There is an obvious convergence of the emulating pro-files (gray) to the original (black). The prmse for eachprofile systematically improves when the number ofneurons in the NN hidden layer increases from 90 to200. This convergence, however, is not uniform at somevertical levels. For example, at level 14 in Fig. 6a,NN150 (gray dashed line) is slightly closer to the origi-nal profile (black solid line) than NN200 (gray dottedline). It is remarkable that for all individual profiles allthe NN emulations are very close to the original pro-files. It shows a high uniform convergence and accuracyat the profile and even gridpoint level.

The analysis of approximation errors presentedabove shows that the NN technique is capable of pro-viding NN emulations with practically zero systematicerrors or biases, and small random errors. Moreover,the distributions of errors are usually narrower than thecorresponding normal distributions, which indicate asignificantly smaller amount of larger errors. An addi-tional analysis shows that larger errors are located in


the areas supported by a smaller amount of trainingdata. In these areas NN is forced to extrapolate. Theseareas should be enriched by additional simulated datato improve the accuracy of the NN emulation there.Namely, the original parameterization should be run inthis subdomain to generate more data.

3. Results of climate simulations

In assessing the impact of using an NN emulation ofthe LW radiation parameterization, the parallel NCARCAM-2 climate simulations were performed with theoriginal LW radiation parameterization (the controlrun) and with its NN emulations, which are described insection 2. The climate simulations have been run for 10

yr starting after the training and validation period (seesection 2), namely for years 3 through 12. All the com-parisons of the control and NN emulation runs pre-sented in this section are done by analyzing the time-(10 yr) mean differences between the results of differ-ent runs.

Preservation of time means of prognostic and diag-nostic fields is one of the most important/necessaryproperties in climate simulations. In the climate simu-lations performed with the original LWR parameteriza-tion and its NN emulations, the time-mean surfacepressure is almost precisely preserved. For example, forthe NN150 run there is a negligible difference of0.0001% between the NN and control runs (see Table2). Other time global means, some of which are also

FIG. 3. Typical level statistics (for the 20th from the top level, about 625 hPa). (upper left) The scatterplot Y(original HRs) vs YNN (NN150 HRs) both in K day�1. Large dark circles and associated bars show average in thebin and an error bar. (lower left) The distribution of errors or differences Y(i, j) � YNN(i, j) and the normaldistribution with the same mean and standard deviation (dotted line); horizontal axis corresponds to errors in Kday�1. (upper right) Bias and (lower right) RMSE (K day�1) as functions of HRs (K day�1) with event distribution(dashed lines).


presented in Table 2, show a profound similarity be-tween the simulations in terms of these global timemeans, with the differences usually within about0.03%–0.1% and not exceeding 0.1%–0.3%. Othersimulations (with NN90 and NN200) show similar re-sults.

Let us now consider some key simulated diagnosticand prognostic fields and their differences, produced inthe control and NN emulation runs. The time-meanvertical distributions of zonal means for LW radiationHRs (QRL), potential temperature (T), zonal wind(U), and specific humidity (Q) are presented in Figs.7–10. The control simulation (with the original LWRparameterization) is shown in (a); (b) and (c) are twosimulations, with NN90 and NN150 emulations, corre-spondingly; (d) and (e) are biases or deviations of theseNN90 and NN150 simulations from the control simula-tion or (b) – (a) and (c) – (a), correspondingly. There-fore, the biases are calculated against the control run.The general assessment of the presented field distribu-tions and their biases is as follows. The field distribu-tions for the control and NN emulation runs are veryclose, showing a striking similarity to each other thatcan be seen by the comparing (a) with (b) and (c) forFigs. 7–10. Such a profound pattern similarity is furtherconfirmed and quantified by the small biases shown in(d) and even smaller, almost negligible biases shown inpanel (e) of Figs. 7–10.

Now let us discuss the results in more detail. TheLWR HRs for the NN90 run (Fig. 7b) show four minorspots located within the 400–500-hPa layer that are notpresent in the control (Fig. 7a) and the NN150 (Fig. 7c)runs. These differences are more visible in the bias pat-tern of the NN90 run (Fig. 7d). Actually, small positiveand negative biases, in the 0.05–0.2 K day�1 range, arepresent in the troposphere. For some spots, within thelower-tropospheric 900–1000-hPa layer, the bias in-creases to 0.6–1.0 K day�1 (Fig. 7d). For the NN90 run,

bias can be reduced by extending training datasets forthe tails of the distribution as mentioned above in sec-tion 2. However, for the NN150 run (Fig. 7e), the biaspattern has cleared up significantly (bias is mostlywithin 0.01–0.05 K day�1) and contains just a few smallpositive and negative spots in the lower-troposphericlayer, including those with a maximum magnitude of0.2–0.4 K day�1 located near the poles. Bias in thestratospheric domain (above 100 hPa) is significantlysmaller than that of the tropospheric domain. For bothNN90 and NN150 runs, it is well below 0.05 K day�1,especially for the NN150 run (Figs. 7d and 7e), with theexception of polar areas near the top model levels forthe NN90 run, where bias reaches 0.1–0.15 K day�1

within the polar domain (Figs. 7d).The distributions of temperature, zonal wind, and

specific humidity for the control, NN90, and NN150runs are practically indistinguishable from each other[for Figs. 8–10, cf. (a), (b), and (c)]. The differencesbetween runs can be seen only in bias distributions [cf.(d) and (e) for Figs. 8–10]. Temperature bias for theNN90 run (Fig. 8d) is mostly limited by magnitude to0.5 K. It increases to a maximum of 1 K for polar do-mains within the 300–200-hPa layer, and for a few spotsat the �100 hPa level. In the middle-stratosphere polardomains, it increases to 1–1.5 K. Bias for the NN150 run(Fig. 8e) is largely reduced everywhere compared tothat of the NN90 run (Fig. 8d), mostly to 0.1–0.2 K bymagnitude, with the exception of a few very small spotswhere it approaches 0.5 K.

Zonal wind bias for the entire troposphere andstratosphere domain in the NN90 run (Fig. 9d) is small,mostly below 0.5 m s�1, and does not exceed 1 m s�1 bymagnitude. Bias is slightly larger within the small spotaround 25°S at 100 hPa, and it increases to 2.5 m s�1 forthe upper model level within the polar domains. For theNN150 run (Fig. 9e), bias is significantly reduced to0.1–0.2 m s�1, with a maximum magnitude of only 0.5m s�1 for a few small spots. It is noteworthy that me-ridional wind bias for the NN90 run (not shown) ismostly below 0.05 m s�1 and is within 0.1 m s�1 by mag-nitude. Only in the small areas around the tropicaltropopause does bias reach 0.1–0.15 m s�1, which doesnot affect the Hadley and Ferrell circulations. Bias forthe NN150 run is further reduced to 0.01–0.02 m s�1,with the exception of a few very small spots where itincreases to 0.05 m s�1.

Specific humidity bias in the NN90 run (Fig. 10d)shows a maximum of 0.3–0.4 g kg�1 around �700 hPain the equatorial domain. For the NN150 run (Fig. 10e),bias is significantly smaller, mostly 0.01–0.02 g kg�1 bymagnitude, with a maximum of 0.05 g kg�1 over acouple of small spots in the subtropical lower tropo-sphere.

The above comparison of biases for NN90 andNN150 runs [see (d) and (e) of Figs. 7–10] confirms thatincreasing the number of hidden neurons from 90 to150 leads to a measurable bias reduction that positively

FIG. 4. The distribution of the PRMSE [(4)]—profile rms errors(K day�1) and the normal distribution with the same mean andstandard deviation (dotted line).


affects the accuracy of the NN150 climate simulation interms of its profound similarity to the control simula-tion. Most importantly, biases for both NN90 andNN150 10-yr simulations do not accumulate over time.The LW radiation HRs obtained in these climate simu-lations maintain the level of approximation accuracyconsistent with that obtained in section 2.

4. Discussion and concluding remarks

In this study, we presented a new approach based ona synergetic combination of deterministic modeling anda machine learning technique within an atmosphericmodel. This approach uses neural networks as a statis-tical or machine learning technique to develop highly

FIG. 5. (left) Absolute zonal mean bias and (right) zonal mean rmse for (top) NN90, (middle) NN150, and (bottom) NN200.


accurate and fast approximations for model physicscomponents. The approach consists of four major steps:

1) Analysis of the structure and complexity of theoriginal parameterization for determining the topol-ogy (architecture) of the future NN emulation byspecifying all the inputs and outputs and selectingthe initial number of neurons in the NN hiddenlayer.

2) Generation of representative datasets for training,testing, and validation. This approach is based onusing data simulated during the GCM runs with theoriginal parameterization, which allows us to pro-duce NN emulations that are physically identical tothe original parameterizations. For weather predic-tion applications, the use of blended (simulated, as-similated, and observational) data for NN training

could be beneficial. To account for insufficient sam-pling for some events, it is possible to run the origi-nal parameterization offline and generate comple-mentary data to extend sampling. This offline simu-lation can also be used in the context of adjustingNN emulations for future climate change.

3) NN training. Several different versions of NNs,with different number of neurons in a hiddenlayer, should be trained to determine the optimalsize of the hidden layer, which provides the suffi-cient accuracy of approximation; several initializa-tion procedures and training algorithms should beapplied to assure that an optimal minimization isachieved.

4) Validation of trained NN emulation consisting oftwo steps. The first step is validation of the NN ap-proximation against the original parameterization

TABLE 2. Time and global means for mass (mean sea level pressure) and other model diagnostics for the NCAR CAM-2 climatesimulations with the original LWR parameterization and its NN emulation (NN150) and their differences (%).

FieldOriginal LWR

parameterization NN emulation Difference (%)

Mean sea level pressure (hPa) 1011.480 1011.481 0.0001Surface temperature (K) 289.003 289.001 0.0007Total precipitation (mm day�1) 2.275 2.273 0.09Total cloudiness (fractions 0.1 to 1.0) 0.607 0.609 0.3LWR heating rates (K day�1) �1.698 �1.700 0.1–OLR (W m�2) 234.43 234.63 0.08Latent heat flux (W m�2) 82.84 82.82 0.03

FIG. 6. Typical profiles with PRMSE [(4)] (a) above, (b) close to, and (c) below PRMSE [(5)]. Solid black—original profile; solidgray—its NN90 emulation; dashed gray—NN150 emulation; and dotted gray—NN200 emulation.


using an independent validation dataset. The secondvalidation step consists of performing comprehen-sive parallel model runs with the original parameter-ization (the control run) and the developed NNemulations.

In this study, we presented an NN emulation ofan atmospheric LW radiation parameterization usedin NCAR CAM-2. The LW radiation has been se-lected as the most time-consuming component ofthe NCAR model physics. We evaluated the accu-racy and computational performance of this NNemulation. The obtained results show the following:

(i) The conceptual and practical possibility of de-veloping an accurate NN emulation of modelphysics components, which preserves the integ-rity and all the detailed features of atmosphericphysical processes. The practical possibility ofexperimentations with NNs with hundreds ofinputs, hundreds of neurons in a hidden layer,and tens or more of outputs, using a trainingdataset with about 100 000 records. The effi-cient software for a standard workstation with-out any hardware acceleration has been devel-oped, allowing such experimentations.

(ii) These accurate NN emulations are very fast (upto 80 times) despite the large size of the corre-sponding NNs, so the significant speedup ofmodel physics calculations can be achievedwithout compromising its accuracy.

(iii) These statistical/machine learning techniquescan be successfully combined with determinis-tic climate model components so that their syn-ergy can be efficiently used for climate simula-tions without any negative impacts on simula-tion quality as it was shown by the presenteddecadal climate simulation.

(iv) This productive synergy or the new combina-tion of the state-of-the-art deterministic andstatistical learning approaches leads to new op-portunities in climate simulation and weatherprediction. For example, more accurate andmore sophisticated atmospheric parameteriza-tions, which may be currently computationallyprohibited because they are too time consum-ing even for most powerful supercomputersavailable, may exist or be developed in the fu-ture. After developing NN emulations for theseparameterizations they may become computa-tionally feasible.

The systematic error introduced by NN emulation isnegligible and does not accumulate over the model in-tegration in time. The random error for NN emulationis also small, as shown in section 2. The distributions oferror are usually narrower than the corresponding nor-mal distributions, which indicates a significantly smalleramount of larger errors. The application of this ap-proach allows us to accelerate the calculation of the LW

FIG. 7. Zonal and time- (10 yr) mean ver-tical distributions of instantaneous long-wave radiation (LWR) heating rates (QRL)for (a) the control NCAR CAM-2 simula-tion with the original LW radiation param-eterization; (b) NCAR CAM-2 simulationusing NN90 emulation of LW radiation pa-rameterization; (c) NCAR CAM-2 simula-tion using NN150 emulation of LW radia-tion parameterization; (d) bias or deviationof the NN90 simulation from the controlsimulation or (b) – (a); (e) bias or deviationof the NN150 simulation from the controlsimulation or (c) – (a). The contour inter-vals for (a), (b), and (c) are 0.5 K day�1 andfor (d) and (e) 0.05 K day�1.


radiation parameterization by about 50–80 times (ortakes only 1%–2% of the original parameterizationcomputation time) for NN150 and NN90, correspond-ingly, without compromising the accuracy and integrityof the original longwave radiation parameterization.

The impact of using NN emulation on climate simu-lation has been assessed by a comparison of some basicclimate characteristics of parallel NCAR CAM-2 simu-lations, calculated with the original LW radiation pa-rameterization and its NN emulations. The differencesbetween the simulations with the original LW radiationparameterization and its NN emulations appear to bevery small for simulated fields. The results obtainedshow that the NN emulation of the considered atmo-spheric LW radiation parameterization is accurate andprovides a significantly improved computational effi-ciency.

There are several important topics—like adjustmentsof NN emulations to account for climate change, anexplicit evaluation of NN emulation Jacobian, and deal-ing with giant NNs in the case of higher resolutions—that go beyond the scope of this study. However, thesetopics should and will be addressed in our follow-upinvestigations. In this methodological study, we dealtwith the past climate. In the follow-up efforts we willconsider different options (like, e.g., extensions of thetraining domain, using recurrent NNs, and adoptingsome control theory tools), which machine learningtechniques can provide, to adjust NN emulations to cli-mate change.

In Krasnopolsky et al. (2002) we demonstrated theacceptable quality of the NN emulation Jacobian formoderate-size NN emulations. For the large-size NNemulations, an explicit investigation of the quality ofthe NN emulation Jacobian should be conducted.When the Jacobian is a very highly dimensional object,a special study is desirable, like that of Chevallier andMahfouf (2001). In our case, the high accuracy of ap-proximation and interpolation demonstrated by ourNN emulation on an independent validation datasetand during the parallel run of NCAR CAM-2 demon-strates, although indirectly, the practically sufficient ac-curacy of Jacobians in terms of the considered applica-tion.

Increasing the model resolution will result in increas-ing the size of NNs. Training such NNs will becomemore and more time consuming. In our previous studies(Krasnopolsky et al. 2002; Tolman et al. 2005), we pro-posed a solution for this problem. Before applying anNN technique, inputs and outputs are decomposed us-ing EOFs (or another complete basis). Then NN is ap-plied to relate the coefficients of these decompositions.This approach allowed us to reduce the size of inputand output vectors (and the size of the NN emulation)by an order of magnitude.

The success of the approach introduced in this paperfor approximating the longwave radiation parameter-ization opens the opportunity for a complete open-minded reexamination of computations for all modelphysics components. The next logical steps would be

FIG. 8. Same as Fig. 7, but for tempera-ture. The contour intervals for (a), (b), and(c) are 5 K and for (d) and (e) 0.5 K.


FIG. 10. Same as Fig. 7, but for specifichumidity. The contour intervals for (a), (b),and (c) are 2 g kg�1 and for (d) and (e) 0.05g kg�1.

FIG. 9. Same as Fig. 7, but for zonal wind.The contour intervals for (a), (b), and (c)are 5 m s�1 and for (d) and (e) 0.5 m s�1.


developing NN emulations for the full atmospheric ra-diation (including shortwave) block and then cautiouslyapproaching the more nonlinear and, therefore, chal-lenging moisture physics block, including convection,cloudiness, and turbulence. Eventually, NN emulationsfor all the diabatic forcing components could be intro-duced. This, in turn, could potentially make an impor-tant positive impact on extensive experimentation withthe complex models needed to improve climate changeand variability assessments, as well as weather predic-tion. It should be emphasized that the results obtainedon the accuracy and efficiency of the NN emulationmay facilitate a collaborative effort (with model physicsscheme developers) to develop new, more sophisticatedparameterizations of model physics (superparameter-izations, e.g., cloud physics) that are now computation-ally prohibitive. This is also true for computationalbottlenecks in model dynamics like complicated solv-ers, iterations, transformations, inversions, etc.

Acknowledgments. The authors thank Drs. W. Col-lins, P. Rasch, J. Tribbia, S. Lord, D. B. Rao, and F.Chevallier for useful discussions and consultations. Theauthors thank Mr. R. Govindaraju for useful consulta-tions and Mr. A. Belochitsky for programming support.This study is based upon the work supported by theDepartment of Energy Grant ER63197-0007185.

REFERENCES

Attali, J.-G., and G. Pagès, 1997: Approximations of functions bya multilayer perceptron: A new approach. Neural Networks,6, 1069–1081.

Chen, T., and H. Chen, 1995a: Approximation capability to func-tions of several variables, nonlinear functionals and operatorsby radial basis function neural networks. Neural Networks, 6,904–910.

——, and ——, 1995b: Universal approximation to nonlinear op-erators by neural networks with arbitrary activation functionand its application to dynamical systems. Neural Networks, 6,911–917.

Chevallier, F., and J.-F. Mahfouf, 2001: Evaluation of the Jacobi-ans of infrared radiation models for variational data assimi-lation. J. Appl. Meteor., 40, 1445–1461.

——, F. Chéruy, N. A. Scott, and A. Chédin, 1998: A neuralnetwork approach for a fast and accurate computation oflongwave radiative budget. J. Appl. Meteor., 37, 1385–1397.

——, J.-J. Morcrette, F. Chéruy, and N. A. Scott, 2000: Use of aneural-network-based longwave radiative transfer scheme inthe EMCWF atmospheric model. Quart. J. Roy. Meteor. Soc.,126, 761–776.

Collins, W. D., 2001: Parameterization of generalized cloud over-lap for radiative calculations in general circulation models. J.Atmos. Sci., 58, 3224–3242.

——, J. K. Hackney, and D. P. Edwards, 2002: An updated pa-

rameterization for infrared emission and absorption by watervapor in the National Center for Atmospheric ResearchCommunity Atmosphere Model. J. Geophys. Res., 107, 4664,doi:10.1029/2001JD001365.

Cybenko, G., 1989: Approximation by superposition of sigmoidalfunctions, in mathematics of control. Signals Syst., 2, 303–314.

Duffy, P. B., B. Govindasamy, J. P. Iorio, J. Milovich, K. R. Sper-ber, K. E. Taylor, M. F. Wehner, and S. L. Thompson, 2003:High-resolution simulations of global climate. Part 1: Presentclimate. Climate Dyn., 21, 371–390.

Fox-Rabinovitz, M. S., L. L. Takacs, and R. C. Govindaraju, 2001:A variable resolution stretched grid GCM: Regional climatesimulation. Mon. Wea. Rev., 129, 453–469.

——, ——, and ——, 2002: A variable-resolution stretched-gridgeneral circulation model and data assimilation system withmultiple areas of interest: Studying the anomalous regionalclimate events of 1998. J. Geophys. Res., 107, 4768,doi:10.1029/2002JD002177.

Funahashi, K., 1989: On the approximate realization of continu-ous mappings by neural networks. Neural Networks, 2, 183–192.

Hornik, K., 1991: Approximation capabilities of multilayer feed-forward network. Neural Networks, 4, 251–257.

Krasnopolsky, V., 1997: A neural network forward model for di-rect assimilation of SSM/I brightness temperatures into at-mospheric models. Research Activities in Atmospheric andOceanic Modeling, CAS/JSC Working Group on NumericalExperimentation, Rep. 25, WMO Tech. Doc. 792, 1.29–1.30

——, and H. Schiller, 2003: Some neural network applications inenvironmental sciences. Part I: Forward and inverse prob-lems in satellite remote sensing. Neural Networks, 16, 321–334.

——, L. C. Breaker, and W. H. Gemmill, 1995: A neural networkas a nonlinear transfer function model for retrieving surfacewind speeds from the Special Sensor Microwave Imager. J.Geophys. Res., 100, 11 033–11 045.

——, D. V. Chalikov, and D. B. Rao, 2000: Application of neuralnetworks for efficient calculation of sea water density or sa-linity from the UNESCO equation of state. Proc. SecondConf. on Artificial Intelligence, Long Beach, CA, Amer. Me-teor. Soc., 27–30

——, ——, and H. L. Tolman, 2002: A neural network techniqueto improve computational efficiency of numerical oceanicmodels. Ocean Modell., 4, 363–383.

——, M. S. Fox-Rabinovitz, and D. V. Chalikov, 2004: Using neu-ral networks for fast and accurate approximation of the longwave radiation parameterization in the NCAR communityatmospheric model: Evaluation of computational perfor-mance and accuracy of approximation, Proc. 15th Symp. onGlobal Change and Climate Variations, Seattle, Washington,Amer. Meteor. Soc., CD-ROM, P1.20

Ripley, B. D., 1997: Pattern Recognition and Neural Networks.Cambridge University Press, 403 pp.

Schoendorf, J., H. Rabitz, and G. Li, 2003: A fast and accurateoperational model of ionospheric electron density. Geophys.Res. Lett., 30, 1492, doi:10.1029/2002GL016649.

Tolman, H. L., V. M. Krasnopolsky, and D. V. Chalikov, 2005:Neural network approximations for nonlinear interactions inwind wave spectra: Direct mapping for wind seas in deepwater. Ocean Modell., 8, 253–278.


New Approach to Calculation of Atmospheric Model Physics ...

Documents

Transcript of New Approach to Calculation of Atmospheric Model Physics ...