A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS...

A PERFORMANCE EVALUATION OF THE ETA - CMAQ

AIR QUALITY FORECAST MODEL

FOR THE SUMMER OF 2004

CMAS WorkshopChapel Hill, NC

20 October, 2004

Contributors

Brian Eder Daiwen KangTanya Otte Hsiu-mu Lin Jon Pleim Shaocai YuGeorge Pouliot STCKen SchereJeff YoungASMDARL, NOAA

All: On assignment to NERL, U.S. Environmental Protection Agency

Disclaimer: “The research presented here was performed under the Memorandum of Understanding between the U.S. Environmental Protection Agency (EPA) and the U.S. Department of Commerce's National Oceanic and Atmospheric Administration (NOAA) and under agreement number DW13921548. Although it has been reviewed by EPA and NOAA and approved for publication, it does not necessarily reflect their policies or views.

Forecast Configuration

- Eta Meteorology- CBIV Mechanism- SMOKE Emissions (Offline)- 12 km grid resolution - 22 Vertical Layers- 48 Hr. Forecast (12Z Init.)

Simulation Period

- 1 May – 30 September 2004- 1 June – 30 September 2004 (presented here)

Domain

Models-3 CMAQ

Lambert Conformal Projection

142

166

This performance evaluation used:

Hourly O3 (ppb) from EPA’s

AIRNOW network:

- over 600 stations (mostly urban)

- for a four month period (J, J, A, S)

A suite of statistical metrics for:

- discrete forecasts (MB, NMB, RMSE, NME)

- categorical forecasts (A, B, POD, FAR, CSI)

of hourly, max. 1-hr and max. 8-hr O3 (presented here).

Discrete Forecast / Evaluation

[Observed] [Observed] versus versus [Forecast] [Forecast]

Statistics

- Summary

- Biases MB

NMB

- Errors

N M EM odel O bs

O bs

N

N

1

1

100%( )

R M SE M odel O bsN

N

1 2

1

0 5

( )

.

Category Forecast / Evaluation

Observed Exceedances, Non-ExceedancesObserved Exceedances, Non-Exceedances versusversus

Forecast Exceedances, Non-ExceedancesForecast Exceedances, Non-Exceedances

a b

c d

Fo

reca

st E

xcee

danc

e

N

o

Yes

No YesObserved Exceedance

a b

c d

Ab c

a b c d%

100

Ba b

b d

FARa

a b%

100

C SIb

a b d

100%

PO Db

b d%

100

Monthly summaries:

- June, July, August, September

- Both Discrete and Categorical evaluation

Daily summaries:

- August 4 -12, Discrete evaluation

- Examine performance over a full synoptic cycle

- Illustrating both good and poor model performance

Forecast / Evaluation

Meteorological Conditions for the Summer (J, J, A) 2004

Meteorological Conditions for the Summer (J, J, A) 2004

Temperature Precipitation

Monthly Summaries

Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

46.1 53.9 0.51 7.8 16.9 14.1 24.0

A B POD FAR CSI a=40 b=19

99.5 0.79 25.3 67.8 16.5 c=18274 d=56

a b

c d

June Summary

Correlation Mean Bias (ppb)

NMB (%) NME (%)

June Summary

Domain Wide Daily Max. 8- Hour Means

0

10

20

30

40

50

60

70

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

June

(pp

b)

OBS_MEAN MOD_MEAN

June Summary

1 15 30

June Summary

Bias

Error

1 15 30

Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

47.4 57.3 0.55 9.9 20.8 16.2 27.2


98.0 1.9 50.5 74.1 20.7 c=18258 d=93

July Summary

Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

43.4 55.1 0.62 11.7 27.1 16.5 30.9


98.7 5.2 39.6 92.4 6.8 c=19000 d=29

August Summary

Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

41.7 52.8 0.65 11.1 26.7 15.6 30.6


99.52 4.50 6.25 98.61 1.15 c=18018 d=15

September Summary

Daily Summaries

Synoptic Scale Meteorology 4 August

.01 0.5 1.0 2.0 3.0

20:15

http://www.hpc.ncep.noaa.gov/dailywxmap/dwm_minmax_20040805.html

http://www.hpc.ncep.noaa.gov/dailywxmap/dwm_prcn_20040804.html



http://www.hpc.ncep.noaa.gov/dailywxmap/dwm_stnplot_20040804.html

Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

44.1

(58.5)

55.8

(62.1)

0.41

(0.27)

11.7

(3.5)

26.5

(6.0)

17.9

(13.1)

32.8

(17.3)

48 Hour Max. 8-Hour Ozone Forecast Init: 3 Aug (12 Z) Valid: 4 Aug TP (04 Z to 03 Z)

Model performance is poorer alongthe cold front where cloud cover andheavy rain dominate


.01 0.5 1.0 2.0 3.0

20:15






Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

43.8

(58.2)

51.6

(63.0)

0.73

(0.66)

7.8

(4.9)

17.9

(8.4)

12.3

(10.4)

22.3

(14.5)


Model performance is still poor alongthe cold front where cloud cover andheavy rain dominate


.01 0.5 1.0 2.0 3.0

20:15






Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

37.8

(52.6)

44.3

(55.3)

0.82

(0.67)

6.4

(2.6)

17.0

(5.0)

8.7

(6.9)

18.9

(10.1)


Model performance is very good throughoutthe domain, except for sections of NewEngland and southeast Ontario, where cloudcover is greatest.


.01 0.5 1.0 2.0 3.0

20:15






Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

41.2

(50.1)

45.9

(50.7)

0.79

(0.71)

4.7

(0.6)

11.5

(1.3)

8.6

(5.3)

16.2

(8.2)


Model performance is again very good, except for sections of New England, southeastOntario and Wisconsin, where cloud cover is greatest.


.01 0.5 1.0 2.0 3.0

20:15






Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

48.7

(54.8)

52.0

(55.2)

0.75

(0.62)

3.3

(0.4)

6.8

(0.7)

9.6

(6.6)

14.5

(9.5)


Model continues to perform well as thecP airmass undergoes modification. Warmertemperatures allow for higher ozone values (mean observed up to 48.7). New Englandstill has more cloud cover.


.01 0.5 1.0 2.0 3.0

20:15





Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

52.9

(57.1)

55.2

()58.1

0.78

(0.61)

2.3

(1.0)

4.4

(1.8)

8.0

(7.4)

11.6

(9.8)


Good model performance continues, as thecP airmass continues to warm. Highestozone values (52.9) observed on this day. Cloud cover in New England dissipates resulting in better model performance there.


.01 0.5 1.0 2.0 3.0

20:15




Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

49.4

(58.4)

57.7

(60.8)

0.69

(0.59)

8.6

(2.4)

17.4

(4.2)

15.6

(9.4)

24.5

(12.6)


Model performance degrades somewhat as cloud cover and precipitation increases overthe domain, most notably with a cold frontapproaching from the Great Lakes and a low forming over south


.01 0.5 1.0 2.0 3.0

20:15





Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

42.2

(51.8)

54.9

(60.7)

0.71

(0.33)

12.7

(9.0)

30.2

(17.3)

15.7

(12.7)

31.7

(19.3)


Model performance continues to suffer over the Great Lakes area as cloud cover, precipitation and very cool temperatures dominate this region.


.01 0.5 1.0 2.0 3.0

20:15




Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

32.5

(44.9)

51.1

(56.6)

0.38

(0.18)

18.6

(11.7)

57.4

(26.1)

21.0

(14.1)

58.0

(28.1)


Model performance at its worst as extensive cloud cover, precipitation and very cool temperatures dominate the entire domain.

Monthly Summary Obs

Mean

Model

Meanr

MB

(ppb)

NMB

(%)

RMSE

(ppb)

NME

(%)

June 46.1 53.9 0.51 7.8 16.9 14.1 24.0

July 47.4 57.3 0.55 9.9 20.8 16.2 27.2

August 43.4 55.1 0.62 11.7 27.1 16.5 30.9

September 41.7 52.8 0.65 11.1 26.7 15.6 30.6

Summary

Model performed reasonably well over the four month period, despiteanomalously cool and wet conditions.

Performance was closely tied to meteorology:

M very good performance when skies were generallyclear and void of precipitation,

M poorer when cloud cover and precipitation occurred.

A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS...

Documents

Transcript of A PERFORMANCE EVALUATION OF THE ETA - CMAQ AIR QUALITY FORECAST MODEL FOR THE SUMMER OF 2004 CMAS...