Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
Transcript of Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
1/26
GlobAEROSOL Data User Element
Algorithm Theoretical Basis Document
for the creation of a
Merged Global Aerosol Product
DRAFT Version 2.1
6 February 2007
R. Siddans, C.A. Poulsen, B. J. Kerridge
Space Science and Technology Department,
CCLRC Rutherford Appleton Laboratory, U.K.
G. E. Thomas, S. M. Dean, E. Carboni, R. G. Grainger
Atmospheric, Oceanic, and Planetary Physics,
Clarendon Laboratory, University of Oxford, U.K.
CCLRC Rutherford Appleton Laboratory
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
2/26
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
3/26
Contents
1 Document version history. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 WP Document. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
5 The Individual-Sensor Aerosol Products . . . . . . . . . . . . . . . . . . . . . . . 7
5.1 General characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
5.2 AATSR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
5.3 MERIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.4 SEVIRI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.5 ATSR-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
6 Deliverable products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
7 Definition of the GlobAerosol grid . . . . . . . . . . . . . . . . . . . . . . . . . . 10
7.1 Gridding schemes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
7.2 Selected grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
8 The Merging algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
8.1 Temporal characteristics of the IGAPs and MGAP . . . . . . . . . . . . . 15
9 Merging co-located retrievals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159.1 General approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
9.2 Merging parameter maps . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
9.3 Treatment of bad data . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
9.4 Treatment ofAngstrm coefficient . . . . . . . . . . . . . . . . . . . . . 18
9.5 Treatment of aerosol speciation . . . . . . . . . . . . . . . . . . . . . . . 18
9.6 Determination of the covariances . . . . . . . . . . . . . . . . . . . . . . 18
9.7 Correcting sensor dependent biases . . . . . . . . . . . . . . . . . . . . . 19
10 Step-by-step implementation of the merging algorithm . . . . . . . . . . . . . . . 19
11 Outputs of the merging scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
12 Requirements on retrieval algorithms imposed by Merging scheme . . . . . . . . . 2413 MPM file format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
13.1 Basic assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
13.2 File format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
13.3 MPM file type 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
13.4 MPM file type 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
13.5 MPM string . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
4/26
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
5/26
1. DOCUMENT VERSION HISTORY 5
1 Document version history
1.0: Original document issued 4 February 2005.
1.1: Issued 31 May 2005.
Added section13: Definition of MPM file format.
Added definition of sinusoidal grid coordinates, u, vto section7.1.
Defined grid selected for study (sinusoidalNeq = 4008).
2.0: 6 February 2007
Introduced IOAP as a delivered product and re-defined IGAP.
Updated section5.
Added figure3to illustrate nominal time of merged product.
Added section10,giving step-by-step description of the merging implementation. Sec-
tion includes quality control criteria determined during preliminary GAP validation.
Added figures4and5to illustrate merging process.
Criteria for selection of merged aerosol type changed / updated in section 9.5.
Operational settings defined during WP5200 included.
2.1: 29 Jun 2007
Minor correction to sinusoidal grid definition. Addition of algorithm to identify index
to compact array storage of gridded data.
2 Introduction
This document describes the algorithm for merging the global aerosol products (GAPs) derived
individually from the AATSR, MERIS and SEVIRI sensors into a single merged dataset. The
document describes the merging scheme and in addition details requirements which are imposed
on the individual sensor retrieval schemes to facilitate the merging process.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
6/26
6
3 WP Document
This document covers the input required under work package 2330:
4 Overview
This document is structured in the following way:
A brief description of key aspects of ATSR-2, AATSR, MERIS and SEVIRI products is
provided. Only details which impact the merging algorithm are provided. For further infor-
mation see the relevant ATBD.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
7/26
5. THE INDIVIDUAL-SENSOR AEROSOL PRODUCTS 7
The distinct product types which will be delivered are defined.
The spatial and temporal characteristics of the merged data-set are defined.
The approach to combine observations from each sensor is described.
Key outputs required from the merging scheme are listed.
A summary of the requirements imposed by the merging scheme on the individual sensor
retrieval schemes is given.
5 The Individual-Sensor Aerosol Products
5.1 General characteristics
Each IOAP is nominally produced at 10km spatial resolution. The inherent spatial resolu-tions of each sensor differ, but all are significantly finer than 10km. An approach to grid the
sensor data up to the 10km GlobAerosol resolution is defined in this document such that the
same grid is used for all delivered products.
The products consist of retrieved estimates of aerosol optical depth (AOD) at 0.55 and 0.865
m, the Angstrm coefficient .
For ATSR-2, AATSR and SEVIRI these products are produced using the Oxford-RAL Aerosol
and Clouds (ORAC) scheme, separately for each of 5 different aerosol models (namely con-
tinental, desert, maritime, urban and biomass). After these retrievals, the most-
likely aerosol type is identified using quality-control criteria. For MERIS no type informa-tion is available.
The AOD,, is defined by
() =
0e(z, ) dz=
0(s+a)(z, ) dz (1)
The total extinction coefficient, e, is defined as the sum of the extinction due to absorption,a, and scattering,s.zis altitude and is wavelength.
The Angstrm coefficient is defined as
d log [()]
d log [] . (2)
By assuming that varies linearly with in the visible, it is directly calculated from the tworetrieved optical depths:
A0.55,0.865 = log[0.865/0.55]
log[0.865/0.55]. (3)
5.2 AATSR
AATSR data is available from April 2002 to the present.
One product is produced per day corresponding to the time of each Envisat observation
(defined by the sun-synchronous orbit with descending node equator crossing of 10:30am
local solar time).
The AATSR swath width is 512i km centred on the sub-satellite point.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
8/26
8
5.3 MERIS
MERIS data is available from April 2002 to the present.
As AATSR, One product is produced per day corresponding to time of each Envisat obser-
vation.
The MERIS swath width is 1150 km centred on the sub-satellite point.
5.4 SEVIRI
SEVIRI is on the Meteosat-8 platform (formerly named MSG-1) which is currently in geo-
stationary orbit above 0 latitude and longitude. Data is available every 15 minutes.
SEVIRI was launched on 28 August 2002. The mission was declared operational on 29
January 2004. It is assumed that data will be available to this project only after the latterdate.
The geographical coverage available is illustrated in figure1.
The SEVIRI products will be produced at 10:15 and 16:15 UT (univeral time = GreenwichMean Time, GMT). (Level 1 data is available every 15 minutes but only 2 time-slots per day
can be processed within available computational resources.)
Figure 1: The Earth disk viewed by SEVIRI. The line-of-sight zenith angle of SEVIRI from the
ground is shown by the colour scale.
5.5 ATSR-2
ATSR-2 data is available from April 1995 to the present. However (a) the pointing accu-
racy of the ERS-2 platform was degraded by gyro failure after January 2001 and (b) after
the ERS-2 tape recorders failed in June 2003 geographical coverage is restricted to regions
observed while ERS-2 has line-of-sight contact with a number of ground-stations. In any
event, comprehensive data-sets after January 2001 will are unlikely to be available to this
projects (routine production and cataloguing of UBT data at RAL was suspended after gyro
failure, pending the implementation of an adequate geolocation scheme).
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
9/26
6. DELIVERABLE PRODUCTS 9
The ATSR-2 swath width is 512km centred on the sub-satellite point, though due to down-
link bandwidth restrictions, various reduced coverage modes are often operated.
6 Deliverable productsThree types of product are identified. Each product contains data referenced to the defined GlobAerosol
grid:
1. Individual-sensor Orbit-based Aerosol Products (IOAP). These files contain all the output of
the individual sensor retrievals, with coverage corresponding to that of the original level 1
(radiance) data from the sensor. In particular:
For ORAC retrievals (ATSR-2, AATSR and SEVIRI) these files contain the results for
all of the five aerosol types, plus an indication of the best type, i.e. a flag indicating
which aerosol retrieval is considered most representative of the scene, based on qualitycontrol criteria which are defined following the preliminary validation of the individual
sensor aerosol products (see section 10, below). No information should be lost in
translating the original ORAC files (one of which is produced for each aerosol type
separately) into the amalgamated IOAP.
Since, in all cases, radiance data is gridded onto the selected GlobAerosol grid (see grid
selection below) before the retrievals are run, the IOAPs contain results which corre-
spond directly to GlobAerosol grid boxes. However, data is only stored for grid boxes
for which a retrieval was attempted for the given orbit (ATSR-2, AATSR, MERIS) or
time-slot (SEVIRI).
2. Individual-sensor Global Aerosol Products (IGAP). One IGAP is produced per day for each
sensor. The IGAPs contain records for all GlobAerosol grid boxes (filled with null-records
were no retrieval is available for a given box). For ORAC retrievals results are only reported
for the best aerosol type (and this type is identified).
In all cases the IGAPs contain estimates of aerosol optical depth (AOD) at 0.55 and 0.865
mand the Angstrm coefficient .
Note that for all instruments there may be more than one retrieval present in a given grid
square. For SEVIRI this is clear since 10:15 and 16:15 UT time slots are processed. For
ATSR-2, AATSR and MERIS two or more results will potentially be available where swaths
overlap towards high latitude. To resolve these ambiguities the IGAPs are produced usinga merging algorithm, defined below, which attempts to derive the aerosol property most
appropriate to a specific time which is defined for each grid box.
In addition, the reported aerosol properties in the IGAPs are also bias corrected based on
results from the preliminary validation of the IOAPs.
IGAPs therefore constitute bias-corrected daily maps of aerosol properties derived from each
sensor, at a specific time of day.
3. Merged Global Aerosol Products (MGAP). One MGAP is produce per day which combines
the results from all available sensors into a single map in the same format as the IGAPs,
valid for the same specific time of day. The same algorithm is defined below to implement
the merging steps necessary to generate both the IGAPs and MGAPs from the IOAPs. From
the processing point of view IGAPs are produced by running the merging scheme using
IOAPs from one instrument only.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
10/26
10
MGAPs therefore constitute daily maps of what is considered to be the best representation of
the aerosol field for the specific time of day, based on information from all included sensors.
7 Definition of the GlobAerosol grid7.1 Gridding schemes
All individual sensors are required to be produced at 10km spatial resolution. For the individual
sensors, it would be possible to satisfy this requirement by defining a grid based on the inherent
sampling characteristics of the instrument (e.g. averaging radiances or retrievals from 10x10 1km
pixels in order of acquisition by the instrument). However such grids would differ between
the IOAPs. Spatial interpolation of the IOAPs onto a common grid defined for the IGAP/MGAP
is not desirable, since this would imply a degradation of spatial resolution. It is preferable for
the IGAP/MGAP grid to be adopted at the earliest possible processing stage. If the 10km IOAP
and IGAP/MGAP grids are not the same it would be necessary to store retrievals at finer spatialresolution in the IGAPs or tolerate some degradation (factor 2) in the MGAP resolution.
Many approaches are possible to define a grid which has a nominal spatial resolution of 10km.
It is fundamentally impossible to define a grid in which all boxes have exactly similar area and
shape. Even neglecting the ellipticity of the Earth, there is no regular tessellation of the sphere
having more faces than the icosahedron - i.e. 20 equilateral triangles.
It is assumed that the following approach is adopted to up-scale finer resolution radiances or
aerosol retrievals:
A set of grid-points are defined (see below) corresponding to the centre of implied grid-
boxes.
Each fine-resolution radiance / retrieval is associated with the grid-point nearest to it.
All radiances / retrievals associated with each grid-point are averaged to form the 10 km
resolution product.
The grid-boxes are therefore implicitly defined by polygons centred on each grid-point. Depending
on the gridding scheme, the polygons may be more or less irregular. Although not required directly
for production of the GAPs, the edges and vertices of each polygon (in latitude and longitude
coordinates) are required to validate the GAPs and by users to properly interpret the data.
This approach can be adopted whatever the method used to define the grid-points themselves.
Each of the schemes described below could therefore be implemented without particularly com-plicating the GAP retrieval / merging schemes.
Note that representation of the global field in terms of basis functions analogous to spherical
harmonics (i.e spectral grids) are not considered suitable here because of the spatially discrete and
sparse nature of the GAPs. (On a single day there will be data gaps due to clouds in all products,
between the Envisat swaths and outside the SEVIRI field-of-view. It is not considered sensible to
base GAPs on a gridding scheme which would implicitly interpolate across such data-gaps.)
Regular latitude-longitude grids
I.e. a grid is defined which is equally spaced in latitude and longitude. A grid spacing of
0.1degree would give 10x10km spatial resolution at the equator.
Advantages:
Simple to implement.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
11/26
7. DEFINITION OF THE GLOBAEROSOL GRID 11
Easily understood by users.
Disadvantages:
The area of each grid box rapidly reduces towards higher latitudes (tending to 0 at the poles).
Gaussian grids
Gaussian grids are used by ECMWF to provide a compact representation of atmospheric and sur-
face data which is amenable to rapid conversion to and from the spectral grids used by their global
circulation model.
To quote from ECMWF documentation 1:
A Gaussian grid is a latitude/longitude grid. The spacing of the latitudes is not regular. How-
ever, the spacing of the lines of latitude is symmetrical about the Equator. Note that there is no
latitude at either Pole or at the Equator. A grid is usually referred to by its number N, which is
the number of lines of latitude between a Pole and the Equator.
The longitudes of the grid points are defined by giving the number of points along each line of
latitude. The first point is at longitude 0 and the points are equally spaced along the line of latitude.
In a regular Gaussian grid, the number of longitude points along a latitude is 4N. In a reduced
Gaussian grid, the number of longitude points along a latitude is specified. Latitudes may have
differing numbers of points but the grid is symmetrical about the Equator. A reduced Gaussian
grid may also be called a quasi-regular Gaussian grid.
In the reduced grids used by ECMWF, the number of points on each latitude row is chosen
so that the local east-west grid length remains approximately constant for all latitudes, with the
restriction that the number should be suitable for the Fast Fourier Transform used to interpolate
spectral fields to grid point fields, ie number= 2p 3q 5r. (Wherep,q,r are integers.)ECMWF currently employ grids up to number N512 which corresponds to a latitude / longitude
spacing of 0.08785 or 20km. For the purposes of this project a N1024 grid would be required to
give 10km spatial resolution. For illustration purposes, the N32 grid is defined in table1.
Advantages:
Grid boxes are much closer to having equal area in the reduced Gaussian grid scheme than a
simple latitude-longitude grid.
Easily understood by users already familiar with ECMWF data.
Code to handle data in this format appears to be available from ECMWF (tbc).
Meteorological data from ECMWF is available in a similar form (though at reduced resolu-
tion).
Disadvantages:
To enable proper interpretation of the data, the vertices and edges of each grid-box must be
defined and supplied to users.
Code to transform to a simple latitude-longitude grid would be needed by users.
The implied grid boxes are irregularly shaped.
1http://www.ecmwf.int/publications/manuals/libraries/interpolation/gaussianGridsFIS.html
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
12/26
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
13/26
7. DEFINITION OF THE GLOBAEROSOL GRID 13
Sinusoidal grid
Longitude, is transformed as a function of latitude,, as follows
= cos() (4)A regular grid in transformed longitude and (normal) latitude is then adopted. See figure
2.
The surface of the earth is covered by rectangular tiles in the transformed latitude and longitude
coordinates. These tiles can be identified by coordinates uandv.
u= Rgcos()+u0 (5)
v= Rg+v0 (6)
Where
Rg =Neq/360 (7)
u0= 180.Rg+ 0.5 (8)
v0 = 90.Rg+ 0.5 (9)
Neq is the number of tiles around the equator and should be an even integer. IfNeq is divisible by4 then the equator lies on the boundary between grid boxes. Integer values ofu,v correspond tothe centre of each tile in the sinusoidal grid.
Data can be stored easily in a 2-d array indexed by integer values of u and v which run from 1 to
Neqand 1 toNeq/2, respectively. However this storage is somewhat wasteful as many points in the
array do not correspond to valid latitude / longitude combinations. It may therefore be convenientto store data in a 1-d array containing only valid grid points. u andv coordinates can be convertedto an index to this array as follows:
i= u+Bv Neq/2 +Nv/2 (10)
Whereu andv are here assumed to be rounded to the nearest integer and are therefore consideredas indices to the column and row of a given grid box. Nv is the number of valid grid boxes in rowvandBv is the first index assigned to a grid box in rowv . Nv is equal tocos().Neq, rounded upto the nearest even integer. Bv is the sum ofNv over all rows from 1 to v 1, except for the firstrow whereB1= 0.
Advantages:
Grid boxes are approximately equal area.
A similar grid is used by the MODIS land albedo product (used by the AATSR retrieval
scheme).
Easily understood by users already familiar with MODIS data.
Disadvantages:
Code to transform to a simple latitude-longitude grid would be needed by users. (This could
possibly be based on existing MODIS code.)
Grid boxes are poorly defined at +/180 degrees longitude (which is of little consequencefor land products but of some importance for atmospheric products).
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
14/26
14
Icosahedral grids
Approximately regular grids can be defined as follows:
Inscribe a regular polygon with triangular faces into the sphere. Usually an icosahedron is
chosen (20 equilateral triangles, 12 vertices).
Subdivide each face into 4 triangles (projected onto the surface of the surface of the sphere)
by bisecting the great circle between each vertex.
Iterate the above step until the desired resolution is reached.
The process is illustrated in figure2.
Advantages:
Grid boxes are approximately equal area.
All grid boxes are approximately regular hexagons, with the exception of 12 pentagon-
shaped boxes centred on the original icosahedron vertices.
Disadvantages:
Code to transform to a simple latitude-longitude grid would be needed by users.
Grid spacings can only be chosen within a factor of 2 of the desired resolution (due to the
iterative bisections).
Most users are unlikely to be familiar with this approach.
7.2 Selected grid
For GlobAerosol it has been decided to adopt a sinusoidal grid withNeq= 4008.
Figure 2: Illustration of the sinusoidal (left) and icosahedral (right) gridding schemes.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
15/26
8. THE MERGING ALGORITHM 15
8 The Merging algorithm
8.1 Temporal characteristics of the IGAPs and MGAP
A single merged product will be produced per day which corresponds to the Envisat observ-ing time (i.e. approximately 10:30am local solar time). This implies:
For most of the globe no interpolation in time is required for AATSR and MERIS (but
see below).
SEVIRI data must be interpolated in time from the universal time (UT) at which each
scene is acquired to the Envisat observation time (which is dependent on longitude and,
to lesser extent, latitude). This will be implemented as follows:
The UT corresponding to the Envisat sub-satellite point when over each grid point
will be determined for the nominal Envisat orbit. This will be defined in a merg-
ing parameter map (see below) once and for all before the merging scheme isapplied. The nominal times for each grid square are illustrated in figure 3
The two SEVIRI IOAPs which bound in time the UT of each grid point will be
identified2
SEVIRI products from both the bounding IOAPs will be input to the merging
algorithm. The covariance associated with each of the two SEVIRI products (see
equation12below) will be weighted to effectively perform interpolation in time,
simultaneously accounting for any differences between the retrieval uncertainties
of the two products:
Sx
1 = ek(tnomts)2
Sx
1 (11)
Where Sx is the original matrix valid at SEVIRI GAP time ts before or after theMGAP nominal time,tnom. k is a temporal de-correlation factor. At present it isassumed that k = 0.0192541hours2, such that the variances are doubled for a6 hour time difference. S
x
for each of the two SEVIRI products will be used in
equation12.
Near the poles multiple retrievals may be available from overlapping AATSR and (par-
ticularly) MERIS swaths. These cases will be treated in the same way as SEVIRI: The
two retrievals which bound in time the nominal UT of the Envisat nadir point will be
identified and merged with other results using the same de-correlation factor, k. I.e. in
general the approach outlined for SEVIRI should, in fact, be implemented for all instru-ments. Over most of the globe, this will result in no temporal interpolation for AATSR
and MERIS, but will exploit multiple retrievals at high latitudes where appropriate.
9 Merging co-located retrievals
9.1 General approach
Assumptions:
2In principle all retrieval throughout the day could be considered to allow cloud-free retrievals from distant times
to be interpolated when those nearest in time are contaminated by cloud. It is not proposed to implement this level
of sophistication since the changes in meteorological conditions implied by the changing cloud cover increase in an
ill-defi ned way the interpolation uncertainties.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
16/26
16
0
5
10
15
20
Time / GMT
Figure 3: Nominal time in UT of each grid square in the GlobAerosol grid, approximately cor-
responding to the time at which Envisat over-flies each square (i.e. 10:30 local solar time of thedescending node crossing). Negative values indicate a time on the preceeding day (offset by 24
hours).
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
17/26
9. MERGING CO-LOCATED RETRIEVALS 17
Merging is only applied to retrievals which correspond to the same GlobAerosol grid box.
No attempt is made to spatially interpolate retrieved data.
For a given box, we are given Nobservations of the same set of quantities described by
vectorsxi=1...N. Eachxiis subject to errors described by a Gaussian probability density function with covari-
ance Sxi.
Observation errors are not correlated between the N instances ofx.
The most likely estimate ofxis given by the weighted mean:
x=
Ni=1Sx1i
1
Ni=1Sx1i xi
(12)
The covariance of this merged result is given by:
Sx = Ni=1
Sx1
i1
(13)
In the current context eachi corresponds to a different IOAP. No spatial interpolation will beperformed, so N will vary from 0 to 6 (AATSR, MERIS, SEVIRI observations at two times which
bound the nominal time of the merged product), depending on the presence of valid results from
each IGAP at each grid-point.
A number of issues arise when considering practical implementation of this approach. These
are discussed below:
9.2 Merging parameter maps
For the purpose of controlling the merging scheme a number of parameters must be defined. Many
parameters will be expected to exhibit spatial variation to some degree. To maintain generality and
simplicity in the merging code, most parameters will be defined in a merging parameter map
(MPM), a file in which a given parameter is specified at each of the MGAP grid points.
The spatial variation within each file will be defined on a case-by-case basis in WP5200, but
whatever variation is implemented will have no implications for the coding of the merging scheme.
In many cases the quantities in the MPMs will vary to account for differing retrieval scheme
performance under specific geophysical conditions (e.g. over land / sea or particular surface type).
Multiple MPMs may be required to cope with varying observing conditions (particular solar zenith
angle). For these cases valid ranges (e.g. of solar zenith angle) will be defined and the appropriate
MPM selected (no interpolation between MPMs will be required).
9.3 Treatment of bad data
Before merging a number of tests will be performed on each IGAP retrieval at each grid point. If
a test is failed, the particular AATSR / SEVIRI / MERIS IGAP point will be flagged as bad and
not included in the merging. Tests include:
Satisfactory cost function value at solution.
Satisfactory cloud fraction.
Satisfactory estimated errors from the retrieval scheme.
Level of contamination by sun-glint.
Each threshold will be defined in an MPM (though some may not vary spatially).
Specific settings are given in section10, below.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
18/26
18
9.4 Treatment ofAngstrm coefficient
To ensure consistency in the final merged product only the two optical depths (0.55 and 0.865 m)will be merged via equation12. Angstrm coefficient will be derived from the resulting merged
optical depths using equation3
9.5 Treatment of aerosol speciation
It is an assumption of equation12that the xicorrespond to the same quantity. This assumption willbe violated if the IGAP optical depths are derived assuming different aerosol types. It is therefore
important to first identify the type which is to be assigned to the merged product. The merging
equation12can then be applied to combine AODs from SEVIRI and ASTSR which correspond to
the same type with that from MERIS (which has no assigned type). I.e. an overall best type is
identified for each scene in the MGAP and it is the AATSR and SEVIRI retrievals for the overall
best type that are merged (provided satisfactory retrievals exist for this type in each case), and
these will not necessarily be the retrievals which are flagged as best in the individual IOAP files.
The ORAC-based AATSR and SEVIRI IGAPs include AOD for all 5 modelled aerosol types.
Criteria are defined to select the best type based on experience gained during the preliminary val-
idation of the products. However there is generally insufficient information in the ORAC retrieval
for an unambiguous identification of the aerosol type. It is therefore likely that the AATSR and
SEVIRI IOAPs, and even the IOAPs from the same sensor at the two times bounding the nominal
merging time, will not agree as to which is the best aerosol type for a given grid box.
Since SEVIRI provides more uniform coverage over the disk it observes, it is likely to result in
fewer / less pronounced discontinuities in the MGAP if SEVIRI is used to define the aerosol type
where SEVIRI data is present. However the SEVIRI scene at 16:15 UT is quite distant in time
from the nominal merging time (within the SEVIRI observed disk).The merging algorithm will select the aerosol type by taking the type identified as best from
the first IOAP in the following priority list with valid retrievals for a given scene:
SEVIRI IOAP at the time nearest to the nominal merging time,t0. (This will be the 10:15UT scene.)
AASTR IOAP att0.
SEVIRI IOAP at the other time bounding the nominal merging time, t1. (This will be the16:15 UT scene.)
AASTR IOAP att1.
If only MERIS data is present then the aerosol type will be flagged as unknown.
9.6 Determination of the covariances
The Sxfor each IGAP will be a 2x2 matrix (corresponding to the two AODs).
Currently it is proposed to neglect error correlations between the two optical depths and there-
fore the off-diagonals of this matrix are set to 0. The variances are defined as follows:
For ORAC-based retrievals variances are taken from the estimated retrieval error reported in
the IOAP. For MERIS no errors are reported in the IOAP and so the variances are estimated from the
spread of the retrieval values about co-located Aeronet measurements, as determined in the
preliminary validation.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
19/26
10. STEP-BY-STEP IMPLEMENTATION OF THE MERGING ALGORITHM 19
9.7 Correcting sensor dependent biases
Equation12assumes errors are not correlated between the IGAPs. In practice each IGAP will
have specific geo-temporally dependent biases. It is proposed to correct for these biases by (again)
using results from the preliminary IGAP validation: Scatter-plots of IGAP AOD vs Aeronet AOD will be produced.
Linear (optionally second order) fits to the relationship between IGAP and Aeronet AOD
will be calculated (using a least-squares fitting approach).
The coefficients of these fits are stored in regions local to the corresponding Aeronet stations
in MPMs.
Each IGAP AOD is corrected by the fitted first or second order polynomial before application of
equation12.I.e. each IGAP AOD is bias corrected relative to the Aeronet AODs.
Coefficients will be defined for each IGAP, for each of the five aerosol types in a specific MPM.
MPMs may be defined for particular ranges of solar zenith angle depending on the outcome of the
validation.
10 Step-by-step implementation of the merging algorithm
Having defined the basic principles and equations behind the merging process in the preceeding
sections, the following provides a step-by-step description of the complete process, filling-in de-
tails required for the coding of the algorithm.
The process is also illustrated in figures4and5. Figure4illustrates the overall process, includ-
ing the operation of the ORAC retrievals on level 1 radiance data. Figure5illustrates the mergingprocess, including the identification of the best type for the MGAP.
Note that the scheme is applied 5 times: Four times to the IOAPs of each sensor individually
to generate IGAPs, and a final time including IOAPs from all sensors (other that ATSR-2) together
to form the MGAP. Here we describe the MGAP formation, since this is the general case. IGAP
formation is identical, but taking only IOAPs from the specific sensor as input. IOAPs are also gen-
erated using ATSR-2 data, which is not included in the MGAP because ATSR-2 is not processed
over the period contemporary with the other instruments.
To generate ATSR-2, AATSR and SEVIRI data ORAC is run using input files which contain
cloud-free radiances averaged onto the GlobAerosol sinusoidal grid. MERIS data is also
produced on the same grid. This ensures that no spatial interpolation is required to generatethe IOAP, IGAP or MGAP.
ORAC output files which are produced for every orbit and every aerosol type are amalga-
mated into the IOAPs. These contain all the information in the original ORAC output files
for a given orbit for all aerosol types. Additionally, the best aerosol type is indicated by a
flag. The best type is determined by (i) applying quality control criteria to screen out bad
retrievals (ii) selecting the retrieval with lowest cost from those retrievals which pass these
tests. The following quality control criteria have been determined during the preliminary
GAP validation and are stored in MPMs:
For SEVIRI:
urban-type retrievals are never considered valid for this purpose. Urban and
polluted maritime results are not considered in the amalgamation (doing so led to
confusing results).
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
20/26
20
Cloud fraction according to the SEVIRI mask is 0.
The surface albedo at 0.55m is less than 0.12.
The retrieval has converged.
The optical depth at 0.55
m is less than 3.
More than one iteration of the retrieval was performed.
The retrieved effective radius is less than 8m.
The following cost function thresholds are applied depending on aerosol type and
whether the grid-box is over land or sea (partial land is considered as land for this
purpose).
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
21/26
11. OUTPUTS OF THE MERGING SCHEME 21
2. The SEVIRI, AATSR and MERIS retrievals in the same grid box which are closest in
time are identified. The closest time (in UT) of each instrument is referred to ast0.
3. In each case, if it exists, the retrieval at the other time,t1which boundstnomis identi-fied. t0 is always the closest in time so it may be the case that either t0 < tnom < t1ort0 > tnom > t1.t0and/ort1 may be on the previous or next day relative to the nominalday of the MGAP. Retrievals more than 12 hours from the nominal time will not be
considered in any case.
4. For the ORAC-based retrievals (ATSR-2, AATSR and SEVIRI), the retrievals which
pass the quality control criteria above are again identified.
5. ??? GMV Define MERIS Quality Control here ???
6. The type which will be assigned to the given grid-box is identified by taking the identi-
fied best retrieval from the first data-set with valid retrievals in the priority list defined
in section9.5. If the only valid data present is MERIS, then the type will be classified
as unknown.
7. The set of retrievals which (a) pass the quality control criteria and (b) correspond to the
selected type or are from MERIS (unknown type) are included in the merged value for
the given grid point. These retrievals are handled as follows:
(a) The22estimated error covariance matrix for the optical depths at 0.55 and 0.87m is formed for each retrieval, as described in section9.6.It must be ensured thatthese error covariances are appropriate to the representation of the retrieval optical
depths inlog10units.
(b) Equation11is applied to each covariance to account for the error growth with time
away from the nominal merging time.(c) The optical depth values are bias-corrected using the gain and offset parameters
determined from linear-fits to the Aeronet vs. retrieved optical depth scatter plots
during the preliminary validation exercise.
() =c0+c1() (14)
Where()is the retrieved optical depth,()is the corrected optical depth, andc0, c1are offset and gain parameters defined in MPMs. Note that the correction isdefined in absolute optical depth units, hence the stored (log10) values must be con-verted into absolute units before applying this equation and the result subsequently
converted back intolog10units.
(d) Equation12Is applied to the corrected optical depths to determine the merged
optical depths. Note this equation should be applied directly to the storedlog10optical depths.
8. The merged Angstrm coefficient is determined from the merged optical depths using
equation3.
11 Outputs of the merging schemeThe following parameters are required in the MGAP product.
A version number identifying the IGAP processor versions.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
22/26
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
23/26
11. OUTPUTS OF THE MERGING SCHEME 23
Figure 5: The merging algorithm including quality control and selection of the merged aerosol
type.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
24/26
24
A version number identifying the MGAP processor versions.
A version number identifying the set of MPMs used.
Time and date of data applicability and processing.
The latitude, longitude and UT of the MGAP grid points. (The implied grid-box polygons
should also be defined in a separate, time independent file).
For each grid-point:
Index of the grid point.
AOD at 0.55 and 0.865 microns.
Estimated covariance of the two AODs.
Angstrm coefficient .
Aerosol type indication.
Indication of most likely type from the AATSR / SEVIRI IGAP if different.
Flag indicating which IGAPs were valid and therefore merged, including identification
of the two possible observations bounding the nominal MGAP UT for each instrument.
12 Requirements on retrieval algorithms imposed by Merging
scheme
The 5 aerosol classes selected for retrieval should employ similar optical models and altitudeprofile shapes.
The schemes should employ either (a) the same gridding scheme as the MGAP or (b) retain
retrievals at finer spatial resolution such that interpolation onto the MGAP grid does not
result in degraded information.
13 MPM file format
13.1 Basic assumptions
The following is a description of the format adopted for the MPM files. The following general
points apply:
Parameters in an MPM file can be defined as a function of month. Month is an integer
giving the month number since April 1995 (the start of the ATSR-2 mission). If a specific
parameters is only defined for 1 month then it is to be assumed that the given values apply
for all time. Linear interpolation in time is to be assumed where MPMs are defined for more
than one month. Extrapolation should not be performed, the defined value which is closest
in time to the required time should be taken if the required value is outside the defined time-
range.
MPM files are binary files written by IDL on a 32-bit linux machine.
Long integers are 4 bytes long. Short integers are 2 bytes. Floats are 4 bytes. Doubles are 8
bytes.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
25/26
13. MPM FILE FORMAT 25
13.2 File format
The following describes the contents of each file:
An MPM string (see13.5) containing the file version.
Long integer defining the MPM type, MPM_TYPE. This can have value 1 or 2:
MPM_TYPE=1: Each tile in the sinusoidal grid is given a region number (e.g. 0 for
land, 1 for ocean). MPM parameters are defined for each of the numbered regions. If
only one region is defined then the parameter is assumed to apply globally.
MPM_TYPE=2: Each tile in the sinusoidal grid is given a specified value of the merging
parameter.
The type is chosen on a parameter by parameter basis to minimise the size of the MPM file.
Long integer defining the number of grid points round the equator in the sinusoidal grid.
This will be 4008 for GlobAerosol:
MPM string containing the name of the file when originally produced.
MPM string describing the origin of the file (RAL, Oxford, GMV etc).
MPM string describing the history of the file.
MPM string containing the creation data of the file as returned by the IDL systime command,
e.g. Wed May 31 12:59:41 2006.
MPM string defining the instrument to which the MPM applies. This can be one of the
following:
SEVIRI, AATSR, ATSR-2, MERIS
1 long integer defining the number of parameters in the file: N_PAR.
N_PARMPM strings giving the name of each parameter.
1 long integer defining the number of aerosol types for which the parameters are defined:N_AER. If this is 1 then the parameters apply to all aerosol types.
N_AERMPM strings giving the name of aerosol type.
1 long integer defining the number of wavelengths for which the parameters are defined:
N_WL. If this is 1 then the parameters apply to all wavelengths.
N_WLfloat values giving the value of each wavelength.
1 long integer defining the number of months for which the parameters are defined: N_MONTHS.
If this is 1 then the parameters apply to all months.
N_MONTHS long values giving the value of each month.
The remaining file contents depend on MPM_TYPE.
-
7/26/2019 Algorithm Theoretical Basis Document (ATBD) Merging_atbd_v2p1
26/26
26
13.3 MPM file type 1
1 long integer defining the number of regions for which parameters are defined:N_REG.
1 long integer giving the type, I_TYPE, of the MPM variable as follows:
1: byte
2: short signed integer (2 bytes)
3: long signed integer (4 bytes)
4: float (4 bytes)
5: double (8 bytes)
An array, PARS, of typeI_TYPE with dimensions
N_PAR, N_WL, N_AER, N_MONTHS, N_REG
This contains the values of the merging parameter.
If the number of regions is greater than 1 then regions are defined on the sinusoidal grid as
follows:
1 long integer giving the number of tiles in the sinusoidal grid being used:N_GRID.
N_GRID short signed integers giving the region number for each grid tile, I_REG (zero-
based index to
N_GRIDshort signed integers giving the u-coordinate of each grid tile (see section 7.1).
N_GRIDshort signed integers giving the v-coordinate of each grid tile (see section 7.1).
13.4 MPM file type 2
1 long integer giving the number of tiles in the sinusoidal grid being used:N_GRID.
1 long integer giving the type, I_TYPE, of the MPM variable as defined in section13.3.
An array, PARS, of typeI_TYPE with dimensions
N_PAR,N_WL,N_AER,N_MONTHS,N_GRID
This contains the values of the merging parameter.
N_GRIDshort signed integers giving the u-coordinate of each grid tile (see section 7.1).
N_GRIDshort signed integers giving the v-coordinate of each grid tile (see section 7.1).
13.5 MPM string
1 long integer defining the length of the string: LEN
LENbytes of ASCII encoded text (if LEN>0).