MIT REMOTE SENSING AND ESTIMATION GROUP Cho 1 Anomaly Compensation and Cloud Clearing of AIRS...

MITREMOTE SENSING AND ESTIMATION GROUP

http://rseg.mit.edu Cho1

Anomaly Compensation and Cloud Clearing of AIRS Hyperspectral Data

Presented at IEEE GRSS

Boston Section Meeting

August 24, 2005

Choongyeun (Chuck) Cho



Overview I Problem statement

Definition of anomaly

Background Atmospheric InfraRed Sounder (AIRS), Advanced

Microwave Souding Unit (AMSU), and Humidity Sounder for Brazil (HSB) instruments

Signal characterization and reduction of artifacts Principal component analysis (PCA) and its variants Artifacts in AIRS data



Overview II Stochastic cloud-clearing

Background and prior work Description of stochastic-clearing (SC) algorithm

Validation of stochastic-clearing algorithm ECMWF Physical clearing

Conclusion Summary / contributions Future work



Where Are We? Problem statement

Definition of anomaly Background Signal characterization and reduction of artifacts Stochastic cloud-clearing Validation of SC algorithm Conclusion



Problem Statement

What is hyperspectral data? Hundreds or thousands of contiguous channels

What is anomaly? Defined as an unwanted spatial or spectral signature,

statistically distinct from its surrounding: Given X and a priori information about Δ, what is best

estimate of Δ or X? A priori info about anomaly can have different forms:

• Spectral statistical description, or usually ensembles• Spatial structure or texture• Joint spatial/spectral description

XX~

˜



Examples of Anomalies

Anomalies of AIRS data discussed in this talk Instrumental noise Noisy channels

• AIRS data exhibits consistently noisy channels Scan-line miscalibration

• Resulting in striping patterns Cloud contamination

• Clouds generally make IR observations colder• Compensation for cloud impact is critical for accurate

retrieval



Where Are We? Problem statement Background

AIRS/AMSU/HSB instruments Signal characterization and reduction of artifacts Stochastic cloud-clearing Validation of SC algorithm Conclusion



AIRS/AMSU/HSB Instruments on Aqua

Atmospheric InfraRed Sounder (AIRS) 2378-channel infrared spectrometer covering

3.7-15.4μm 1.1˚ FOV (13.5km at nadir)

Advanced Microwave Sounding Unit (AMSU) 15 microwave channels 3.3˚ FOV (40.5km at nadir)

Humidity Sounder for Brazil (HSB) 1.1˚ FOV (13.5km at nadir, same as AIRS) 4 microwave channels (150,190 GHz) Scan motor failure since Feb 2003

AIRS/HSB

AMSU

“golfball”



Sample AIRS Brightness Temperatures

AIRS sample spectra for 3-by-3 FOVs AIRS sample image

Data: August 21, 2003 near Great Lakes



Where Are We? Problem statement Background Signal characterization and reduction of artifacts

Principal component analysis (PCA) and its variants Artifacts in AIRS data

Stochastic cloud-clearing Validation of SC algorithm Conclusion



Principal Component Analysis Useful to characterize multivariate signal, and reduce

dimensionality

Can be defined recursively (x: m-dim multivariate signal):

Solution: y = WTx where W = [w1|w2|…|wn], wi are eigenvectors of CXX which is often estimated

To reduce dimension, n << m The resulting reconstructed signal: (PC filter)xWWx Tˆ

xwyxww

xwyxww

kkT

kiwwwk

Tw

k

),var(maxarg

),var(maxarg

1,...,1,1

1111



Noise-Adjusted Principal Component

Principal components are sensitive to arbitrary scaling

Noise-adjusted PC (NAPC) Normalize data before applying PCA: Guarantees maximum SNR

What if noise variances are not known? Need to be estimated using blind signal separation (BSS)

technique such as Iterative Order and Noise (ION) estimation

XGX 2

1

na



Noise-Adjusted Principal Component

Typical “variances versus PC index” plot (truncated at 400)

NAPC shows sharper break than PC 6 NAPCs explain 99.8% of variances

Cumulative explained varianceScree plot for AIRS data

Var

ian

ce



Artifacts in AIRS Data

Measurements from a sensor can have different sources of artifacts

AIRS data has different types of unwanted artifacts Instrumental noise Noisy channels Scan-line miscalibration

Each dot in the image is:

K 2.0|ˆ| ,, jiji xx



Instrumental Noise

Instrumental white noise is unavoidable NAPC filtering provides adaptive noise filtering

NAPC filtering is extensively used in our stochastic clearing algorithm and noisy channel detector.

Before

After

Histogram of difference



Noisy Channels I

Some channels are consistently noisy These channels need to be excluded for further analysis and

retrieval of physical parameters Noisy channel detection is done with NAPC filtering such

that a channel having 5σ even once is flagged “noisy”

Block diagram of noisy channel detector



Noisy Channels II

AIRS science team has their own list of bad channels.

Detection rates for noisy and popping channels:

Type of bad channel Number of NASA compiled bad channels

Number detected by proposed algorithm

% detected

High noise 5 5 100%

Detector response exhibits unexpected steps (Popping)

2 2 100%

Total 7 7 100%



Scan-line Artifacts I

Miscalibrated scan-line results

in stripes

A simple low-pass filter (in along-track direction) in NAPC domain corrects this artifact efficiently

Block diagram of removing-stripe algorithm



Scan-line Artifacts II Results Original Images Processed Images



Where Are We? Problem statement Background Signal characterization and reduction of artifacts Stochastic cloud-clearing

Background and prior work Description of stochastic-clearing (SC) algorithm

Validation of SC algorithm Conclusion



Background and Prior Work

What is cloud-clearing? Cloudy radiances (or TB) cause inaccurate retrieval Cloud-cleared radiances: radiances which would have

been observed if FOV contains no clouds

Prior work on cloud-clearing Ignore cloudy FOVs: only ~5% of AIRS FOVs are clear! Physical cloud-clearing: iterate between estimation of

physical parameters and calculation of observed radiance Adjacent-pair clearing: use adjacent FOVs which have

different fractional cloud cover Purely spatial processing: restore 2-D temperature field

from sparse clear data



Stochastic Clearing (SC) How does stochastic clearing (SC) work?

SC estimates cloud contaminations solely based on statistics without using any physical models

Hyperspectral measurements may contain sufficient information about clouds in an obscured manner

Robust and stable training is necessary Nonlinearity is accommodated using stratification

(sea/land, latitude, day/night), multiplicative scan angle correction, etc.

Advantages of SC approach Simple: SC does not require physical models (retrieval or

radiative transfer). Fast: Based on matrix addition and multiplication



Block Diagram of SC Algorithm

Linear Operator A

3x3 AIRS

TB’sSelect/average

FOV’s

5 microwave ’sLand fraction

Secant

Linear Operator B

Linear Operator C

Linear Operator D

CloudyTest

Morecloudy

Less cloudy

N

1 PC-cloud 2 TB’s1

7

N

1 PC

Cleared AIRS TB’s

N = 314 channels



Operators for SC AlgorithmECMWF + SARTA (v1.05)

clear TB’sN AIRS TB’s

AMSU ch.5,6,8,9,10

secant Land fraction

AIRS-cloudPC’s

TB’s

AIRS Cleared

TB’s

Find warmest † among 9 pixels*

Find coldestamong 9 pixels*

Noise-Adjusted

PC’s

Noise-Adjusted

PC’sLINEAR

ESTIMATOR PC-1

N

7

3

5 4

TB-

+

++

Trained with >1000 golfballs

N

N

N

Operator A

Select & avg FOV’s

Trained with >1000 golfballs

Operator B Operators

C, D

* Warmest/coldest based on 11 4-m channels peaking 1-3 km

†Average 4 warmest pixels for 5-10 km WF, 9 for WF >10 km



Clearing Corrections vs ECMWF ’s

-10 0 10 20 30 40 50 60oK

AIR

S c

lear

ed –

EC

MW

F/S

AR

TA

(oK

)

10

5

0

-5

-10

10

5

0

-5

-10

Weighting function peak ~0.47 km, 2217.4 cm-1

1oK threshold

2oK threshold

Red circles are best 37 percent

“good golfballs” for nighttime ocean,

all angles (operator C)

Blue circles are cloudy golfballs

(operator D)

Classification Algorithm: “good

golfballs” are below both 1K

and 2K thresholds at operator B

Weighting function peak ~2.7 km, 2231.5 cm-1

-10 0 10 20 30 40 50 60 70oK



AIRS-ECMWF for 827 Channels

Weighting function peak height (km)

314 good channels

Best 22 percent of golfballs

(operator C)

Thresholds were 0.8K and 3K

Nighttime ocean, all scan angles

Includes all 4- and 15-m

channels plus one-fifth of the

rest

0 1 2 3 4 5 7 9 19 29 39

314 good channels



Where Are We? Problem statement Background Signal characterization and reduction of artifacts Stochastic cloud-clearing Validation of SC algorithm

ECMWF Physical clearing

Conclusion



ECMWF Data Set Used ECMWF profiles are used to simulate cloud-cleared TB’s via

SARTA v1.05 radiative transfer, for all scan angles, 314 channels

Global, 3 days: 8/21, 9/3, 10/12/03 (L1B v3.0.8)

AIRS instrument noise reduced by averaging 1, 4, or 9 of the warmest* 15-km pixels for weighting function peaks 0-5, 5-10, and >10 km, respectively

Used 1000 golf balls to regress each of 10 categories based on day/night, land/sea, and |lat|<40 or 40<|lat|<70



RMS Difference (AIRS-ECMWF) for Ocean

For best 28%: sea + |lat|<40º + day (best for sea)



RMS Difference (AIRS-ECMWF) for Land

For best 28%: land + |lat|<40º + night (best for land)



Cloud-cleared AIRS vs. ECMWF (best 28%)

RMS difference (oK) between cloud-cleared AIRS and ECMWF/SARTA, 10 different estimators

RMS > 0.7 K are boxed Excellent agreement for ocean and equatorial regions Degradation over daytime land near surface

Weighting function

peak height (km)

Ocean|Lat|<40 30<|Lat|<70

Day Night Day Night

Land|Lat|<40 30<|Lat|<70

Day Night All Day Night All

0 – 1

1 – 2

4 – 5

6 – 7

10 – 11

0.38 0.4 0.86 0.91 1.68 0.77 1.36 1.48 0.78 1.19

0.27 0.29 0.54 0.57 0.94 0.38 0.75 0.84 0.44 0.70

0.28 0.30 0.45 0.45 0.34 0.29 0.33 0.41 0.33 0.39

0.23 0.27 0.34 0.36 0.25 0.24 0.28 0.34 0.26 0.31

0.24 0.27 0.33 0.35 0.23 0.25 0.26 0.24 0.28 0.27



Global Cloud-Clearing Images (2392.1cm-1)

SC applied to 8/21/2003 descending orbits (L1B v4.0.9) 2392.1cm-1, WF peak=0.23 km

Observed AIRS Cloud-cleared (best ~78%)




CC Residual = Spatially high-pass filtered version of CC WF peaks at 0.23 km

Observed AIRS CC residual



Cloud-Clearing Images and Sea Surf. Temperature

Angle-corrected TB images at window channels

Clearing works well even if there is no hole (clear FOV)

Ob

serv

ed

Cle

ared

SS

T

AIRS 2390.1cm-1: near Hawaii AIRS 2399.9cm-1: near SW Indian Ocean



Validation with Physical Clearing Visible vs. AIRS 8.15-m Data

Visible Ch 3

Granule 91(9/6/02)

(solar reflection)

1227.7 cm-1

Channel 1284(H2O)

5

15

25

35

45

140

120

100

80

60

40

20

5

15

25

35

45

290

285

280

275

270

265

260

270

265

260

255

250



Determination of AIRS Cloud-Cleared Brightness Temperature Baselines

Channel 1284 (H2O) (1227.7 cm-1)Granule 91 (9/6/02) Indian Ocean, LAT -26.3, LON 70.2

Stochastic CC Clear Mask † Fitted ‡

† Clear mask determined from Visible Ch 3, Brown being clear‡ Polynomial fit: 4th order in scan angle, 3rd order downtrack



Stochastic versus Physical Cloud-Cleared8.15-m Brightness Temperatures


Filter subtracts cloud-cleared baseline and convolves with 33 boxcar

-Visible Ch 3Stochastic Clearing Physical Clearing



Correlation Coefficients: Visible (v3) vs. Cloud-Cleared 8.15-m AIRS TB’s

AIRS channel 1284(1227.7 cm-1)(water vapor)September 6, 2002

Lon -26.3/Lat 70.2Indian Ocean

Lon -12.5/Lat -106.1Eastern Southern Pacific



Stochastic and Physics-BasedCloud-Cleared Window Channel TB

Channel 2121 (2399 cm-1), WF peak ~200m9/6/02 North of Bermuda

* Note that the physical clearing is most recent version (PGE v4.0.0), calculated this week.

Stochastic Clearing Physics-Based Clearing*



Where Are We? Problem statement Background Signal characterization and reduction of artifacts Stochastic cloud-clearing Validation of SC algorithm Conclusion

Summary / contributions Future work



Summary

Anomaly compensation techniques are discussed based on signal processing techniques (NAPC and ION) and nonlinear estimators

Anomalies: Gaussian instrument noise, noisy channels, scan-line miscalibration, and cloud contamination

Stochastic clearing (SC) algorithm tested to be successful using different validation schemes: ECMWF Sea surface temperature (SST) Physical algorithm



Contributions: Methodology We developed “architected nonlinear estimators”, taking

advantage of simplicity of linear estimator and robustness of general nonlinear estimator

NAPCs are used to reduce dimension and suppress noise, making more robust and stable estimation

Prior knowledge (either spectral or spatial) about anomaly is utilized to meaningfully structure a nonlinear estimator

Linear

Estimator

GeneralNonlinearEstimator

ArchitectedNonlinearEstimator

Dimensionreduction

Prior infoabout anomaly• Stable

• Simple/Fast• Less powerful

• May be unstable • Complex/Slow • Most powerful

• Stable• Simple/Fast• Sufficiently powerful



Contributions: Stochastic Clearing

SC algorithm developed based on anomaly compensation and nonlinear estimation techniques: Enjoys excellent agreement with numerical weather

prediction model (ECMWF) Performs superior to physical clearing Very fast: depends on matrix addition and multiplication;

consists of 664 lines of Matlab code, ~20 minutes to cloud clear an entire day of AIRS data on ordinary PC

Clearing at extreme scan angles is good; hole-hunting using high spatial resolution may not be essential for cloud-clearing



Future Work

Anomaly compensation theory Optimization of combining spectral and spatial processing More extensive spectral processing techniques

SC algorithm Joint cloud-clearing and retrieval Better model for nonlinearities Optimum architecture for SC algorithm using efficient

design-of-experiment approach



Where Are We? Problem statement Background Signal characterization and reduction of artifacts Stochastic cloud-clearing Validation of SC algorithm Conclusion

That’s all, folks.

Any questions?



Back-up Slides




Zoom-in images in Southeastern Pacific WF peaks at 0.23 km

AIRS CC CC residual



Determination of AIRS Cloud-Cleared Brightness Temperature Baselines


Stochastic CC Cloud Mask † Fitted ‡

† Brown being clear (accepted)‡ Polynomial fit: 4th order in scan angle, 3rd order downtrack



SC Validation w.r.t. SST

NCEP sea surface temperature



AIRS Stratified Stochastic Cloud ClearingFirst-pass results Multiplicative Scan-angle Stratified results

(AIR

S c

lear

ed r

adia

nce)

– (

EC

MW

F/S

AR

TA

) (o K

)

46% golfballs good

54% rejected

46% golfballs good

54% rejected

0.8K threshold

3K threshold

Estimate of cloud-clearing radiance correction (oK)



AIRS Stratified Stochastic Cloud Clearing

Altitude of weighting function peak (km)

RM

S:

clea

red

AIR

S r

adia

nce

s vs

. EC

MW

F/S

AR

TA

First pass, all golfballs, all scan angles

Second pass, multiplicative scan angle

Third pass, best 46% golfballs (all scan angle)Third pass’, best 46% golfballs using |θ|<16○

0.5K



AMSU Contributions for Land

For land + 30º<|lat|<70º + night



Iterative Order and Noise (ION) Estimation

To apply NAPC, the noise variances need to be estimated

Signal model: x = Ap + Gn, where A is a mixing matrix, p is signal of unknown dimension k, G is diagonal noise covariance matrix, n is unit-variance white Gaussian noise.

ION iteratively estimates signal order (k), mixing matrix (A) and noise variances (G): Estimate order (k): using scree plot Estimate A and G: using Expectation-Maximization

algorithm



Blind Signal Separation

Observation model: x = Ap + G1/2w- A, p, G, w and k (dimension of p) are unknown.- A: mixing matrix (n×k), p: source signal vector (k×1)- G: diagonal cov matrix (n×n), w: white Gaussian noise vector (n×1)- Use matrix X, P and W to denote concatenated samples of x, p and w: X = AP + G1/2 W

Given X, how to estimate A, G and k.

Previous signal separation techniques assume either k or G is known

Iterative Order and Noise (ION) estimation:- First, estimate signal order, k, using eigenvector decomposition- Estimate A and G using Expectation-Maximization algorithm



Blind Signal Separation II

Estimation of signal order, k:

- Selected based on scree plot, sorted eigenvalue vs.

eigenvalue index

Knee point separates

signal from noise

X = AP + G1/2W



Blind Signal Separation III

Expectation-Maximization (EM) algorithm iteratively finds maximum likelihood (ML) estimate of parameters where model depends on hidden (latent) variable

Expectation step: estimate unobserved data (P and PTP) using estimate of A and G

Maximization step: compute ML estimate of A and G using estimates of P and PTP

X = AP + G1/2W



ION Algorithm Block Diagram

i iˆ ˆG ,Z

ii, AG

1-i1,-i AG

X, optionally normalize rows to zero mean and unit variance

oG I, i 1 Set imax, e.g. imax = 10

Noise Normalization

1 2n i 1

ˆX G X

nX

Order Estimation

Scree Plot

SVD

EM Algorithm

Expectation

Maximization

1k

i<imax

Yes

No

ii, AG



Physics of Radiative Transfer

Radiative transfer links environmental parameters to hyperspectral data

For a black body, spectral brightness is defined as:

For microwave channels (hf << KT) , this is linear with temperature:

1-1-2-/2

3

HzsterWm)1(

2),(

KThfec

hfTfI

Tc

kfTfI

2

22),(




Upwelling radiation received at a sensor at altitude L has four contributions:

Can be rewritten as:

)sec(2

0

)sec(),()sec(

)sec(

0

)sec(),(

0

00

0

)(

),(),()sec()(

))(1(

),(),()sec(),(

eIf

dzezfzfIef

eIf

dzezfzfILfI

cosmic

L dzzf

surface

L dzzf

z

Lz

L

s dzzfIzfWfILfI0

),(),()(),(




Weighting functions for AIRS/AMSU

AIRS Weighting function peaks

AMSU Weighting functions



Physics of Radiative Transfer Microwave/IR atmospheric absorption spectrum

MW: Water vapor absorption lines at 22, 183, 325 GHz, O2 absorption at 118, 368 GHz, etc.

IR: distinct water vapor, O3, CO2 absorptions

Microwave Infrared

MIT REMOTE SENSING AND ESTIMATION GROUP Cho 1 Anomaly Compensation and Cloud Clearing of AIRS...

Documents

Transcript of MIT REMOTE SENSING AND ESTIMATION GROUP Cho 1 Anomaly Compensation and Cloud Clearing of AIRS...