L2OS RFI status

Nicolas Lamquin, Jean-Luc VergelyJacqueline Boutin

Paul Spurgeon

ICM, Barcelona, 16/17 May 2013

RFI flags in v600 processingVersion of products:

• v600 source: sources-L2OS-core-v6.00.tar.gz

• job orbit: descending SM_TEST_MIR_SCSF1C_20110505T040402_20110505T045804_600_001_0

• L1c flag description : SO-TN-IDR-GS-0005 (24/10/2012)

• CNFOSF file: SM_TEST_AUX_CNFOSF_20050101T000000_20500101T000000_001_023_8

• RFI region: arctic

L1c RFI flags (measurement)

• Fm_rfi_mitigation_tot (bit 7) (v600) rfi_xx– analysis of NIR BT in X polarization

• Fm_rfi_strong_tot (bit 15) (v600) rfi_yy– analysis of NIR BT in Y polarization

• Fm_rfi_tails_tot (bit 12) never raised – affected by tails identified in the AUX_RFI list

• Fm_rfi_point_tot (bit 16)– affected by point source identified in the AUX_RFI

list

L1c L2 (DAP: measurements)

• L2 flag « L1c_rfi »: True if any of L1c RFI flag True

snapshot 34 (id:79011306): Fm_rfi_point_tot flag in L1c L2 « L1c_rfi » is True

No RFI detection from NIR

Tails in Fm_rfi_point_tot ?

Y

green = RFI_XX True

red = RFI_XX True and RFI_YY True

snap. 632id:79012029

snap. 633id: 79012031

snap. 634id: 79012032

?

Y

X

L2 RFI flags (measurements)

• Fm_rfi_L1• Fm_l2_rfi• Fm_l2_rfi_outlier• Fm_l2_rfi_snapshot_out_of_range• Fm_l2_rfi_high_snapshot_std• Fm_l2_rfi_high_snapshot_std_st3• Fm_l2_rfi_high_snapshot_std_st4

L2 RFI flags (measurements)

• Fm_l1c_rfi• Fm_l2_rfi• Fm_l2_rfi_outlier• Fm_l2_rfi_snapshot_out_of_range• Fm_l2_rfi_high_snapshot_std• Fm_l2_rfi_high_snapshot_std_st3• Fm_l2_rfi_high_snapshot_std_st4

gridpoint analysis (risk of RFI + RFI_nsig test) local information per measurement

snapshotanalysis raised for complete snapshot

from L1c

True if Fm_l2_rfi_outlier or Fm_out_of_range, nominally includes snapshot analysis below but not in current configuration

Fm_l2_rfi = Fm_l2_rfi_outlierOR Fm_out_of_range

• Fm_out_of_range: threshold test on residual per measurement (Tm_out_of_range=50 K)

• Fm_l2_rfi_outlier: analysis of all residuals per gridpoint (« risk of RFI ») + threshold test on residuals (comparison to std)

snapshot 34 (id:79011306)

• extra points in Fm_l2_rfi = Fm_out_of_range• Fm_l2_rfi_outlier is not raised although

Fm_out_of_range is raised bRiskOfRFI seems not raised

• sensitivity to examine (threshold 50 K)

snapshot 649 (id: 79012050)Sun tails ?

• flags all snapshot if any Fm_out_of_range

• does not proceed if some coast/ice nearby: set to True only if %(coast/ice) < Ts_snapshot_out_of_rangeTs_snapshot_out_of_range = 20%

test threshold

Fm_l2_rfi_snapshot_out_of_range

Fm_l2_rfi_snapshot_out_of_rangeexample 1 snapshot 34 (id:79011306):

example 2 snapshot 649 (id: 79012050):

Land contamination :

Snapshot flag not raised

Fm_l2_rfi_high_snapshot_stdFm_l2_rfi_high_snapshot_std_st3

and Fm_l2_rfi_high_snapshot_std_st4

• snapshot standard deviation tests• per measurement in snapshot (« Fs_high_std... »)• propagates through complete snapshot

• test 1: amount of data for statistics must be > Ts_meas_min (default: 35%)

• test 2: sqrt(std / nMeas) > Ts_std (default: 2.5)

• same threshold for « std », « std_st3 », and « std_st4 »

refinement of threshold(s) ?

Test definition

• for each configuration : process 1 month of data ? NO ! (too much thresholds to be tested)

• tests on one orbits and do relative comparison : relative SSS, number of valid TBs …etc.

• tests on several orbits in a (lat,lon) window in order to check the salinity estimation (comparison with ISAS maps).

• after a choice of configuration : monthly test

Remarks

• would be useful to have snapshot_ID for all measurements in L2 DAP

• need « Fs_high_std... » in DAP ?• DPM not coherent in 8_2-1 and 8_3-1 (OK in

the code)• Some variables not defined in the DPM:

RFI_c1, RFI_c2• Fm_RFI_L1_tot (type ‘s’) in DPM ? (OK in the

code : all L1c flags are used)

L2OS RFI detectionDecision tree

Introduction

• Difference between outlier and RFI ?

RFIs : unexpected physical source + Gibbs and tailsOutliers : reconstruction problem ?

- we expect snapshot signature for RFI and outlier- long time period contamination for RFI : seen by all the orbits which overpass the source : AUX_DGG_RFI information

Flags in the decision treeif Fm_Resol(im,igp) or (Fm_outlier(im,igp) and not Fg_ctrl_suspect_RFI) or (Fm_L2_RFI(im,igp) and Fg_ctrl_suspect_RFI) or Fm_out_of_range(im,igp) or Fm_border_fov(im,igp) or Fm_Moon_SpecDir(im,igp) or Fm_gal_noise_error(im,igp) or Fm_sun_limit(im,igp) or Fm_L1c_sun(im,igp) thenFm_valid = falseendif

Fg_ctrl_suspect_RFI is raised if (number of Fm_L2_RFI==1 )/Na > Tg_num_RFI_max

Fm_outlier : comes from comparison between TB_smos and TB_model corrected from a global shift applied at the dwell line level. nsigma comparison.

Fm_L2_RFI : Fm_L2_RFI_outlier or Fm_L2_RFI_snapshot_out_of_range or Fm_L2_RFI_high_snapshot_std

Fm_L2_RFI_outlier : comes from comparison between TB_smos and TB_model. RFI_nsigma comparison.

Fm_outlier and Fm_L2_RFI_outlier very close: NO REASON TO USE DIFFERENT nsigma criteria (?).

Proposition : suppress Fm_L2_RFI_outlier and keep only snapshot information for Fm_L2_RFI.

INSTEAD OF :

if Fm_Resol(im,igp) or (Fm_outlier(im,igp) and not Fg_ctrl_suspect_RFI) or (Fm_L2_RFI(im,igp) and Fg_ctrl_suspect_RFI) or Fm_out_of_range(im,igp) or Fm_border_fov(im,igp) or Fm_Moon_SpecDir(im,igp) or Fm_gal_noise_error(im,igp) or Fm_sun_limit(im,igp) or Fm_L1c_sun(im,igp) thenFm_valid = falseEndif

PUT :

if Fm_Resol(im,igp) or Fm_outlier(im,igp) or (Fm_L2_RFI(im,igp) and Fg_ctrl_suspect_RFI) or Fm_out_of_range(im,igp) or Fm_border_fov(im,igp) or Fm_Moon_SpecDir(im,igp) or Fm_gal_noise_error(im,igp) or Fm_sun_limit(im,igp) or Fm_L1c_sun(im,igp) thenFm_valid = falseEndif

WITH

Fm_L2_RFI(im,igp) : Fm_L2_RFI_snapshot_out_of_range or Fm_L2_RFI_high_snapshot_std

Remarks

• Fm_l2_RFI and Fg_ctrl_suspect_RFI do not integrate L1c RFI information

• Fm_l2_RFI and Fg_ctrl_suspect_RFI do not integrate AUX_DGG_RFI information

-> AUX_DGG_RFI and L1c information not used in the decision tree

Extra slides

Snapshot out-of-range TB detection (PRP_8_2)

-> Fs_out_of_range

-> Fm_l2_rfi_snapshot_out_of_range

Snapshot standard deviation tests (PRP_8_3)

-> Fs_high_std

-> Fm_l2_rfi_high_snapshot_std

Measurement outlier detection (PRP_8_4)

-> Fm_l2_RFI_outlier

-> Fm_outlier

OTT correction

RFI AUX detection (PRP_8_5)

-> Fm_l2_RFI (from Fm_l2_RFI_outlier, Fm_l2_rfi_snapshot_out_of_range,

Fm_l2_rfi_high_snapshot_std)

-> Fg_ctrl_suspect_RFI

Snapshot level

Snapshot level

Dwell line level

Grid point level