LOFAR offline interference detection
Transcript of LOFAR offline interference detection
LOFAR offline interference detection
André Offringa1
Ger de Bruyn1,2
Saleem Zaroubi1
Michael Biehl3
1Kapteyn Astronomical Institute, Groningen2ASTRON
3Faculty of Computing Science, University of Groningen
Contents
● About interference detection algorithms● Results for LOFAR● Perspectives for SKA
About detecting interference
● Motivation for automated detection:– Enormous data sizes in new telescopes
such as LOFAR
– No possibilities for manually “browsing” the data for bad baselines, bad antennae, contaminated frequencies, or any other bad dimension of the cube
About RFI detection (2)
● Many methods for RFI removal:– Blanking before correlation at highest time
resolution
– Spatial filtering before correlation
– Flagging after correlation
– RFI modeling and subtraction before and/or after correlation
● After: e.g. fringe fitting● Before: e.g. cyclostationair filtering, satellite
subtraction
● Most of the methods are complementary
Flagging methods
Approach of automatic (baseline) flaggers:● Estimate the value of a sample using its
neighboring time/frequency samples.
● If this deviates substantially from its real value, flag it.
Flagging methods
Approach of automatic (baseline) flaggers:● Estimate the value of a sample using its
neighboring time/frequency samples.– Median value, polynomial fit, weighted
Gaussian filter
● If this deviates substantially from its real value, flag it.
– By thresholding, line detection or combinatorial thresholding
See: Bhat et al. (2005), Winkel et al. (2006), Offringet al. (2010)
Freq
Time (Offringa et al., MNRAS, 2010, in press)
Flagging methods
● Combinatorial thresholding strategy● Idea:
– Sum samples and use different thresholds
A > threshold1? → FLAG AA+B > threshold2? → FLAG A, BA+B+C > threshold3? → FLAG A, B, CA+B+C+D > threshold4? → FLAG A, B, C, DA+E > threshold2? → FLAG A, EA+E+F > threshold3? → FLAG A, E, FA+E+F+G > threshold4? → FLAG A, E, F, GB > threshold1? → FLAG BB+C > threshold2? → FLAG B, C.......
Method comparison
● Compare methods with the help of test sets and ROC curves
● One of the test sets:
(Offringa et al., MNRAS, 2010, in press)
Without flagging After flagging:
LOFAR results
● Data quality is very good
● Flagging somewhat between 1% to 3%
● No show-stopping RFI
LOFAR results
● We see some in sito RFI, but it is very weak
● Too weak to subtract– Done at the GMRT
● Effects after long time integration not yet known
(Athreya, AJ, 2009)
LOFAR results (2)
● Many different flaggers are needed
Different needs per KSP for flagger sensitivity and flagging strategy:
– EoR needs a very stringent approach
– Pulsars need an approach not based on time smoothness
– For transients the same for frequency
– Other option is to quickly scan for RFI to compress the data immediately after
SKA and RFI
SKA and RFI
● The SKA will also be influenced by RFI:– Satellites, aeroplanes
– Lightning
– In situ RFI
– Car engines, (imperfect) transmitters, electrical fences
– Classes of RFI currently not observed
● ...and more detection requirements... :– Failing hardware and software
SKA and RFI
● We are not sure yet what it takes to flag LOFAR data sufficiently
● >>5002 cross-correlations with 10 x LOFAR time resolution and 50 x LOFAR beams is not trivial, if at all possible in 2020
● The order of the data might be a crucial problem:
– Accurate flagging needs large chunks of the cubes in T/F directions
– Calibration needs chunks in the baseline-direction
SKA and RFI
● If necessary, detectors can always trade off speed for accuracy, e.g.:
– Flag on time/frequency integrated data
– Flag on auto correlations
– Flag on groups of antennae
● But is it wise to build more antennae than we can possible process?
Conclusions
● LOFAR can very well deal with its RFI● For SKA, processing will be a major issue● IO and transport might be problematic● The more antennae, the less accurate you
can deal with RFI.● Carefully watch the pathfinders...
● If anyone wants to automatically flag data... => [email protected]