John Hubbert, Greg Meymaris, Mike Dixon, Scott Ellis

NATIONAL CENTER FOR ATMOSPHERIC RESEARCHBoulder, Colorado

NEXRAD TAC MeetingNorman OK

29 April 2019

Rethinking Clutter Filteringand

Improving Signal Statistics

0

.2

.4

.6

.8

1.0

-.2

-.4

-.6

-.8

-1.0

-1.2

-1.4

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Time (sec)

Clutter signal

2 sec.

Spectra-Based Clutter filtering• Since the advent of fast digital receivers this has been the

standard, e.g., GMAP• Replaced time domain filters• Very common in weather radars

Discrete Fourier Transform

Fourier Transform pairFrequency domain

Time domain,i.e., sum of sinusoids

0

.05

.10

.15

.20

.25

.30

.35

.40

.45

.50

-.05

-.101 12

Real Part of a S-Pol Time Series.

Raw Time series

0

.05

.10

.15

.20

.25

.30

.35

.40

.45

.50

-.05

-.101 128 256 384 512

Time (ms)

Real Part of a S-Pol Time Series. DFT creates a periodic signal

Raw Time series

0

.05

.10

.15

.20

.25

.30

.35

.40

.45

.50

-.05

-.101 128 256 384 512

Time (ms)

Raw Time series In order to create a

sum of sinusoids that can replicate this jump discontinuity, many higher frequency sinusoids are required.

Real Part of a S-Pol Time Series. DFT creates a periodic signal

0

.04

.08

.12

.16

.20

.24

.28

.32

1 128 256 384 512

Time (ms)

0

.05

.10

.15

.20

.25

.30

.35

.40

.45

.50

-.05

-.101 128 256 384 512

Time (ms)

Raw Time series

HanningWindowedTime series

In order to create a sum of sinusoids that can replicate this jump discontinuity, many higher frequency sinusoids are required.

HanningWindowedTime series

Real Part of a S-Pol Time Series. DFT creates a periodic signal

0

10

-10

-20

-30

-40

-50

-60

02468

1012141618202224

-2-4-6-8

-10

Raw spectrumHanning windowed

spectrum

m/s m/sVelocity Velocity

Window Effects on SpectraRaw Hanning windowed

The smooth curve spectrum is an artifact of the jump discontinuity

-40dB

-10dB

0

.2

.4

.6

.8

-.2

-.4

-.6

-.8

-1.0

-1.2

-1.4

-1.6

-1.8

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Time (sec)

0

.2

.4

.6

.8

1.0

-.2

-.4

-.6

-.8

-1.0

-1.2

-1.4

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Time (sec)

(a) (b)

Scan of Rocky Mountains by S-Pol at 8 deg/sec.

(16 degrees az.)

Typical Clutter I and Q Signals

I Q

Degs.

What is Regression Filtering?

Clutter + Weather Weather Only

Represent the I and Q Signals as a Sum of These Orthogonal PolynomialsFast, low round off error algorithm: http://jean-pierre.moreau.pagesperso-orange.fr/

Regression Clutter filtering• Regressions filters have a history in biomedical field

•

• And weather radar• Torres, S. and D. Zrnic ́, 1999: Ground clutter canceling with a

regression filter. J. Atmos. Oceanic Technol.

Regression Filter Example on S-Pol Data

Power Spectrum Time series

0

.05

.10

.15

.20

.25

-.05

20 40 60 80 100 120

Time (ms)

RealPart

Raw dataFitted data

0

.05

.10

.15

.20

.25

-.05

20 40 60 80 100 120

Time (ms)

RealPart

Raw dataFitted data

8th order polynomial fit

Subtracting the polynomial fit from the time series…..

Regression Filtered versus Raw Spectrum

-14

-45

Blackman Window and Notch Technique

Regression Window and NotchComparison

Regression versus Window and Notch

• WHY USE A REGRESSION FILTER???

Signal Statistics5.23dB attenuation

• Hanning window: 4.19 dB attenuation

About 50% increase in variance!

About 35% increase in variance!

Regression Frequency Response64 point time series for polynomial orders 3 - 7

Frequency Response 264 point time series for polynomial orders 9, 11, 13, 15, 19

Frequency response depends on sequence length and polynomial order

Modified Regression Filtering

1 2 3 4 ….. …N

Standard technique: Take length N sequence, fit a polynomial to it and subtract to remove the low frequency clutter components.

Modified technique: Break the sequence into blocks, lets say 4 for this example:

1 Nl l l

N/4 N/2 3N/4

1. Do 4 regression fits, thus suppressing the ground clutter in each block2. Concatenate the blocks back into one sequence3. Calculate radar variables

block 1 block 2 block 3 block 4

time series

1 N

m points

m points

Length N sequence broken into overlapping blocks. The overlap is m pointsand these regions are shown in green.After filtering, overlapping pairs of points are averaged so that a N pointfiltered sequences is created. This is done to smooth the transition from oneblock to the next.

The Blocks Can Be Overlapped and WeightedThis is to reduce slight discontinuities at end points.

.1 .3 .7 .9

.9 .7 .3 .1

weightsBlock1

Block2

S-Pol Clutter Environmentat Marshall

22 April 20180.5 elevation

Compare Clutter Rejection of the Regression and GMAP like filters (notch width is constant)

-15

-20

-25

-30

-35

-40

-45

-50

-55

-60

-65

-70

-75-15-20-25-30-35-40-45-50-55-60-65-70-75

-15

-20

-25

-30

-35

-40

-45

-50

-55

-60

-65

-70

-75-15-20-25-30-35-40-45-50-55-60-65-70-75

++

+

+

+

+

+

+

+

+

+

+

+

+

+

+

++

+

++

+

+

+

+

+

+

+

+

+

+

+

+

+

+

++

+

Scatter Plots Regression vs Window and Notch(4 blocks of 16)

KFTG Data. “Bomb Cyclone” 13 March 2019Short PRT data from VCP 212

25 km50 km

225o

Z Vel

Raw Blackman-Nuttall

9 pt notch

KFTG LPRT Data VCP212, 20 deg/sec, 16 points

6th order

KFTG 227 Az. 0.45 elev13 March 2019

Unfiltered data

5th order regression filter Blackman window 7 point notch

A Comparison of Regression versus Window and Notch Clutter Filters

Smearing Effects of Windowing

Raw data (rectangular window) Blackman-Nuttall Window

S-Pol data

Resulting Implications for the Clutter Filter Bandwidth

5 pt notch insufficient!!

Window function spreads the clutterpower so that wider bandwidth is needed

Vmax-Vmax Vmax

Clutter 0.25 m/s

0

Weather 2 m/s

Modeling Efforts Have Begun

28 m/s-28 m/s

64 points30 dB SNR weather30 dB CNR clutterBlackman Window, 9pt notchRegression 7th order

~ 50% increase in STD~ 30% increase in STD

Application to Data Sets Has Begun

Bomb Cyclone Level 2 Data Times Series Processed with Regression Filter

KFTG

Old frequency response

Compensatedfrequency response

Passband Stopband

Frequency

Frequency Compensation

After regression filtering, FFT the resulting timeseries and compensate the frequencies as desired as guided by the frequency response of the regression filter.

An interesting possibility

Conclusions

much better signal statistics can be achieved.

• Windowing the data, while containing “clutter leakage”, spreads the clutter (causes wider clutter bandwidth). This then requires a wider bandwidth notch as compared to regression filtering.

• Thus underlying weather signal is is more effectively recovered with a regression filter