Fourier Transformations Jeff Edmonds York University COSC 6111 Change from Time to Polynomial Basis...

Fourier Transformations

Jeff Edmonds

York UniversityCOSC 6111

• Change from Time to Polynomial Basis• Evaluating & Interpolating• FFT in nlogn Time• Roots of Unity• Same FFT Code & Butterfly• Inverse FFT• Sin and Cos Basis• FFT Butterfly• Polynomial Multiplication• Integer Multiplication

Fourier Transformation Fourier Transform• are a change of basis from the time basis to

• sine/cosine basis• JPG• or polynomial basis

Purposes:• Signal Processing• Compressing data (eg images with .jpg)• Multiplying integers in n logn loglogn time.• Error correcting codes.• ….

http://www.dspguide.com/ch8.htm

The Scientist and Engineer's Guide toDigital Signal Processing

By Steven W. Smith, Ph.D.

Amazingly once you include complex numbers,

the FFT codefor sine/cosines and for polynomials

are the SAME.

http://www.dspguide.com/ch8.htm

Fourier Transformation

Instead of using sine and cosines

as the basis,

PolynomialBasis


Instead of using sine and cosines

as the basis,

We use polynomials.

PolynomialBasis

Change of Basis: T([a1,a2,…,ad]) = [A1,A2,…,Ad]Changes the basis used to describe an object.

• The Time basis of a vector space:• A tuple <w1,w2,…,wd> of basis objects• Linearly independent• Spans the space uniquely

"v $[a1,a2,…,ad], v = a1w1+a2w2 +… + adwd• The Polynomial basis of a vector space:• A tuple <W1,W2,…,Wd> of basis objects• Linearly independent• Spans the space uniquely

"v $[A1,A2,…,Ad], v = A1W1+A2W2 +… + AdWd

PolynomialBasisFourier Transformation

[3,2]

v =

[a1,a2] = [11/5,32/5][A1,A2] =

Change of Basis: T([a1,a2,…,ad]) = [A1,A2,…,Ad]Changes the basis used to describe an object.

"v $[a1,a2,…,ad], v = a1w1 +a2w2 +… + adwd

"v $[A1,A2,…,Ad], v = A1W1+A2W2 +… + AdWd

=[w1,w2]=[ , ]

StandardBasis

v =

PolynomialBasis

NewBasis =[W1,W2]

=[ , ]W1[1]

W1[2]

W2[1]

W2[2]

[ ][ ] =[ ]W1[1]W1[2]

W2[1]W2[2]

a1a2

A1A2



f = a0∙B0+a1∙B1+a2∙B2 +… + an-1∙Bn-1

=[B0,B1,..]

f =

a0 =11/5

a1 =32/5

PolynomialBasis

=[ , ]

Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a1,a2,…,an-1]Changes the basis used to describe an object.

A discrete function

x

f(x)

f(x) = a0 +a1x +a2x2 + …+ an-1xn-1

coefficients of the polynomial

[a1,a2,…,an-1] =

PolynomialBasis

Have you seen Taylor Expansions of a Function?They show that functions f(x) can be expressed by specifying the coefficients of the polynomial.

F(x) = a0+a1x +a2x2 +a3x3 + …

Eg: f(x) = 1/(1-x) F(x) = 1+x +x2 +x3 + …

Fourier Transformation PolynomialBasis

xe


F(x) = a0+a1x +a2x2 +a3x3 + …


)sin(x


F(x) = a0+a1x +a2x2 +a3x3 + …



Time Basis =[ , ]

f =


y[0]=3

y[1]=2

=[I0,I1,…]=[?,?]The time basis

x

Ij[x]

zeroone

j

"y $[y[0],y[1],…,y[n-1]], y = y[0]I0 +y[1]I1 +… + y[n-1]In-1

PolynomialBasis

A discrete function

x

f(x)

the value f(j) of the function at x=j.

y[j] =



Instead of evaluatingthe polynomials onj = 0, 1, 2,…, n-1

We evaluatethem on

x0, x1, x2, …, xn-1

These xj are special fixed values.

Later we set xj = e2i j/n

PolynomialBasis


y[j] =



Time Basis =[ , ]

f =


y[0]=3

y[1]=2

=[I0,I1,…]=[?,?]The time basis

x

Ij[x]

zeroone

xj

"y $[y[0],y[1],…,y[n-1]], y = y[0]I0 +y[1]I1 +… + y[n-1]In-1

PolynomialBasis

A discrete function

x

f(x)

the value f(xj) of the function at x=j.

y[j] =


x0 x1 x2 x3 x4 … xn-1

• A Fourier Transform is a change in basis.• It changes the representation of a function

• from the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1

• This amounts to evaluating f at these points.

Evaluating &Interpolating

x

• to the value f(xi) at key values xi.

x0 x1 x2 x3 x4 … xn-1

y0 y1 y2 y3 y4 … yn-1


yi = f(xi)





(x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

(x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1

(xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1

(x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1

Vandermonde matrixInvertible if xi distinct.


yi = f(xi)


• to the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1

• This amounts to interpolating these points.

• An Inverse Fourier Transform is the reverse. • It changes the representation of a function


x

• from the value f(xi) at key values xi.

x0 x1 x2 x3 x4 … xn-1

y0 y1 y2 y3 y4 … yn-1

yi = f(xi)




Given a set of n points in the plane with distinct x-coordinates,

there is exactly one (n-1)-degree polynomial going through all these points.

• An Inverse Fourier Transform is the reverse. • It changes the representation of a function


f(x) = a0+x(a1+x(a2 + … + x(an-1) … ))

Fourier Transformation• My input: • (a0,a1,a2,…,an-1) & x

• My output: • f(x) = a0+a1x +a2x2 + … + an-1xn-1

1+2+3+..+n-1 multiplications?

n multiplications and n additions.

Given n distinct values (x0,x1,x2,…,xn-1).Can they all be done in less than n×O(n) time?


Fourier Transformation• My input: • (a0,a1,a2,…,an-1) & x


Given n distinct values (x0,x1,x2,…,xn-1)Can they all be done in less than n×O(n) time?

(x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

(x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1

(xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1

(x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1

yi = f(xi)

I hope so


• The Fast Fourier Transform (FFT) is a very efficient algorithm for performing a discrete Fourier transform

• FFT principle first used by Gauss in 18?? (But was not interesting without computers)

• FFT algorithm published by Cooley & Tukey in 1965• In 1969, the 2048 point analysis of a seismic trace

took 13 ½ hours. Using the FFT, the same task on the same machine took 2.4 seconds!

Fast Fourier Transformation FFTnlogn Time

16 32 64 128 256 512 1024

70

60

50

40

30

20

10

0

Err

or

(pp

m)

DFT FFT

Not only do you get faster speed + in place memory processing but fewer

calculations means less round off errors

Maybe I should take CSE6111

after all!


N DFT (N2) FFT (1.5N log N) faster

32 1,024 240 4.3

64 4,096 576 7.1

128 16,384 1,344 12.2

256 65,536 3,072 21.3

512 262,144 6,912 37.9

1024 1,048,576 15,360 68.2

2048 4,194,304 33,792 124.1

4096 16,777,216 73,728 227.6

Discrete Fourier Transform is too slow for real time!


Divide & Conquer - Friends - Recursion.• Trust your friends to solve any subinstance: • as long as smaller and • is an instance to the same problem.

Fast Fourier Transformation

My instance

My friend’sInstance



FFTnlogn Time

f(x) = a0+a1x +a2x2 + … + an-1xn-1

Fast Fourier Transformation• My input: (start with one x)

• (a0,a1,a2,…,an-1) & x

= a0+a2x2+a4x4 + … + an-2xn-2

+ a1x+a3x3+a5x5 + … + an-1xn-1

= a0+a2x2+a4x4 + … + an-2xn-2

+ x( a1+a3x2+a5x4 + … + an-1xn-2 ) = feven(x2) + x( fodd(x2) )


• 1st friend’s input? • feven: (a0,a2,a4,…,an-2) & ?

• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1) & ?

f(x) = feven(x2) + x fodd(x2)

feven(z) = a0+a2z+a4z2+a6z3+ … + an-2zn/2-1

FFTnlogn Time

Fast Fourier Transformation• My input: (start with one x)

• (a0,a1,a2,…,an-1) & x


• 1st friend’s input? • feven: (a0,a2,a4,…,an-2) & x2

• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1) & x2

• 1st friend’s output: • yeven = feven(x2)

• 2nd friend’s output: • yodd = fodd(x2)

f(x) = feven(x2) + x fodd(x2)

• My output: • f(x) = yeven + x yodd

• T(n) = 2 T(n/2) + O(1) = O(n)Ok. So it takes O(n) time to evaluate.

FFTnlogn Time

Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• (x0,x1,x2,…,xn-1)

• My output: • (y0,y1,y2,…,yn-1)• yi = f(xi)

• 1st friend’s input? • feven: (a0,a2,a4,…,an-2)• (x0

2,x12,x2

2,…,xn-12)

• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1)• (x0

2,x12,x2

2,…,xn-12)

• 1st friend’s output: • i y<even,i> = feven(xi

2)

• 2nd friend’s output: • i y<odd,i> = fodd(xi

2)

• My output: • i f(xi) = y<even,i> + xi y<odd,i>

• T(n) = 2 T(n/2) + O(n) = O(n log n)

Wow! That was easy.

FFTnlogn Time



2,x12,x2

2,…,xn-12)

Oops

n coefficientsn values of x

n/2 coefficientsn values of x

Does not meet precondition!

FFTnlogn Time




2,x12,x2

2,…,xn/2-12)


2,x12,x2

2,…,xn/2-12)


• T(n) = 4 T(n/2) + O(n)That’s no good.

• 3rd friend’s input? • feven: (a0,a2,a4,…,an-2)• (xn/2

2,xn/2+12,…,xn-1

2)

• 4th friend’s input? • fodd: (a1,a3,a5,…,an-1)• (xn/2

2,xn/2+12,…,xn-1

2)

= O(n2)

FFTnlogn Time

Fast Fourier Transformation• The values (x0,x1,x2,…,xn-1) are said to be special if:• There are n distinct values.• When you square each of them:

• The set collapses to n/2 distinct values.• Eg:

…, -3, -2, -1, 1, 2, 3, …square each of them

…, 9, 4, 1, 1, 4, 9, …collapse the set

1, 4, 9, …half as many elements.

Roots of Unity

Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• Special (x0,x1,x2,…,xn-1)



2,x12,x2

2,…,xn-12)

• n/2 distinct values


2,x12,x2

2,…,xn-12)

• n/2 distinct values• 1st friend’s output: • i y<even,i> = feven(xi

2)

• 2nd friend’s output: • i y<odd,i> = fodd(xi

2)


• T(n) = 2 T(n/2) + O(n) = O(n log n)That’s better

-3 3

feven(9) fodd(9)

9 9

feven(9) fodd(9)3f(3) -3f(-3)

Roots of Unity

Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• Special (x0,x1,x2,…,xn-1)



2,x12,x2

2,…,xn-12)



2,x12,x2

2,…,xn-12)


To meet preconditionthese also need to be special

Roots of Unity


• The set collapses to n/2 distinct values.• Which are also special

• Eg: …, -3, -2, -1, 1, 2, 3, …square each of them

…, 9, 4, 1, 1, 4, 9, …collapse the set

1, 4, 9, …

But these are not special.

square each of them 2, 16, 81, …

Roots of Unity


• The set collapses to n/2 distinct values.• Which are also special

• Eg: -i, -1, 1, isquare each of them

-1, 1, 1, -1collapse the set

-1, 1 square each of them

1, 1collapse the set

1

= -i,-1, 1, i are said to be

4th roots of unityBecause 4 = 1

Roots of Unity

Fast Fourier Transformation• is said to be an nth root of unity (in a field) if n = 1• (There should be n solutions of this polynomial)• Fermat’s Little Theorem: b≠0 bp-1 =mod p 1

says every nonzero element is an nth root of unity when n=p-1.

Roots of Unity

Fast Fourier Transformation• is said to be an nth root of unity (in a field) if n = 1• is said to be a generator of the field

if the numbers 1,,2, …,n-1 are all distinct• 1,,2, …,n-1 are then special (when n is even)

n/2+n/2-1,…,n/2+3,n/2+2,n/2+1,n/2+0 ,0,1,2,3,…,n/2-1 square each of them

n+n-2, …, n+6, n+4, n+2, n+0, 0,2,4,6, …,n-2 use n = 1

n-2, …, 6, 4, 2, 0, 0,2,4, 6, …,n-2 collapse the set

0,2,4, 6, …,n-2

We need these to be n/2 special values.

1st half2nd half

Roots of Unity

161

160

162

16316

416

5

166

167

168

169

1610

1611

1612

1613

1614

1615

= 1616 = 1

Fast Fourier Transformation16th roots of unity

-1 =

i

-i

(n/2)2 = 1

(n/4)2 = n/2 = -1

(3n/4)2 = n/2 = -1

These could be Z mod 17or complex numbers

×

rr

rθ

reθi = rcosθ + irsinθreθi × seαi = (rs)e(θ+α)i

Roots of Unity

Fast Fourier TransformationGoal: Proof f(θ) = g(θ)

f(0) = g(0)f’(0) = g’(0)

f’’(θ) = -f(θ) g’’(θ) = -g(θ)Proof by induction (over the reals) that f(θ) = g(θ)

f(θ) g(θ)

• For this θ, f(θ) = g(θ) and f’(θ) = g’(θ)• For next θ+, f(θ+) = g(θ+)• f’’(θ) = -f(θ) = -g(θ) =g’’(θ)• For next θ+, f’(θ+) = g’(θ+)

Roots of Unity

Fast Fourier Transformationf(θ) = reθi g(θ) = rcosθ + irsinθ

Goal: Proof f(θ) = g(θ)

f(0) = re0i = r g(0) = rcos0 + irsin0 = r

f’(θ) = ireθi g’(θ) = -rsinθ + ircosθ

f(0) = g(0)

f’(0) = ire0i = ir g’(0) = -rsin0 + ircos0 = ir

f’(0) = g’(0)

f’’(θ) = -reθi g’’(θ) = -rcosθ - rsinθ

= -f(θ) = -g(θ)

Roots of Unity

161

160

162

164

165

166

167

168

169

1610

1611

1612

1613

1614

1615


square each of them

163

and collapse

Roots of Unity

160

162

164

166

168

1610

1612

1614


Are these special?

square each of themand collapse

Roots of Unity

80

81

82

83

84

85

86

87


Are these special?


Roots of Unity

40

41

42

43


Are these special?


Roots of Unity

202

1


Are these special?

= 1


Roots of Unity

Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• (nth roots of unity n

i)

• My output: • (y0,y1,y2,…,yn-1)• yi = f(n

i)

• 1st friend’s input? • feven: (a0,a2,a4,…,an-2)• (n/2

th roots of unity n/2i)

• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1)• (n/2

th roots of unity n/2i)

• 1st friend’s output: • i y<even,i> = feven(n/2

i)

• 2nd friend’s output: • i y<odd,i> = fodd(n/2

i)


• T(n) = 2 T(n/2) + O(n) = O(n log n)Excellent

Roots of Unity

Fourier Transformation Algorithm FFT(y, , n):

Input: y = [a0,a1,a2,…,an-1] (Time Domain) = e2i 1/n (nth root of unity) n = # of samples (2r)Output: Y = [y0,y1,y2,…,yn-1] (Frequency Domain)% Separate even and odd indicesaeven = [a0,a2,a4,…,an-2] aodd = [a1,a1,a5,…,an-1]% Recurseyeven =FFT(aeven, 2, n/2) (2 = e2i 2/n ) yodd =FFT(aodd , 2, n/2)%CombiningFor i = 0 to n/2-1

y[i] = yeven[i] + i ∙yodd[i]y[i+n/2] = yeven[i] + i+n/2 ∙yodd[i]

Return(Y)

FFTCode

• A inverse Fourier transform changes the representation of a function

x0 x1 x2 x3 x4 … xn-1

y0 y1 y2 y3 y4 … yn-1

yi = f(xi)

• from the value f(xi) at key values xi.



Inverse FFTFast Fourier Transformation


Fast Fourier Transformation



(x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

(x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1

(xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1

(x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1

(xi)j

yi = f(xi)

Inverse FFT

xi = i

(x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

(x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1

(xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1

(x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1

(xi)j

(0)0 (0)1 (0)2 (0)3 … (0)n-1 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

(1)0 (1)1 (1)2 (1)3 … (1)n-1 (2)0 (2)1 (2)2 (2)3 … (2)n-1 (3)0 (3)1 (3)2 (3)3 … (3)n-1

(n-1)0 (n-1)1 (n-1)2 (n-1)3 … (n-1)n-1

(i)j

Inverse FFT

(i)j = ij

Vandermonde matrix

(0)0 (0)1 (0)2 (0)3 … (0)n-1 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

(1)0 (1)1 (1)2 (1)3 … (1)n-1 (2)0 (2)1 (2)2 (2)3 … (2)n-1 (3)0 (3)1 (3)2 (3)3 … (3)n-1

(n-1)0 (n-1)1 (n-1)2 (n-1)3 … (n-1)n-1

(i)j

0 0 0 0 … 0 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3

0 n-1 2n-2 3n-3 … (n-1)(n-1)

ij

Inverse FFT

V a = y

a = V-1 y

Vandermonde matrix

0 0 0 0 … 0 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3

0 n-1 2n-2 3n-3 … (n-1)(n-1)

ij

V-1 = 1/n V-1

Inverse FFT

Vandermonde matrix

0 0 0 0 … 0 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3

0 n-1 2n-2 3n-3 … (n-1)(n-1)

ij

V-1 = 1/n V-1

0 0 0 0 … 0 y0

y1

y2

y3

…

yn-1

a0

a1

a2

a3

…

an-1

=

0 -1 -2 -3 … -(n-1) 0 -2 -4 -6 … -(2n-2) 0 -3 -6 -9 … -3(n-3)

0 -(n-1) -(2n-2) -(3n-3) … -(n-1)(n-1)

-ij

1/n

Inverse FFT

• If w is an nth root of unity and a generator of the field

161

160

162

16316

416

5

166

167

168

169

1610

1611

1612

1613

1614

1615

= 1616 = 1-1 =

rr

Fast Fourier TransformationInverse

FFT

• The inverse w-1 of w is– then w-1=wn-1

– And is also an nth root of unity and a generator of the field – Proof: ww-1 = wwn-1 = wn=1

161

160

162

16316

416

5

166

167

168

169

1610

1611

1612

1613

1614

1615

= 1616 = 1-1 =

rr


FFT

• Cancellation Property:

161

160

162

16316

416

5

166

167

168

169

1610

1611

1612

1613

1614

1615

= 1616 = 1-1 =

rr

Fast Fourier Transformation0

1

0

n

j

kj

Inverse FFT

Vandermonde matrix

0 0 0 0 … 0 a0

a1

a2

a3

…

an-1

y0

y1

y2

y3

…

yn-1

=

0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3

0 n-1 2n-2 3n-3 … (n-1)(n-1)

ij

V-1 = 1/n V-1

0 0 0 0 … 0 y0

y1

y2

y3

…

yn-1

a0

a1

a2

a3

…

an-1

=

0 -1 -2 -3 … -(n-1) 0 -2 -4 -6 … -(2n-2) 0 -3 -6 -9 … -3(n-3)

0 -(n-1) -(2n-2) -(3n-3) … -(n-1)(n-1)

-ij

1/n

Inverse FFT

• The FFT and inverse FFT really are inverse operations• Proof: Let A=D -1D. We want to show that A=I, where

• If i=j, then

• If i and j are different, then

1

0

1],[

n

k

kjki

njiA

1111

],[1

0

01

0

nnnn

iiAn

k

n

k

kiki

Property)onCancellati(by 01

],[1

0

)(

n

k

kij

njiA


FFT

Fast Fourier Transformation• The FFT and inverse FFT can use the same hardware FFT• Input: <1, , a0,a1,a2,…,an-1 >• Output: <y0,y1,y2,…,yn-1 >

Inverse FFT• Input: <1/n, -1, y0,y1,y2,…,yn-1 >• Output: <a0,a1,a2,…,an-1 >

Inverse FFT

Modifies DFT frequency coefficient calculations:

ReX[ k ] = x[n] cos(2πkn/N) 0 < k < N/2 x[i] ε Real

ImX[ k ] = - x[n] sin(2πkn/N)

Uses complex and polar numbers as a shorthand:

Xk = ReX[ k ] + i ImX[ k ]

Xk = xn e –i2πkn/N = xn ωkn

Ʃ

Ʃn=0

n=0

N-1

N-1

N-1

Complex

Ʃn=0

N-1

r·e iθ = r·cosθ + i r·sinθ = r θ

Ʃn=0

N-1

ω = e –i2π/N

N = 2r

Fast Fourier TransformationSin & Cosbasis

1. Convert your N real sampled values to complex numbers by adding 0i to them

2. Feed this as the input to the FFT

3. Remove FFT output’s redundant information (i.e. all frequencies above N/2)

xn = xn + 0i 0 < n < N-1

0 N/2 N-1

“Negative” FrequencyReX

Even Symmetry About N/2 (fs/2)

0 N/2

ImX

Odd Symmetry About N/2 (fs/2)

“Negative” Frequency

N-1

Fast Fourier TransformationSin & Cosbasis

Fast Fourier Transformation FFTButterfly

(0)0 (0)1 (0)2 (0)3 … (0)n-1 (1)0 (1)1 (1)2 (1)3 … (1)n-1 (2)0 (2)1 (2)2 (2)3 … (2)n-1 (3)0 (3)1 (3)2 (3)3 … (3)n-1

(n-1)0 (n-1)1 (n-1)2 (n-1)3 …(n-1)n-1 ...

=

x0

x1

x2

x3

…xN-1

X0

X1

X2

X3

…XN-1

Behold the Vandermonde matrix!

But that’s O(N2) !!

The Ugly Math for the FFT

=

x0

x1

x2

x3

…xN-1

X0

X1

X2

X3

…XN-1

But if I multiply the exponents ...

0 n-1 2n-2 3n-3 … (n-1)(n-1)

0 0 0 0 … 0 0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3

...

But that’s still O(N2) !!

The Ugly Math for the FFT

Just watch! For example, if N=8 and I use the N roots of unity ...

0

0

0

0

0

0

0

0

0

1

2

3

4

5

6

7

02

4

6

8

10

12

14

0 3

6

9

12

15

18

21

0 4

8

12

16

20

24

28

05

10

15

20

25

30

35

0 6

12

18

24

30

36

42

0 7

14

21

28

35

42

49

=

x0

x1

x2

x3

x4

x5

x6

x7

X0

X1

X2

X3

X4

X5

X6

X7

4 0

1

23

56

7

p+4 = -p p = p mod 8

w = 8 = e -i2π/8

0 = 1 4 = -1

=

x0

x1

x2

x3

x4

x5

x6

x7

X0

X1

X2

X3

X4

X5

X6

X7

1 1111111

1 2

3

-1--2

-3

1 2

-1-2

1 2

-1-2

1 3

-2

-1-3

2

-

1

-1 1-1 1-1 1-1

1- 2

-3

-1 -2

3

1 -2

-1 2

1-2

-1 2

1-3

-2

--1 3

2

Now the 2nd half of each row either equals the 1st half or its

negative

+ + + -+ + + - + + + -+ + + -

=

x0

x1

x2

x3

x4

x5

x6

x7

X0

X1

X2

X3

X4

X5

X6

X7

1 1111111

1 3

-2

-1-3

2

-

1

-1 1-1 1-1 1-1

1-3

-2

--1 3

2

1 2

3

-1--2

-3

1- 2

-3

-1 -2

3

1 2

-1-2

1 2

-1-2

1 -2

-1 2

1-2

-1 2

x4x0

+ + + -+ + + - + + + -+ + + -

x0 and x4 have identical coefficients (ignoring sign) as do: x1 and x5

x2 and x6

x3 and x7

=

x0

x1

x2

x3

x4

x5

x6

x7

X0

X1

X2

X3

X4

X5

X6

X7


x2 and x6

x3 and x7

+ + + -+ + + - + + + -+ + + -

1 1111111

1 3

-2

-1-3

2

-

1

-1 1-1 1-1 1-1

1-3

-2

--1 3

2

1 2

3

-1--2

-3

1- 2

-3

-1 -2

3

1 2

-1-2

1 2

-1-2

1 -2

-1 2

1-2

-1 2

x1 x5

=

x0

x1

x2

x3

x4

x5

x6

x7

X0

X1

X2

X3

X4

X5

X6

X7


x2 and x6

x3 and x7

1 1111111

1 3

-2

-1-3

2

-

1

-1 1-1 1-1 1-1

1-3

-2

--1 3

2

1 2

3

-1--2

-3

1- 2

-3

-1 -2

3

1 2

-1-2

1 2

-1-2

1 -2

-1 2

1-2

-1 2

x2 x6

+ + + -+ + + - + + + -+ + + -

=

x0

x1

x2

x3

x4

x5

x6

x7

X0

X1

X2

X3

X4

X5

X6

X7


x2 and x6

x3 and x7

1 1111111

1 3

-2

-1-3

2

-

1

-1 1-1 1-1 1-1

1-3

-2

--1 3

2

1 2

3

-1--2

-3

1- 2

-3

-1 -2

3

1 2

-1-2

1 2

-1-2

1 -2

-1 2

1-2

-1 2

x3 x7

+ + + -+ + + - + + + -+ + + -

Now rewrite the matrix as equations in terms of: x0 ± x4, x2 ± x6 , x1 ± x5 , x3 ± x7

Oh my! Half the columns are gone. What’s next?

(x0 + x4) + (x2 + x6) + (x1 + x5) + (x3 + x7) = X0

(x0 - x4) + 2 (x2 - x6) + (x1 - x5) + 3 (x3 - x7) = X1

(x0 + x4) - (x2 + x6) + 2 (x1 + x5) - 2 (x3 + x7) = X2

(x0 - x4) - 2 (x2 - x6) + 3 (x1 - x5) + (x3 - x7) = X3

(x0 + x4) + (x2 + x6) - (x1 + x5) - (x3 + x7) = X4

(x0 - x4) + 2 (x2 - x6) - (x1 - x5) - 3 (x3 - x7) = X5

(x0 + x4) - (x2 + x6) - 2 (x1 + x5) + 2 (x3 + x7) = X6

(x0 - x4) - 2 (x2 - x6) - 3 (x1 - x5) - (x3 - x7) = X7

Think signal flow and construct the equations using the butterfly

operator:Ex. for

(x0 + x4) + (x2 + x6) + (x1 + x5) + (x3 + x7) = X0

(x0 - x4) + 2 (x2 - x6) + (x1 - x5) + 3 (x3 - x7) = X1

(x0 + x4) - (x2 + x6) + 2 (x1 + x5) - 2 (x3 + x7) = X2

(x0 - x4) - 2 (x2 - x6) + 3 (x1 - x5) + (x3 - x7) = X3

(x0 + x4) + (x2 + x6) - (x1 + x5) - (x3 + x7) = X4

(x0 - x4) + 2 (x2 - x6) - (x1 - x5) - 3 (x3 - x7) = X5

(x0 + x4) - (x2 + x6) - 2 (x1 + x5) + 2 (x3 + x7) = X6

(x0 - x4) - 2 (x2 - x6) - 3 (x1 - x5) - (x3 - x7) = X7

Ʃ

Ʃ

ωp

+

++

-

xo

x4

xo + ωp x4

xo - ωp x4

x0 ± x4

(ωp = 1)

Note - the butterfly has a shorthand notation of:

ωk

-1

xo

x4

xo + ωp x4

xo - ωp x4

(x0 + x4) + (x2 + x6) + (x1 + x5) + (x3 + x7) = X0

(x0 - x4) + 2 (x2 - x6) + (x1 - x5) + 3 (x3 - x7) = X1

(x0 + x4) - (x2 + x6) + 2 (x1 + x5) - 2 (x3 + x7) = X2

(x0 - x4) - 2 (x2 - x6) + 3 (x1 - x5) + (x3 - x7) = X3

(x0 + x4) + (x2 + x6) - (x1 + x5) - (x3 + x7) = X4

(x0 - x4) + 2 (x2 - x6) - (x1 - x5) - 3 (x3 - x7) = X5

(x0 + x4) - (x2 + x6) - 2 (x1 + x5) + 2 (x3 + x7) = X6

(x0 - x4) - 2 (x2 - x6) - 3 (x1 - x5) - (x3 - x7) = X7

Damn you and your re “cursed”

friends!

Using Bit Reverse Order and a tree of butterflies, my Decimation in Time

Algorithm can solve this in O (N log N)

No friends this time. They’d just be overhead invading my stack space.

If you’re lying, I’ll claim your soul!

Decimation in Time & Bit Reverse Order ( (rearranging the order of the N samples) 0000 001 010 011 100 101 110

111 0 00 1 2 3 4 5 6 7

0 2 4 6 1 3 5 7

0 4 2 6 1 5 3 7 0000 100 010 110 001 101 011 111

STAGE 1

2 – Point Butterfly




2 combined 2-Point

Butterflies

2 combined 2-Point

Butterflies

4 combined 2-Point

Butterflies

STAGE 3STAGE 2

x0

x4

x2

x6

x1

x5

x3

x7

X0

X1

X2

X3

X4

X5

X6

X7

FFT Block Diagram

-1

-1

-1

-1

-1

-1

X0

X1

X2

X3

X4

X5

X6

X7

-1

-1

-1

-1

-1

-1

x0

x4

x2

x6

x1

x5

x3

x7

81

82

83

80

80

80

80

828

0

80

80 8

2

Stepping Through the FFT

80 = 1 8

2 = 41

-1

-1

-1

-1

-1

X0

X1

X2

X3

X4

X5

X6

X7

-1

-1

-1

-1

-1

-1

81

82

83

80

80

80

80

828

0

80

80 8

2


-1

x0

x4

x2

x6

x1

x5

x3

x7

-1

-1

-1

-1

-1

-1

X0

X1

X2

X3

X4

X5

X6

X7

-1

-1

-1

-1

-1

-1

x0

x4

x2

x6

x1

x5

x3

x7

81

82

83

80

80

80

80

828

0

80

80 8

2


-1

-1

80

80

-1

-1

-1

-1

X0

X1

X2

X3

X4

X5

X6

X7

-1

-1

-1

-1

-1

-1

x0

x4

x2

x6

x1

x5

x3

x7

81

82

83

80

80

80

828

0

80

82


-1

80

-1

-1

80

80

-1

-1 -1

X0

X1

X2

X3

X4

X5

X6

X7

-1

-1

-1

-1

-1

-1

x0

x4

x2

x6

x1

x5

x3

x7

81

82

83

80

80

80

828

0

82

N/2


-1

80

-1

-1

80

80

-1

-1 -1

X0

X1

X2

X3

X4

X5

X6

X7

-1

-1

-1

-1

-1

-1

x0

x4

x2

x6

x1

x5

x3

x7

81

82

83

80

80

80

828

0

82

N/2 + N/2


-1

80

-1

-1

80

80

-1

X0

X1

X2

X3

X4

X5

X6

X7

-1

-1

-1

-1

-1

-1

x0

x4

x2

x6

x1

x5

x3

x7

81

82

83

80

80

80

82

N/2 + N/2


-1

-1

80

82

-1

80

-1

-1

80

80

-1

-1

-1

-1

-1

-1

-1

x0

x4

x2

x6

x1

x5

x3

x7

81

83

80

80

80

82

N/2 + N/2

82


-1

-1

80

82

X0

X1

X2

X3

X4

X5

X6

X7

-1

-1

80

80

-1

-1

X0

X1

X2

X3

X4

X5

X6

X7

-1

-1

-1

-1

-1

-1

x0

x4

x2

x6

x1

x5

x3

x7

81

83

80

80

80

80

82

N/2 + N/2 + N/2 O (N log N)

82


-1

-1

80

82

Algorithm FFT (ReX, ImX) Input: ReX[ ], ImX[ ] = real, imaginary parts of the time samples

Output: ReX[ ], ImX[ ] = cosine, sine coefficients of frequency domain

N = SizeOf( ReX )PutInBitReverseOrder (ReX, ImX) % time domain decomposition

% frequency domain synthesis (done in place)for k = 1 to log 2 N % Loop for each stage

Wre = 1; Wim = 0; θ = 2π/ 2k % Initialize stage constants

for j = 1 to 2k-1 % Loop for each sub DFT for i = j-1 to N-1 step 2k % Loop for each butterfly

ip = i + 2k-1

tmpRe = ReX[ip]·Wre - ImX[ip]·Wim tmpIm = ReX[ip]·Wim + ImX[ip]·Wre

ReX[ip] = ReX[ i ] - tmpRe ImX[ip] = ImX[ i ] - tempIm ReX[ i ] = ReX[ i ] + tempRe ImX[ i ] = ImX[ i ] + tempIm

next i

tempRe = Wre Wre = tmpRe·cos(θ) + Wim·sin(θ) Wim = - tmpRe·sin(θ) + Wim· cos(θ)

next jnext k

return (ReX, ImX) % ReX[ ],ImX[ ] return freq coeffs 0 to N-1

Polynomial Multiplicationf(x) = a0+a1x +a2x2 + … + an-1xn-1

g(x) = b0+b1x +b2x2 + … + bn-1xn-1

[f×g](x) = c0+c1x +c2x2 + … +c2n-2x2n-2

x5 coefficient: c5= a0×b5+a1×b4 + a2×b3 + … + a5×b0

Time = O(n2)

Too much

Convolution

Polynomial Multiplicationf(x) = a0+a1x +a2x2 + … + an-1xn-1

g(x) = b0+b1x +b2x2 + … + bn-1xn-1

[f×g](x) = c0+c1x +c2x2 + … +c2n-2x2n-2

Coefficient Domain aj Evaluation Domain yi

[a0,a1,a2 ,…,an-1]

[b0,b1,b2 ,…,bn-1]

Fast Fourier Transform takes O(nlogn) time!

yi = f(xi)zi = g(xi)

yi×zi = [g×f](xi)

Multipling values pointwise

takes O(n) time![c0,c1,c2 ,…,cn-1]

Multiplying Big IntegersX = 11…10100011101100010010 (N bits)Y = 10…01001100011001001111

X×Y = 10…1110110101001001010100010100110010011110

The high school algorithm takes O(N2) bit operations.Can we do it faster?

I hope so

See Recursion for one way to do it faster.This is another.

Grade School Revisited:How To Multiply Two

Numbers

Multiplying Big Integers

Multiplying Big IntegersX = 0011 … 1010 0011 1011 0001 0010

• Break into m = O(log N) bit blocks

m

Multiplying Big IntegersX = 0000 … 0000 0000 0011 1010 0011 1011 0001 0010

• Break into m = O(log N) bit blocks• Pad with zero• 2N bits to hold product• n blocks where n is a power of 2, ie n=2r.

• Let p be a prime• log p ≥ block size = m• p-1 is divisible by n, so Z mod p has n nth roots of unity.

• View each block as a finite field element in Z mod p. (no actual work)

n=2r blocks O(log p)

an … a7 a6 a5 a4 a3 a2 a1 a0

m

Multiplying Big IntegersX = 0000 … 0000 0000 0011 1010 0011 1011 0001 0010

• View as coefficients of a polynomial.• Note X = f(2m).• Same for Y = g(2m).• Multiply g×f using FFT in time O(nlogn).• Note X×Y = [g×f](2m).• Evaluate [g×f](2m) in time O(n) operations,

but each op could be on O(n) bit numbers for a total of O(n2) time.

f(x) = an-1xn-1 + … + a5x5 + a4x4 + a3x3 + a2x2 + a1x + a0

m

g(x) = bn-1xn-1 + … + b5x5 + b4x4 + b3x3 + b2x2 + b1x + b0

Multiplying Big Integers

X×Y = 0011 1010 0011 1011 0001 0010 1110 0011 0010

• Evaluate [g×f](2m) in time O(n).

[g×f](x) = cn-1xn-1 + … + c5x5 + c4x4 + c3x3 + c2x2 + c1x + c0

mO(log p)

• Some texts say the ci can just be shifted and joined.• Problem: The field elements may be too big.

Multiplying Big Integers• Evaluate [g×f](2m) in time O(n).

[g×f](x) = cnxn + … + c5x5 + c4x4 + c3x3 + c2x2 + c1x + c0

m

O(log p)

101011001101

110001000111

101011100100

010011

• Shift each ci by im.• Add

X×Y = 01 0101 0110 1011 1010 0001 1111 1011

Adding n numbers each n bits long takes O(n2)but here the numbers are sparse.

Multiplying Big Integers• Evaluate [g×f](2m) in time O(n).

[g×f](x) = cnxn + … + c5x5 + c4x4 + c3x3 + c2x2 + c1x + c0

m

O(log p)

101011001101

110001000111

101011100100

010011

• Shift each ci by im.• Add

X×Y = 01 0101 0110 1011 1010 0001 1111 1011

At each point, at most two numbers overlap Carry is at most one O(N) bit operations.

Multiplying Big IntegersX = 11…10100011101100010010 (N bits)Y = 10…01001100011001001111

X×Y = 10…1110110101001001010100010100110010011110

Suppose N is really really big.How many bit operations are needed?

• O(N logN)• O(N logN loglogN)• O(N logN loglogN logloglogN loglogloglogN …)

• FFT time• Time stated in text

• Time as far as I can see

Multiplying Big IntegersX = …101000111011000100101010001001010 …

• Input size = N bits• Field element size = N’ = log(N) bits• # ai = n = N/N’

• # of field ops = O(nlogn)• Time for × field op = ?

N’

X’ = 1010 0111 0110 0010 0101 0100 0100 1010

• Input size = N’ bits• Field element size = N’’ = log(N’) bits• # ai = n’ = N’/N’’

• # of field ops = O(n’logn’)• Time for × field op = ?• Total time:

N’’

• And so on …

O(N’ logN’ loglogN’ logloglogN’ …)

Multiplying Big IntegersX = …101000111011000100101010001001010 …• Input size = N bits• Field element size = N’ = log(N) bits• # ai = n = N/N’

• # of field ops = O(nlogn)• Time for × field op = ?• Total time:• = O( n logn ) × O(N’ logN’ loglogN’ logloglogN’ …) • = O(N/N’ logN/N’) × O(N’ loglogN logloglogN loglogloglogN …)• = O( N logN loglogN logloglogN loglogloglogN …)

N’

O(N’ logN’ loglogN’ logloglogN’ …)

The End

Fourier Transformations Jeff Edmonds York University COSC 6111 Change from Time to Polynomial Basis...

Documents

Transcript of Fourier Transformations Jeff Edmonds York University COSC 6111 Change from Time to Polynomial Basis...