Post on 28-Dec-2015
Fourier Transformations
Jeff Edmonds
York UniversityCOSC 6111
• Change from Time to Polynomial Basis• Evaluating & Interpolating• FFT in nlogn Time• Roots of Unity• Same FFT Code & Butterfly• Inverse FFT• Sin and Cos Basis• FFT Butterfly• Polynomial Multiplication• Integer Multiplication
Fourier Transformation Fourier Transform• are a change of basis from the time basis to
• sine/cosine basis• JPG• or polynomial basis
Purposes:• Signal Processing• Compressing data (eg images with .jpg)• Multiplying integers in n logn loglogn time.• Error correcting codes.• ….
http://www.dspguide.com/ch8.htm
The Scientist and Engineer's Guide toDigital Signal Processing
By Steven W. Smith, Ph.D.
Amazingly once you include complex numbers,
the FFT codefor sine/cosines and for polynomials
are the SAME.
Fourier Transformation
Instead of using sine and cosines
as the basis,
PolynomialBasis
Fourier Transformation
Instead of using sine and cosines
as the basis,
We use polynomials.
PolynomialBasis
Change of Basis: T([a1,a2,…,ad]) = [A1,A2,…,Ad]Changes the basis used to describe an object.
• The Time basis of a vector space:• A tuple <w1,w2,…,wd> of basis objects• Linearly independent• Spans the space uniquely
"v $[a1,a2,…,ad], v = a1w1+a2w2 +… + adwd• The Polynomial basis of a vector space:• A tuple <W1,W2,…,Wd> of basis objects• Linearly independent• Spans the space uniquely
"v $[A1,A2,…,Ad], v = A1W1+A2W2 +… + AdWd
PolynomialBasisFourier Transformation
[3,2]
v =
[a1,a2] = [11/5,32/5][A1,A2] =
Change of Basis: T([a1,a2,…,ad]) = [A1,A2,…,Ad]Changes the basis used to describe an object.
"v $[a1,a2,…,ad], v = a1w1 +a2w2 +… + adwd
"v $[A1,A2,…,Ad], v = A1W1+A2W2 +… + AdWd
=[w1,w2]=[ , ]
StandardBasis
v =
PolynomialBasis
NewBasis =[W1,W2]
=[ , ]W1[1]
W1[2]
W2[1]
W2[2]
[ ][ ] =[ ]W1[1]W1[2]
W2[1]W2[2]
a1a2
A1A2
Fourier Transformation
Fourier Transformation
f = a0∙B0+a1∙B1+a2∙B2 +… + an-1∙Bn-1
=[B0,B1,..]
f =
a0 =11/5
a1 =32/5
PolynomialBasis
=[ , ]
Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a1,a2,…,an-1]Changes the basis used to describe an object.
A discrete function
x
f(x)
f(x) = a0 +a1x +a2x2 + …+ an-1xn-1
coefficients of the polynomial
[a1,a2,…,an-1] =
PolynomialBasis
Have you seen Taylor Expansions of a Function?They show that functions f(x) can be expressed by specifying the coefficients of the polynomial.
F(x) = a0+a1x +a2x2 +a3x3 + …
Eg: f(x) = 1/(1-x) F(x) = 1+x +x2 +x3 + …
Fourier Transformation PolynomialBasis
xe
Have you seen Taylor Expansions of a Function?They show that functions f(x) can be expressed by specifying the coefficients of the polynomial.
F(x) = a0+a1x +a2x2 +a3x3 + …
Fourier Transformation PolynomialBasis
)sin(x
Have you seen Taylor Expansions of a Function?They show that functions f(x) can be expressed by specifying the coefficients of the polynomial.
F(x) = a0+a1x +a2x2 +a3x3 + …
Fourier Transformation PolynomialBasis
Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a1,a2,…,an-1]Changes the basis used to describe an object.
Time Basis =[ , ]
f =
Fourier Transformation
y[0]=3
y[1]=2
=[I0,I1,…]=[?,?]The time basis
x
Ij[x]
zeroone
j
"y $[y[0],y[1],…,y[n-1]], y = y[0]I0 +y[1]I1 +… + y[n-1]In-1
PolynomialBasis
A discrete function
x
f(x)
the value f(j) of the function at x=j.
y[j] =
the value f(j) of the function at x=j.
Fourier Transformation
Instead of evaluatingthe polynomials onj = 0, 1, 2,…, n-1
We evaluatethem on
x0, x1, x2, …, xn-1
These xj are special fixed values.
Later we set xj = e2i j/n
PolynomialBasis
the value f(j) of the function at x=j.
y[j] =
the value f(j) of the function at x=j.
Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a1,a2,…,an-1]Changes the basis used to describe an object.
Time Basis =[ , ]
f =
Fourier Transformation
y[0]=3
y[1]=2
=[I0,I1,…]=[?,?]The time basis
x
Ij[x]
zeroone
xj
"y $[y[0],y[1],…,y[n-1]], y = y[0]I0 +y[1]I1 +… + y[n-1]In-1
PolynomialBasis
A discrete function
x
f(x)
the value f(xj) of the function at x=j.
y[j] =
the value f(j) of the function at x=j.
x0 x1 x2 x3 x4 … xn-1
• A Fourier Transform is a change in basis.• It changes the representation of a function
• from the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1
• This amounts to evaluating f at these points.
Evaluating &Interpolating
x
• to the value f(xi) at key values xi.
x0 x1 x2 x3 x4 … xn-1
y0 y1 y2 y3 y4 … yn-1
Fourier Transformation
yi = f(xi)
• A Fourier Transform is a change in basis.• It changes the representation of a function
Fourier Transformation
• from the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1
• This amounts to evaluating f at these points.
(x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
(x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1
(xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1
(x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1
Vandermonde matrixInvertible if xi distinct.
Evaluating &Interpolating
yi = f(xi)
Fourier Transformation
• to the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1
• This amounts to interpolating these points.
• An Inverse Fourier Transform is the reverse. • It changes the representation of a function
Evaluating &Interpolating
x
• from the value f(xi) at key values xi.
x0 x1 x2 x3 x4 … xn-1
y0 y1 y2 y3 y4 … yn-1
yi = f(xi)
Fourier Transformation
• to the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1
• This amounts to interpolating these points.
Given a set of n points in the plane with distinct x-coordinates,
there is exactly one (n-1)-degree polynomial going through all these points.
• An Inverse Fourier Transform is the reverse. • It changes the representation of a function
Evaluating &Interpolating
f(x) = a0+x(a1+x(a2 + … + x(an-1) … ))
Fourier Transformation• My input: • (a0,a1,a2,…,an-1) & x
• My output: • f(x) = a0+a1x +a2x2 + … + an-1xn-1
1+2+3+..+n-1 multiplications?
n multiplications and n additions.
Given n distinct values (x0,x1,x2,…,xn-1).Can they all be done in less than n×O(n) time?
Evaluating &Interpolating
Fourier Transformation• My input: • (a0,a1,a2,…,an-1) & x
• My output: • f(x) = a0+a1x +a2x2 + … + an-1xn-1
Given n distinct values (x0,x1,x2,…,xn-1)Can they all be done in less than n×O(n) time?
(x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
(x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1
(xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1
(x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1
yi = f(xi)
I hope so
Evaluating &Interpolating
• The Fast Fourier Transform (FFT) is a very efficient algorithm for performing a discrete Fourier transform
• FFT principle first used by Gauss in 18?? (But was not interesting without computers)
• FFT algorithm published by Cooley & Tukey in 1965• In 1969, the 2048 point analysis of a seismic trace
took 13 ½ hours. Using the FFT, the same task on the same machine took 2.4 seconds!
Fast Fourier Transformation FFTnlogn Time
16 32 64 128 256 512 1024
70
60
50
40
30
20
10
0
Err
or
(pp
m)
DFT FFT
Not only do you get faster speed + in place memory processing but fewer
calculations means less round off errors
Maybe I should take CSE6111
after all!
Fast Fourier Transformation FFTnlogn Time
N DFT (N2) FFT (1.5N log N) faster
32 1,024 240 4.3
64 4,096 576 7.1
128 16,384 1,344 12.2
256 65,536 3,072 21.3
512 262,144 6,912 37.9
1024 1,048,576 15,360 68.2
2048 4,194,304 33,792 124.1
4096 16,777,216 73,728 227.6
Discrete Fourier Transform is too slow for real time!
Fast Fourier Transformation FFTnlogn Time
Divide & Conquer - Friends - Recursion.• Trust your friends to solve any subinstance: • as long as smaller and • is an instance to the same problem.
Fast Fourier Transformation
My instance
My friend’sInstance
My friend’sInstance
My friend’sInstance
FFTnlogn Time
f(x) = a0+a1x +a2x2 + … + an-1xn-1
Fast Fourier Transformation• My input: (start with one x)
• (a0,a1,a2,…,an-1) & x
= a0+a2x2+a4x4 + … + an-2xn-2
+ a1x+a3x3+a5x5 + … + an-1xn-1
= a0+a2x2+a4x4 + … + an-2xn-2
+ x( a1+a3x2+a5x4 + … + an-1xn-2 ) = feven(x2) + x( fodd(x2) )
• My output: • f(x) = a0+a1x +a2x2 + … + an-1xn-1
• 1st friend’s input? • feven: (a0,a2,a4,…,an-2) & ?
• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1) & ?
f(x) = feven(x2) + x fodd(x2)
feven(z) = a0+a2z+a4z2+a6z3+ … + an-2zn/2-1
FFTnlogn Time
Fast Fourier Transformation• My input: (start with one x)
• (a0,a1,a2,…,an-1) & x
• My output: • f(x) = a0+a1x +a2x2 + … + an-1xn-1
• 1st friend’s input? • feven: (a0,a2,a4,…,an-2) & x2
• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1) & x2
• 1st friend’s output: • yeven = feven(x2)
• 2nd friend’s output: • yodd = fodd(x2)
f(x) = feven(x2) + x fodd(x2)
• My output: • f(x) = yeven + x yodd
• T(n) = 2 T(n/2) + O(1) = O(n)Ok. So it takes O(n) time to evaluate.
FFTnlogn Time
Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• (x0,x1,x2,…,xn-1)
• My output: • (y0,y1,y2,…,yn-1)• yi = f(xi)
• 1st friend’s input? • feven: (a0,a2,a4,…,an-2)• (x0
2,x12,x2
2,…,xn-12)
• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1)• (x0
2,x12,x2
2,…,xn-12)
• 1st friend’s output: • i y<even,i> = feven(xi
2)
• 2nd friend’s output: • i y<odd,i> = fodd(xi
2)
• My output: • i f(xi) = y<even,i> + xi y<odd,i>
• T(n) = 2 T(n/2) + O(n) = O(n log n)
Wow! That was easy.
FFTnlogn Time
Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• (x0,x1,x2,…,xn-1)
• 1st friend’s input? • feven: (a0,a2,a4,…,an-2)• (x0
2,x12,x2
2,…,xn-12)
Oops
n coefficientsn values of x
n/2 coefficientsn values of x
Does not meet precondition!
FFTnlogn Time
Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• (x0,x1,x2,…,xn-1)
• My output: • (y0,y1,y2,…,yn-1)• yi = f(xi)
• 1st friend’s input? • feven: (a0,a2,a4,…,an-2)• (x0
2,x12,x2
2,…,xn/2-12)
• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1)• (x0
2,x12,x2
2,…,xn/2-12)
• My output: • i f(xi) = y<even,i> + xi y<odd,i>
• T(n) = 4 T(n/2) + O(n)That’s no good.
• 3rd friend’s input? • feven: (a0,a2,a4,…,an-2)• (xn/2
2,xn/2+12,…,xn-1
2)
• 4th friend’s input? • fodd: (a1,a3,a5,…,an-1)• (xn/2
2,xn/2+12,…,xn-1
2)
= O(n2)
FFTnlogn Time
Fast Fourier Transformation• The values (x0,x1,x2,…,xn-1) are said to be special if:• There are n distinct values.• When you square each of them:
• The set collapses to n/2 distinct values.• Eg:
…, -3, -2, -1, 1, 2, 3, …square each of them
…, 9, 4, 1, 1, 4, 9, …collapse the set
1, 4, 9, …half as many elements.
Roots of Unity
Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• Special (x0,x1,x2,…,xn-1)
• My output: • (y0,y1,y2,…,yn-1)• yi = f(xi)
• 1st friend’s input? • feven: (a0,a2,a4,…,an-2)• (x0
2,x12,x2
2,…,xn-12)
• n/2 distinct values
• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1)• (x0
2,x12,x2
2,…,xn-12)
• n/2 distinct values• 1st friend’s output: • i y<even,i> = feven(xi
2)
• 2nd friend’s output: • i y<odd,i> = fodd(xi
2)
• My output: • i f(xi) = y<even,i> + xi y<odd,i>
• T(n) = 2 T(n/2) + O(n) = O(n log n)That’s better
-3 3
feven(9) fodd(9)
9 9
feven(9) fodd(9)3f(3) -3f(-3)
Roots of Unity
Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• Special (x0,x1,x2,…,xn-1)
• My output: • (y0,y1,y2,…,yn-1)• yi = f(xi)
• 1st friend’s input? • feven: (a0,a2,a4,…,an-2)• (x0
2,x12,x2
2,…,xn-12)
• n/2 distinct values
• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1)• (x0
2,x12,x2
2,…,xn-12)
• n/2 distinct values
To meet preconditionthese also need to be special
Roots of Unity
Fast Fourier Transformation• The values (x0,x1,x2,…,xn-1) are said to be special if:• There are n distinct values.• When you square each of them:
• The set collapses to n/2 distinct values.• Which are also special
• Eg: …, -3, -2, -1, 1, 2, 3, …square each of them
…, 9, 4, 1, 1, 4, 9, …collapse the set
1, 4, 9, …
But these are not special.
square each of them 2, 16, 81, …
Roots of Unity
Fast Fourier Transformation• The values (x0,x1,x2,…,xn-1) are said to be special if:• There are n distinct values.• When you square each of them:
• The set collapses to n/2 distinct values.• Which are also special
• Eg: -i, -1, 1, isquare each of them
-1, 1, 1, -1collapse the set
-1, 1 square each of them
1, 1collapse the set
1
= -i,-1, 1, i are said to be
4th roots of unityBecause 4 = 1
Roots of Unity
Fast Fourier Transformation• is said to be an nth root of unity (in a field) if n = 1• (There should be n solutions of this polynomial)• Fermat’s Little Theorem: b≠0 bp-1 =mod p 1
says every nonzero element is an nth root of unity when n=p-1.
Roots of Unity
Fast Fourier Transformation• is said to be an nth root of unity (in a field) if n = 1• is said to be a generator of the field
if the numbers 1,,2, …,n-1 are all distinct• 1,,2, …,n-1 are then special (when n is even)
n/2+n/2-1,…,n/2+3,n/2+2,n/2+1,n/2+0 ,0,1,2,3,…,n/2-1 square each of them
n+n-2, …, n+6, n+4, n+2, n+0, 0,2,4,6, …,n-2 use n = 1
n-2, …, 6, 4, 2, 0, 0,2,4, 6, …,n-2 collapse the set
0,2,4, 6, …,n-2
We need these to be n/2 special values.
1st half2nd half
Roots of Unity
161
160
162
16316
416
5
166
167
168
169
1610
1611
1612
1613
1614
1615
= 1616 = 1
Fast Fourier Transformation16th roots of unity
-1 =
i
-i
(n/2)2 = 1
(n/4)2 = n/2 = -1
(3n/4)2 = n/2 = -1
These could be Z mod 17or complex numbers
×
rr
rθ
reθi = rcosθ + irsinθreθi × seαi = (rs)e(θ+α)i
Roots of Unity
Fast Fourier TransformationGoal: Proof f(θ) = g(θ)
f(0) = g(0)f’(0) = g’(0)
f’’(θ) = -f(θ) g’’(θ) = -g(θ)Proof by induction (over the reals) that f(θ) = g(θ)
f(θ) g(θ)
• For this θ, f(θ) = g(θ) and f’(θ) = g’(θ)• For next θ+, f(θ+) = g(θ+)• f’’(θ) = -f(θ) = -g(θ) =g’’(θ)• For next θ+, f’(θ+) = g’(θ+)
Roots of Unity
Fast Fourier Transformationf(θ) = reθi g(θ) = rcosθ + irsinθ
Goal: Proof f(θ) = g(θ)
f(0) = re0i = r g(0) = rcos0 + irsin0 = r
f’(θ) = ireθi g’(θ) = -rsinθ + ircosθ
f(0) = g(0)
f’(0) = ire0i = ir g’(0) = -rsin0 + ircos0 = ir
f’(0) = g’(0)
f’’(θ) = -reθi g’’(θ) = -rcosθ - rsinθ
= -f(θ) = -g(θ)
Roots of Unity
161
160
162
164
165
166
167
168
169
1610
1611
1612
1613
1614
1615
Fast Fourier Transformation16th roots of unity
square each of them
163
and collapse
Roots of Unity
160
162
164
166
168
1610
1612
1614
Fast Fourier Transformation16th roots of unity
Are these special?
square each of themand collapse
Roots of Unity
80
81
82
83
84
85
86
87
Fast Fourier Transformation8th roots of unity
Are these special?
square each of themand collapse
Roots of Unity
40
41
42
43
Fast Fourier Transformation4th roots of unity
Are these special?
square each of themand collapse
Roots of Unity
202
1
Fast Fourier Transformation2th roots of unity
Are these special?
= 1
square each of themand collapse
Roots of Unity
Fast Fourier Transformation• My input: • (a0,a1,a2,…,an-1)• (nth roots of unity n
i)
• My output: • (y0,y1,y2,…,yn-1)• yi = f(n
i)
• 1st friend’s input? • feven: (a0,a2,a4,…,an-2)• (n/2
th roots of unity n/2i)
• 2nd friend’s input? • fodd: (a1,a3,a5,…,an-1)• (n/2
th roots of unity n/2i)
• 1st friend’s output: • i y<even,i> = feven(n/2
i)
• 2nd friend’s output: • i y<odd,i> = fodd(n/2
i)
• My output: • i f(xi) = y<even,i> + xi y<odd,i>
• T(n) = 2 T(n/2) + O(n) = O(n log n)Excellent
Roots of Unity
Fourier Transformation Algorithm FFT(y, , n):
Input: y = [a0,a1,a2,…,an-1] (Time Domain) = e2i 1/n (nth root of unity) n = # of samples (2r)Output: Y = [y0,y1,y2,…,yn-1] (Frequency Domain)% Separate even and odd indicesaeven = [a0,a2,a4,…,an-2] aodd = [a1,a1,a5,…,an-1]% Recurseyeven =FFT(aeven, 2, n/2) (2 = e2i 2/n ) yodd =FFT(aodd , 2, n/2)%CombiningFor i = 0 to n/2-1
y[i] = yeven[i] + i ∙yodd[i]y[i+n/2] = yeven[i] + i+n/2 ∙yodd[i]
Return(Y)
FFTCode
• A inverse Fourier transform changes the representation of a function
x0 x1 x2 x3 x4 … xn-1
y0 y1 y2 y3 y4 … yn-1
yi = f(xi)
• from the value f(xi) at key values xi.
• to the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1
• This amounts to interpolating these points.
Inverse FFTFast Fourier Transformation
• A Fourier Transform is a change in basis.• It changes the representation of a function
Fast Fourier Transformation
• from the coefficients of the polynomial f(x) = a0+a1x +a2x2 + … + an-1xn-1
• This amounts to evaluating f at these points.
(x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
(x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1
(xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1
(x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1
(xi)j
yi = f(xi)
Inverse FFT
xi = i
(x0)0 (x0)1 (x0)2 (x0)3 … (x0)n-1 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
(x1)0 (x1)1 (x1)2 (x1)3 … (x1)n-1
(xn-1)0(xn-1)1(xn-1)2 (xn-1)3…(xn-1)n-1
(x2)0 (x2)1 (x2)2 (x2)3 … (x2)n-1 (x3)0 (x3)1 (x3)2 (x3)3 … (x3)n-1
(xi)j
(0)0 (0)1 (0)2 (0)3 … (0)n-1 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
(1)0 (1)1 (1)2 (1)3 … (1)n-1 (2)0 (2)1 (2)2 (2)3 … (2)n-1 (3)0 (3)1 (3)2 (3)3 … (3)n-1
(n-1)0 (n-1)1 (n-1)2 (n-1)3 … (n-1)n-1
(i)j
Inverse FFT
(i)j = ij
Vandermonde matrix
(0)0 (0)1 (0)2 (0)3 … (0)n-1 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
(1)0 (1)1 (1)2 (1)3 … (1)n-1 (2)0 (2)1 (2)2 (2)3 … (2)n-1 (3)0 (3)1 (3)2 (3)3 … (3)n-1
(n-1)0 (n-1)1 (n-1)2 (n-1)3 … (n-1)n-1
(i)j
0 0 0 0 … 0 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3
0 n-1 2n-2 3n-3 … (n-1)(n-1)
ij
Inverse FFT
V a = y
a = V-1 y
Vandermonde matrix
0 0 0 0 … 0 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3
0 n-1 2n-2 3n-3 … (n-1)(n-1)
ij
V-1 = 1/n V-1
Inverse FFT
Vandermonde matrix
0 0 0 0 … 0 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3
0 n-1 2n-2 3n-3 … (n-1)(n-1)
ij
V-1 = 1/n V-1
0 0 0 0 … 0 y0
y1
y2
y3
…
yn-1
a0
a1
a2
a3
…
an-1
=
0 -1 -2 -3 … -(n-1) 0 -2 -4 -6 … -(2n-2) 0 -3 -6 -9 … -3(n-3)
0 -(n-1) -(2n-2) -(3n-3) … -(n-1)(n-1)
-ij
1/n
Inverse FFT
• If w is an nth root of unity and a generator of the field
161
160
162
16316
416
5
166
167
168
169
1610
1611
1612
1613
1614
1615
= 1616 = 1-1 =
rr
Fast Fourier TransformationInverse
FFT
• The inverse w-1 of w is– then w-1=wn-1
– And is also an nth root of unity and a generator of the field – Proof: ww-1 = wwn-1 = wn=1
161
160
162
16316
416
5
166
167
168
169
1610
1611
1612
1613
1614
1615
= 1616 = 1-1 =
rr
Fast Fourier TransformationInverse
FFT
• Cancellation Property:
161
160
162
16316
416
5
166
167
168
169
1610
1611
1612
1613
1614
1615
= 1616 = 1-1 =
rr
Fast Fourier Transformation0
1
0
n
j
kj
Inverse FFT
Vandermonde matrix
0 0 0 0 … 0 a0
a1
a2
a3
…
an-1
y0
y1
y2
y3
…
yn-1
=
0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3
0 n-1 2n-2 3n-3 … (n-1)(n-1)
ij
V-1 = 1/n V-1
0 0 0 0 … 0 y0
y1
y2
y3
…
yn-1
a0
a1
a2
a3
…
an-1
=
0 -1 -2 -3 … -(n-1) 0 -2 -4 -6 … -(2n-2) 0 -3 -6 -9 … -3(n-3)
0 -(n-1) -(2n-2) -(3n-3) … -(n-1)(n-1)
-ij
1/n
Inverse FFT
• The FFT and inverse FFT really are inverse operations• Proof: Let A=D -1D. We want to show that A=I, where
• If i=j, then
• If i and j are different, then
1
0
1],[
n
k
kjki
njiA
1111
],[1
0
01
0
nnnn
iiAn
k
n
k
kiki
Property)onCancellati(by 01
],[1
0
)(
n
k
kij
njiA
Fast Fourier TransformationInverse
FFT
Fast Fourier Transformation• The FFT and inverse FFT can use the same hardware FFT• Input: <1, , a0,a1,a2,…,an-1 >• Output: <y0,y1,y2,…,yn-1 >
Inverse FFT• Input: <1/n, -1, y0,y1,y2,…,yn-1 >• Output: <a0,a1,a2,…,an-1 >
Inverse FFT
Modifies DFT frequency coefficient calculations:
ReX[ k ] = x[n] cos(2πkn/N) 0 < k < N/2 x[i] ε Real
ImX[ k ] = - x[n] sin(2πkn/N)
Uses complex and polar numbers as a shorthand:
Xk = ReX[ k ] + i ImX[ k ]
Xk = xn e –i2πkn/N = xn ωkn
Ʃ
Ʃn=0
n=0
N-1
N-1
N-1
Complex
Ʃn=0
N-1
r·e iθ = r·cosθ + i r·sinθ = r θ
Ʃn=0
N-1
ω = e –i2π/N
N = 2r
Fast Fourier TransformationSin & Cosbasis
1. Convert your N real sampled values to complex numbers by adding 0i to them
2. Feed this as the input to the FFT
3. Remove FFT output’s redundant information (i.e. all frequencies above N/2)
xn = xn + 0i 0 < n < N-1
0 N/2 N-1
“Negative” FrequencyReX
Even Symmetry About N/2 (fs/2)
0 N/2
ImX
Odd Symmetry About N/2 (fs/2)
“Negative” Frequency
N-1
Fast Fourier TransformationSin & Cosbasis
Fast Fourier Transformation FFTButterfly
(0)0 (0)1 (0)2 (0)3 … (0)n-1 (1)0 (1)1 (1)2 (1)3 … (1)n-1 (2)0 (2)1 (2)2 (2)3 … (2)n-1 (3)0 (3)1 (3)2 (3)3 … (3)n-1
(n-1)0 (n-1)1 (n-1)2 (n-1)3 …(n-1)n-1 ...
=
x0
x1
x2
x3
…xN-1
X0
X1
X2
X3
…XN-1
Behold the Vandermonde matrix!
But that’s O(N2) !!
The Ugly Math for the FFT
=
x0
x1
x2
x3
…xN-1
X0
X1
X2
X3
…XN-1
But if I multiply the exponents ...
0 n-1 2n-2 3n-3 … (n-1)(n-1)
0 0 0 0 … 0 0 1 2 3 … n-1 0 2 4 6 … 2n-2 0 3 6 9 … 3n-3
...
But that’s still O(N2) !!
The Ugly Math for the FFT
Just watch! For example, if N=8 and I use the N roots of unity ...
0
0
0
0
0
0
0
0
0
1
2
3
4
5
6
7
02
4
6
8
10
12
14
0 3
6
9
12
15
18
21
0 4
8
12
16
20
24
28
05
10
15
20
25
30
35
0 6
12
18
24
30
36
42
0 7
14
21
28
35
42
49
=
x0
x1
x2
x3
x4
x5
x6
x7
X0
X1
X2
X3
X4
X5
X6
X7
4 0
1
23
56
7
p+4 = -p p = p mod 8
w = 8 = e -i2π/8
0 = 1 4 = -1
=
x0
x1
x2
x3
x4
x5
x6
x7
X0
X1
X2
X3
X4
X5
X6
X7
1 1111111
1 2
3
-1--2
-3
1 2
-1-2
1 2
-1-2
1 3
-2
-1-3
2
-
1
-1 1-1 1-1 1-1
1- 2
-3
-1 -2
3
1 -2
-1 2
1-2
-1 2
1-3
-2
--1 3
2
Now the 2nd half of each row either equals the 1st half or its
negative
+ + + -+ + + - + + + -+ + + -
=
x0
x1
x2
x3
x4
x5
x6
x7
X0
X1
X2
X3
X4
X5
X6
X7
1 1111111
1 3
-2
-1-3
2
-
1
-1 1-1 1-1 1-1
1-3
-2
--1 3
2
1 2
3
-1--2
-3
1- 2
-3
-1 -2
3
1 2
-1-2
1 2
-1-2
1 -2
-1 2
1-2
-1 2
x4x0
+ + + -+ + + - + + + -+ + + -
x0 and x4 have identical coefficients (ignoring sign) as do: x1 and x5
x2 and x6
x3 and x7
=
x0
x1
x2
x3
x4
x5
x6
x7
X0
X1
X2
X3
X4
X5
X6
X7
x0 and x4 have identical coefficients (ignoring sign) as do: x1 and x5
x2 and x6
x3 and x7
+ + + -+ + + - + + + -+ + + -
1 1111111
1 3
-2
-1-3
2
-
1
-1 1-1 1-1 1-1
1-3
-2
--1 3
2
1 2
3
-1--2
-3
1- 2
-3
-1 -2
3
1 2
-1-2
1 2
-1-2
1 -2
-1 2
1-2
-1 2
x1 x5
=
x0
x1
x2
x3
x4
x5
x6
x7
X0
X1
X2
X3
X4
X5
X6
X7
x0 and x4 have identical coefficients (ignoring sign) as do: x1 and x5
x2 and x6
x3 and x7
1 1111111
1 3
-2
-1-3
2
-
1
-1 1-1 1-1 1-1
1-3
-2
--1 3
2
1 2
3
-1--2
-3
1- 2
-3
-1 -2
3
1 2
-1-2
1 2
-1-2
1 -2
-1 2
1-2
-1 2
x2 x6
+ + + -+ + + - + + + -+ + + -
=
x0
x1
x2
x3
x4
x5
x6
x7
X0
X1
X2
X3
X4
X5
X6
X7
x0 and x4 have identical coefficients (ignoring sign) as do: x1 and x5
x2 and x6
x3 and x7
1 1111111
1 3
-2
-1-3
2
-
1
-1 1-1 1-1 1-1
1-3
-2
--1 3
2
1 2
3
-1--2
-3
1- 2
-3
-1 -2
3
1 2
-1-2
1 2
-1-2
1 -2
-1 2
1-2
-1 2
x3 x7
+ + + -+ + + - + + + -+ + + -
Now rewrite the matrix as equations in terms of: x0 ± x4, x2 ± x6 , x1 ± x5 , x3 ± x7
Oh my! Half the columns are gone. What’s next?
(x0 + x4) + (x2 + x6) + (x1 + x5) + (x3 + x7) = X0
(x0 - x4) + 2 (x2 - x6) + (x1 - x5) + 3 (x3 - x7) = X1
(x0 + x4) - (x2 + x6) + 2 (x1 + x5) - 2 (x3 + x7) = X2
(x0 - x4) - 2 (x2 - x6) + 3 (x1 - x5) + (x3 - x7) = X3
(x0 + x4) + (x2 + x6) - (x1 + x5) - (x3 + x7) = X4
(x0 - x4) + 2 (x2 - x6) - (x1 - x5) - 3 (x3 - x7) = X5
(x0 + x4) - (x2 + x6) - 2 (x1 + x5) + 2 (x3 + x7) = X6
(x0 - x4) - 2 (x2 - x6) - 3 (x1 - x5) - (x3 - x7) = X7
Think signal flow and construct the equations using the butterfly
operator:Ex. for
(x0 + x4) + (x2 + x6) + (x1 + x5) + (x3 + x7) = X0
(x0 - x4) + 2 (x2 - x6) + (x1 - x5) + 3 (x3 - x7) = X1
(x0 + x4) - (x2 + x6) + 2 (x1 + x5) - 2 (x3 + x7) = X2
(x0 - x4) - 2 (x2 - x6) + 3 (x1 - x5) + (x3 - x7) = X3
(x0 + x4) + (x2 + x6) - (x1 + x5) - (x3 + x7) = X4
(x0 - x4) + 2 (x2 - x6) - (x1 - x5) - 3 (x3 - x7) = X5
(x0 + x4) - (x2 + x6) - 2 (x1 + x5) + 2 (x3 + x7) = X6
(x0 - x4) - 2 (x2 - x6) - 3 (x1 - x5) - (x3 - x7) = X7
Ʃ
Ʃ
ωp
+
++
-
xo
x4
xo + ωp x4
xo - ωp x4
x0 ± x4
(ωp = 1)
Note - the butterfly has a shorthand notation of:
ωk
-1
xo
x4
xo + ωp x4
xo - ωp x4
(x0 + x4) + (x2 + x6) + (x1 + x5) + (x3 + x7) = X0
(x0 - x4) + 2 (x2 - x6) + (x1 - x5) + 3 (x3 - x7) = X1
(x0 + x4) - (x2 + x6) + 2 (x1 + x5) - 2 (x3 + x7) = X2
(x0 - x4) - 2 (x2 - x6) + 3 (x1 - x5) + (x3 - x7) = X3
(x0 + x4) + (x2 + x6) - (x1 + x5) - (x3 + x7) = X4
(x0 - x4) + 2 (x2 - x6) - (x1 - x5) - 3 (x3 - x7) = X5
(x0 + x4) - (x2 + x6) - 2 (x1 + x5) + 2 (x3 + x7) = X6
(x0 - x4) - 2 (x2 - x6) - 3 (x1 - x5) - (x3 - x7) = X7
Damn you and your re “cursed”
friends!
Using Bit Reverse Order and a tree of butterflies, my Decimation in Time
Algorithm can solve this in O (N log N)
No friends this time. They’d just be overhead invading my stack space.
If you’re lying, I’ll claim your soul!
Decimation in Time & Bit Reverse Order ( (rearranging the order of the N samples) 0000 001 010 011 100 101 110
111 0 00 1 2 3 4 5 6 7
0 2 4 6 1 3 5 7
0 4 2 6 1 5 3 7 0000 100 010 110 001 101 011 111
STAGE 1
2 – Point Butterfly
2 – Point Butterfly
2 – Point Butterfly
2 – Point Butterfly
2 combined 2-Point
Butterflies
2 combined 2-Point
Butterflies
4 combined 2-Point
Butterflies
STAGE 3STAGE 2
x0
x4
x2
x6
x1
x5
x3
x7
X0
X1
X2
X3
X4
X5
X6
X7
FFT Block Diagram
-1
-1
-1
-1
-1
-1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
80
828
0
80
80 8
2
Stepping Through the FFT
80 = 1 8
2 = 41
-1
-1
-1
-1
-1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
81
82
83
80
80
80
80
828
0
80
80 8
2
Stepping Through the FFT
-1
x0
x4
x2
x6
x1
x5
x3
x7
-1
-1
-1
-1
-1
-1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
80
828
0
80
80 8
2
Stepping Through the FFT
-1
-1
80
80
-1
-1
-1
-1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
828
0
80
82
Stepping Through the FFT
-1
80
-1
-1
80
80
-1
-1 -1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
828
0
82
N/2
Stepping Through the FFT
-1
80
-1
-1
80
80
-1
-1 -1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
828
0
82
N/2
Stepping Through the FFT
-1
80
-1
-1
80
80
-1
-1 -1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
828
0
82
N/2
Stepping Through the FFT
-1
80
-1
-1
80
80
-1
-1 -1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
828
0
82
N/2
Stepping Through the FFT
-1
80
-1
-1
80
80
-1
-1 -1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
828
0
82
N/2 + N/2
Stepping Through the FFT
-1
80
-1
-1
80
80
-1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
82
N/2 + N/2
Stepping Through the FFT
-1
-1
80
82
-1
80
-1
-1
80
80
-1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
82
83
80
80
80
82
N/2 + N/2
Stepping Through the FFT
-1
-1
80
82
-1
80
-1
-1
80
80
-1
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
83
80
80
80
82
N/2 + N/2
82
Stepping Through the FFT
-1
-1
80
82
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
80
80
-1
-1
X0
X1
X2
X3
X4
X5
X6
X7
-1
-1
-1
-1
-1
-1
x0
x4
x2
x6
x1
x5
x3
x7
81
83
80
80
80
80
82
N/2 + N/2 + N/2 O (N log N)
82
Stepping Through the FFT
-1
-1
80
82
Algorithm FFT (ReX, ImX) Input: ReX[ ], ImX[ ] = real, imaginary parts of the time samples
Output: ReX[ ], ImX[ ] = cosine, sine coefficients of frequency domain
N = SizeOf( ReX )PutInBitReverseOrder (ReX, ImX) % time domain decomposition
% frequency domain synthesis (done in place)for k = 1 to log 2 N % Loop for each stage
Wre = 1; Wim = 0; θ = 2π/ 2k % Initialize stage constants
for j = 1 to 2k-1 % Loop for each sub DFT for i = j-1 to N-1 step 2k % Loop for each butterfly
ip = i + 2k-1
tmpRe = ReX[ip]·Wre - ImX[ip]·Wim tmpIm = ReX[ip]·Wim + ImX[ip]·Wre
ReX[ip] = ReX[ i ] - tmpRe ImX[ip] = ImX[ i ] - tempIm ReX[ i ] = ReX[ i ] + tempRe ImX[ i ] = ImX[ i ] + tempIm
next i
tempRe = Wre Wre = tmpRe·cos(θ) + Wim·sin(θ) Wim = - tmpRe·sin(θ) + Wim· cos(θ)
next jnext k
return (ReX, ImX) % ReX[ ],ImX[ ] return freq coeffs 0 to N-1
Polynomial Multiplicationf(x) = a0+a1x +a2x2 + … + an-1xn-1
g(x) = b0+b1x +b2x2 + … + bn-1xn-1
[f×g](x) = c0+c1x +c2x2 + … +c2n-2x2n-2
x5 coefficient: c5= a0×b5+a1×b4 + a2×b3 + … + a5×b0
Time = O(n2)
Too much
Convolution
Polynomial Multiplicationf(x) = a0+a1x +a2x2 + … + an-1xn-1
g(x) = b0+b1x +b2x2 + … + bn-1xn-1
[f×g](x) = c0+c1x +c2x2 + … +c2n-2x2n-2
Coefficient Domain aj Evaluation Domain yi
[a0,a1,a2 ,…,an-1]
[b0,b1,b2 ,…,bn-1]
Fast Fourier Transform takes O(nlogn) time!
yi = f(xi)zi = g(xi)
yi×zi = [g×f](xi)
Multipling values pointwise
takes O(n) time![c0,c1,c2 ,…,cn-1]
Multiplying Big IntegersX = 11…10100011101100010010 (N bits)Y = 10…01001100011001001111
X×Y = 10…1110110101001001010100010100110010011110
The high school algorithm takes O(N2) bit operations.Can we do it faster?
I hope so
See Recursion for one way to do it faster.This is another.
Grade School Revisited:How To Multiply Two
Numbers
Multiplying Big Integers
Multiplying Big IntegersX = 0011 … 1010 0011 1011 0001 0010
• Break into m = O(log N) bit blocks
m
Multiplying Big IntegersX = 0000 … 0000 0000 0011 1010 0011 1011 0001 0010
• Break into m = O(log N) bit blocks• Pad with zero• 2N bits to hold product• n blocks where n is a power of 2, ie n=2r.
• Let p be a prime• log p ≥ block size = m• p-1 is divisible by n, so Z mod p has n nth roots of unity.
• View each block as a finite field element in Z mod p. (no actual work)
n=2r blocks O(log p)
an … a7 a6 a5 a4 a3 a2 a1 a0
m
Multiplying Big IntegersX = 0000 … 0000 0000 0011 1010 0011 1011 0001 0010
• View as coefficients of a polynomial.• Note X = f(2m).• Same for Y = g(2m).• Multiply g×f using FFT in time O(nlogn).• Note X×Y = [g×f](2m).• Evaluate [g×f](2m) in time O(n) operations,
but each op could be on O(n) bit numbers for a total of O(n2) time.
f(x) = an-1xn-1 + … + a5x5 + a4x4 + a3x3 + a2x2 + a1x + a0
m
g(x) = bn-1xn-1 + … + b5x5 + b4x4 + b3x3 + b2x2 + b1x + b0
Multiplying Big Integers
X×Y = 0011 1010 0011 1011 0001 0010 1110 0011 0010
• Evaluate [g×f](2m) in time O(n).
[g×f](x) = cn-1xn-1 + … + c5x5 + c4x4 + c3x3 + c2x2 + c1x + c0
mO(log p)
• Some texts say the ci can just be shifted and joined.• Problem: The field elements may be too big.
Multiplying Big Integers• Evaluate [g×f](2m) in time O(n).
[g×f](x) = cnxn + … + c5x5 + c4x4 + c3x3 + c2x2 + c1x + c0
m
O(log p)
101011001101
110001000111
101011100100
010011
• Shift each ci by im.• Add
X×Y = 01 0101 0110 1011 1010 0001 1111 1011
Adding n numbers each n bits long takes O(n2)but here the numbers are sparse.
Multiplying Big Integers• Evaluate [g×f](2m) in time O(n).
[g×f](x) = cnxn + … + c5x5 + c4x4 + c3x3 + c2x2 + c1x + c0
m
O(log p)
101011001101
110001000111
101011100100
010011
• Shift each ci by im.• Add
X×Y = 01 0101 0110 1011 1010 0001 1111 1011
At each point, at most two numbers overlap Carry is at most one O(N) bit operations.
Multiplying Big IntegersX = 11…10100011101100010010 (N bits)Y = 10…01001100011001001111
X×Y = 10…1110110101001001010100010100110010011110
Suppose N is really really big.How many bit operations are needed?
• O(N logN)• O(N logN loglogN)• O(N logN loglogN logloglogN loglogloglogN …)
• FFT time• Time stated in text
• Time as far as I can see
Multiplying Big IntegersX = …101000111011000100101010001001010 …
• Input size = N bits• Field element size = N’ = log(N) bits• # ai = n = N/N’
• # of field ops = O(nlogn)• Time for × field op = ?
N’
X’ = 1010 0111 0110 0010 0101 0100 0100 1010
• Input size = N’ bits• Field element size = N’’ = log(N’) bits• # ai = n’ = N’/N’’
• # of field ops = O(n’logn’)• Time for × field op = ?• Total time:
N’’
• And so on …
O(N’ logN’ loglogN’ logloglogN’ …)
Multiplying Big IntegersX = …101000111011000100101010001001010 …• Input size = N bits• Field element size = N’ = log(N) bits• # ai = n = N/N’
• # of field ops = O(nlogn)• Time for × field op = ?• Total time:• = O( n logn ) × O(N’ logN’ loglogN’ logloglogN’ …) • = O(N/N’ logN/N’) × O(N’ loglogN logloglogN loglogloglogN …)• = O( N logN loglogN logloglogN loglogloglogN …)
N’
O(N’ logN’ loglogN’ logloglogN’ …)
The End