Remarks on History of FFT and Related Algorithms

Radomir S. Stankovic, Miomir S. Stankovic

Dept. of Computer Science, Faculty of Electronics, 18 000 Nis, Serbia

Why to Discuss History of FFT Treatise by Gauss in 1805

Gauss, C.F., ”Nachlass – Teoria interpolationis methodo nova tractata”, 265-303, in Carl Friedrich Gauss, Werke, Band 3, Gottingen, Konglichen Gesellschaft der Wissenschafteln, 1866.

Asteroid Juno

Discovered 1 September 1804 by Karl L. Harding

Referenced but Unnoticed

Burkhardt, H., ”Trigonometische interpolation”, in ch 9, Encyklopaedie der Mathematischen Wissenschaften, Vol. 2, Part 1, 1st half, 643-693, Leipzig, B.G. Teubner, 1899-1916

Burkhardt, H., 1904

Goldstine, H.H., 1977

Goldstine, H.H., A History of Numerical Analysis from the 16th Through the 19th Century, Springer, 1977, 249-253

In 2008, question about developments after Colley-Tukey FFT

Danielson, G.C. 1942 Lanczos, C. 1942

Prehistory of FFT

Gauss, C.F. 1805 Carlini, F. 1828 Smith, A. 1846 Everett, J.D. 1860 Runge, C. 1903 Stumpff, K. 1939

Thomas, L.H. 1948 Good, I.J. 1958

Any composite integer 12

4,8,16,32

12

2nk

2nk,3nk

2n

Any integer with relatively prime factors

Any integer with relatively prime factors

Cooley-Tukey FFT Cooley, J.W., Tukey, J.W., "An algorithm for the ma- chine calculation of complex Fourier series", Math. of Comput., Vol. 19, April 1965, 297- 301.

Reference to Runge and Konig, 1903, 1905, 1924

Cooley Tukey

Garwin

Motivations for FFT President Kennedy's Scientific Advisory Committee, 1963

Bill Dorn Manager of the Computing Center at the IBM T.J. Watson Research Center

John W. Tukey Professor of Mathematics, Princeton University Dick Garwin Columbia University IBM Watson Research Center, New York City

James W. Cooley

Three-dimensional Fourier transform of spin orientations of He3 Spectral analyses of data from seismic sensing devices

Reaction to FFT Runge, C., "Uber die Zerlegung empirisch periodischer Funktionen in Sinnus-wellen", Zeit. fÄur Math, und Physik, Vol. 48, 443-456, 1903. Runge, C., Zeit fur Math, und Physik, Vol. 53, 1-17, 1905. Runge, C., Konig, H., "Die Grundlehren der Mathematischen Wissenschaften", Vorlesungen Äuber Numerisches Rechnen, Vol. 11 Berlin, Julius Springer,1924.

Runge Konig

Runge-Konig Algorithm Cooley-Tukey FFT – a generalization to an arbitrary non-prime N

Due to symmetry of sinus and cosinus 2N transform Two N transforms with a bit > of operations

Repeated to sequences of the length 2m

m = log2 N fields with 2m+1 of the length 2m-1

N/2 fields with sequences of 2 elements

# of operations O(Nlog2N)

Algorithm by Rudnick

Rudnick, P., "Note on the calculation of Fourier series", Math, of Comput., Vol. 20, July 1966, 429-430.

Danielson, G.C., Lanczos, C., "Some improvements in practical Fourier analysis and their application to X- ray scattering from liquids", J. Franklin Inst., Vol. 233, April 1942, 365-380, and 435-452.

Refering to N = 2n # of operations N log2 N

Runge and Konig Scripps Institution of Oceanography, La Jolla, California

Stumpff, K., Tafeln und Aufgaben zur Harmonischer Analyse und Periodogrammrechnung, Berlin: Julius Springer, 1939.

Thomas, a generalization of work by Stumpff

Thomas, L.H., "Using a computer to solve problems in physics", in Application of Digital Computers, Boston, Mass., Ginn, 1963.

Algorithm by Thomas

1D transform replaced by multi-dimensional transforms

Thomas

Llewellyn Thomas, Columbia University

Algorithm by Good

A generalization of the algorithm by Thomas

nrrN ...1= ri –mutually prime

i

n

iRR

1=⊗= nCCR ...1=

ij

n

ji CC1=

⊗=

≠=

=.,,,

jiji

i

iij I

RC

Transform size

Kronecker

Good-Thomas and FFT

1. In the Good algorithm factors of N must be mutually prime numbers. 2. In this case, steps are purely multidimensional transform without twiddle factors. 3. The correspondence between single and multiple dimensional indexing are completely different.

Good

Mapping of Indices N = r1r2

Cooley-Tukey Indices of the array in lexicographic order

n = r2n1 + n0

1D 2D

k = r1k1 + k0

Good r1 and r2 mutually prime

k - Ruritanian correspondence

n - Sino representation n = r1n0 + r2n1 mod N n0 = k mod r1 (0≤ n0 < r1) n1 = k mod r2 (0≤ n1 < r2)

DFT = wNkn

k = r2pk0 + r1qk1 mod N (0 ≤ k < N) p, q – solutions for r2p = 1 mod r1 (p < r1) r1q = 1 mod r2 (q < r2)

10

1

01

2

0

1

1

2

2110 nkr

nkr

knr

knr

rknN

rknN

knN wwwwwww ===

1-D into 2-D by keeping lexicographic ordering among indices of the two-dimensional array

Cooley-Tukey and Good-Thomas N = 24, r1 = 8, r2 = 3

n = r2n1 + n0 = 3n1+ n0

k = r1k1 + k0 = 8k1+ k0

0 1 2

3 4 5

6 7 8

9 10 11

12 13 14

15 16 17

18 19 20

21 22 23

0 1 2

0 1 2 3 4 5 6 7

0 8 16

1 9 17

2 10 18

3 11 19

4 12 20

5 13 21

6 14 22

7 15 23

0 1 2

0 1 2 3 4 5 6 7

n1

n0

k0

k1

0 8 16

3 11 19

6 14 22

9 17 1

12 20 4

15 23 7

18 2 10

21 5 13

0 1 2

0 1 2 3 4 5 6 7

0 16 8

9 1 17

18 10 2

3 19 11

12 4 20

21 13 5

6 22 14

15 7 23

0 1 2

0 1 2 3 4 5 6 7

n1

n0

k0

k1

n = r1n0 + r2n1 = 8n0 + 3n1 mod 24

k = r2pk0 + r1qk1 = 9k0 + 16k1 mod 24 Ruritanian correspondence

Sino representation

Matrix Representation of Good - FFT )()2()1( ... rMMMM ⊗⊗⊗=

)( )(,

)( ννjim=M∏

=

=r

nknk mm1

)(,,

ν

ννν

1,...,1,0 −= νpi1,...,1,0 −= νqj

)()2()1()()2()1( ...... rr AAAMMMM =⊗⊗⊗=

)( )(,

)( ννnka=A

),(...),(),( 12312)(,

)(, 1 −= rrnknk nknknkma

rδδδνν

1121121 0,...,0,0,...,0,0 −+−+−+ <≤<≤<≤<≤<≤ ννννν qkqkpkpkpk rrrr

rrrrr qnqnpnpnpn <≤<≤<≤<≤<≤ +−−++ 0,...,0,0,...,0,0 112211 νννν

Good FFT N = 12 = 3×4 From Chinese remainder theorem

)0,2(8)0,1(4)0,0(0

→→→

)1,0(9)1,2(5

)1,1(1

→→→

)2,1(10)2,0(6)2,2(2

→→→

)3,2(11)3,1(7)3,0(3

→→→

DFT(12) = A(1)A(2)

=

wwww

wwww

wwww

wwww

2

2

2

2

2

2

2

2

1000000000000100000000000010000000000001

1000000000000100000000000010000000000001111000000000000111000000000000111000000000000111

)1(A

−−−−

−−−−

−−−−

−−−−

−−

=

iiii

ii

iiii

ii

110000000000001100000000000011111100000000000011110000000000001111

110000000000001100000000000011111100000000000011110000000000001111

)2(A

Good FFT Alterantive

If M(ν) square matrices with pν = qν, then )()2()1()()2()1( ...... rr CCCMMMM =⊗⊗⊗=

rrr IIMC ⊗⊗⊗= ...2

)1()1(

rrr IMIC ⊗⊗⊗= ...)2()2(1

)()( ...21

rrr

r MIIC ⊗⊗⊗=

.

.

.

⊗

⊗ ⊗ ( ( ) ) .

=

=

)1(M )2(M

)1(M )1(I )1(I )2(M

Good – Thomas FFT N = 12 = 3×4

=

wwww

wwww

wwww

wwww

0000001000000000010000000000100000000001

0000001000000000010000000000100000000001100010001000010001000100001000100010000100010001

2

2

2

2

2

2

2

2

)1(C

−−−−

−−

−−−−

−−

−−−−

−−

=

ii

ii

ii

ii

ii

ii

1100000000111100000000

11000000001111000000000000110000000011110000000011000000001111000000000000110000000011110000000011000000001111

)2(C

Matrix Interpretation of FFT F – DFT matrix F = PQR

R =

...

Q =

permutation

P =

1 1

1

1

block diagonal block matrix

This image cannot currently be displayed.

),(),( 121 cfrfcrcfrc ++→+

Theilheimer,1969 N = r1r2 Sande-Tukey

F = RTQPT

Cooly-Tukey Theilheimer

Theilheimer and Glassman

−

−

−

=

=

42

34

531

42

12345

2424

333

4242

54321

01000001

0000001101010010101

110000110000001100001100000011000011

100000001000000010010000000100000001

111

11111

1111111

wwww

wwwww

wwwwwwwwwwwwwwwwwwwww

R

=

=

3

3

3

45

24

3

42

21

12345

2424

333

4242

54321

001000001000001100100010010001001

10000001110000001

1000000111

111

11111

1111111

ww

w

wwww

www

ww

wwwwwwwwwwwwwwwwwwwww

R

Rader-Brenner Algorithm DFT by FFT requires multiplication of complex numbers z = x + iy Multiplication simplified if Re(z) = x = 0 or Im(z) = y = 0

Rader-Brenner algorithm = FFT with Mul(z) Mul(x) or Mul(y)

Rader

C. M. Rader and N. M. Brenner, "A new principle for fast Fourier transform," IEEE Trans. Acoust., Speech, Signal Processing, Vol. 24, 1976, 264-265

Brenner

Rader-Brenner Algorithm (Theory) { })(nx mN 2=

{ })(nb{ })(nc

)2()( nxnb =

Qnxnxnc +−−+= )12()12()(12/,...,1,0 −= Nn

∑−

=

+=1

0)12(2 N

nnx

NQ

)0()0()0( CBX +=

)()2(eccos21)()( kC

NkjkBkX π

−= 2/,0 Nk ≠

)0()0()2/( CBNX −=

R = DFT(r)

When cosec large, small error

huge error in the Out

1976

Rader and his Paper Rader, C.M., ”Discrete Fourier transforms when the number of data samples is prime”, Proc. IEEE, Vol.5, No.6, June 1968, 1107-1108.

If N = prime, then

2. DFT = cyclic convolution of the lenght N-1

1. Reordering of indices Multiplication of indices mod N

Addition of indices mod (N-1)

Rader

Winograd Algorithm

Winograd = Method by Rader with cyclic convolution implemented using convolution theorem performed with minimum number of multiplications

Min # of multiplication to calculate product of two polynomials

This image cannot currently be displayed.

)(mod)()()( xPxHxXxY = over a field F is 2N - k, where N = deg P(x)

Multiplication with constants in F is not counted

k - # of irreducible factors of P(x)

Winograd

Matix Form of Winograd Algorithm Coefficients of X(x), H(x) and Y(x) represented as vectors

[ ]TNxx )1(,...),0( −=x[ ]TNhh )1(,...),0( −=h[ ]TNyy )1(,...),0( −=y

Algorithm with min # of multiplications

)( AxBhCy ⋅=

x(0) x(1)

x(N-1)

h(N-1)

1

2

2N-k

h(0) h(1)

A

B

C

y(N-1)

y(0) y(1)

A, B = DFT C=DFT-1

Hadamard Calculation of cyclic convolution

Prime Factors Algorithms

1. 1D-DFT nD-DFT by Good algorithm

2. Transforms of small dimensions by Winograd algorithm

Good-Winograd algorithms

Winograd-Fourier algorithms (WFTA)

Different convolution of algorithms of small dimensions by Chinese remainder theorem and Kronecker product representable matrices

Algorithms for Walsh and Haar

⊗

⊗ ⊗ ( ( ) ) .

=

=

Good-Thomas Algorithms

=

wwww

wwww

wwww

wwww

2

2

2

2

2

2

2

2

1000000000000100000000000010000000000001

1000000000000100000000000010000000000001111000000000000111000000000000111000000000000111

)1(A

Different Matrix Representations

Decision Diagram Methods

FFT over DD

Exploits Properties

Transform R FFT

Decision diagrams

Decision trees

Function f

Decision diagrams

Functions and Decision Trees

x3

x1

x2

S

S S

S S S S

f

0

0 0

0 0 0 0

1

1 1

1 1 1 1

0 0 00 0 10 1 00 1 11 0 01 0 11 1 01 1 1

x x x1 2 3 ff(000) Sf(000)f(001) Sf(001)f(010) Sf(010)f(011) Sf(011)f(100) Sf(100)f(101)

Sf(101)f(110)Sf(110)f(111)Sf(111)

Decompositionrules

G C= 2 × × C C2 2

BDD n = 3

S S

S S S S

1 0 0 1 0 1 1 1

x2 x2

x3 x3 x3 x3x3 x3 x3 x3

x2 x2

_ _

_ _ _ _

f

x1 x1

_ S S

S S

S S

1 0

x1

x2x2

x3 x3

x3 x3

x2 x2

x1

_

_ _

_ _

)7()6()5()4()3()2()1()0(

321321321321

321321321321

fxxxfxxxfxxxfxxxfxxxfxxxfxxxfxxxf

⊕⊕⊕⊕⊕⊕⊕=

321321

21321

xxxxxxxxxxxf

⊕⊕⊕=

Reduction Rules

Bit-level

1. Delete nodes whose both outgoing edges point to the same node. The input edge of the deleted node connect to the corresponding successing node.

2. Share isomorphic subtrees.

BDD

S

S Sxixi

_

g gg

x ⊕ x = 1 _

x ∨ x = x

DD-Methods

⊗

⊗ ⊗ ( ( ) ) .

=

=

Calculations with sub-functions

At each node perform the basic transform matrix

Spectral Transforms over DD

Reed-Muller Workshop September 1993, Hamburg, Germany

Reed-Muller

Transform

Walsh

This image cannot currently be displayed.

−−−−

=)1()1()1()1(

)(nnnn

nRR0R

R

−−−−−

=)1()1()1()1(

)(nnnn

nWWWW

W

Walsh and Haar over DD

Haar Walhs

BDD into WDD

1+11 -1

20=

1 -1 1 -1

x3

x2 x2

x1x1

x2 x2

x3

x3x3_

_ _

_

_

S

S S

S S

f

-1 -1-1+1

-2 0=

1 -1-1-(-1)

0 2= -1+1

-1 -1 0 -2=

2020

-2 0-2 0

0040

+

-=

0202

0 -2 0 -2

0004

+

-=

00400040

00040004

0 0 4 4 0 0 4-4

+

-

= MTBDD

WDD

MTBDD WDD

Size

5 4

DD

WW

W

W

0 -44

1

1-2x2

1-2x3

1 (2-2 )x3⋅1 (2-2 )x3⋅

f

1

q x x = (1-2 )(2-2 )2 3

q

1 1-2x1

f

cordic cps duke2 misex2 vg2

23 24 22 25 25

2 109

29 18

8

1206 654

87 29 10

In Out Cubes

Functions that cannot be processed by FFT over vectors Calculation over decision diagrams can be applied

DD-Methods over FFT

# of ri

FFT and DD-Methods

FFT – constant complexity for any function of a given length

DD – complexity O(# of nodes in DD( f ))

Compact representation that permits calculation of spectral transforms of large fucntions.

Acknowledgments

Mathematical Institute of the Serbian Academy of Sciences and Arts