AMultigrid Tutorial
ByWilliam L. Briggs
Presented byVan Emden Henson
Center for Applied Scientific ComputingLawrence Livermore National Laboratory
This work was performed, in part, under the auspices of the United States Department of Energy by Universityof California Lawrence Livermore National Laboratory under contract number W-7405-Eng-48.
2 of 119
Outline• Model Problems• Basic Iterative Schemes
– Convergence; experiments– Convergence; analysis
• Development of Multigrid– Coarse-grid correction– Nested Iteration– Restriction and Interpolation– Standard cycles: MV, FMG
• Performance– Implementation– storage and computation costs
• Performance, (cont)– Convergence– Higher dimensions– Experiments
• Some Theory– The spectral picture– The subspace picture– The whole picture!
• Complications– Anisotropic operators and grids– Discontinuous or anisotropic
coefficients– Nonlinear Problems: FAS
3 of 119
Suggested Reading
• Brandt, “Multi-level Adaptive Solutions to Boundary ValueProblems,” Math Comp., 31, 1977, pp 333-390.
• Brandt, “1984 Guide to Multigrid Development, withapplications to computational fluid dynamics.”
• Briggs, “A Multigrid Tutorial,” SIAM publications, 1987.• Briggs, Henson, and McCormick, “A Multigrid Tutorial, 2nd
Edition,” SIAM publications, 2000.• Hackbusch, Multi-Grid Methods and Applications,” 1985.• Hackbusch and Trottenburg, “Multigrid Methods, Springer-
Verlag, 1982”• Stüben and Trottenburg, “Multigrid Methods,” 1987.• Wesseling, “An Introduction to Multigrid Methods,” Wylie,
1992
4 of 119
Multilevel methods have beendeveloped for...
• Elliptic PDEs• Purely algebraic problems, with no physical grid; for
example, network and geodetic survey problems.• Image reconstruction and tomography• Optimization (e.g., the travelling salesman and long
transportation problems)• Statistical mechanics, Ising spin models.• Quantum chromodynamics.• Quadrature and generalized FFTs.• Integral equations.
5 of 119
Model Problems
• One-dimensional boundary value problem:
• Grid:
• Let and for
>σ,<<)(=)(σ+)(″− xxfxuxu 010uu =)(=)( 010
Nh =
1
xuv ii )(≈ xff ii )(≈
...,,=,=, Nihixi 10
Ni ...,,= 10
x0 x1 x2 xi xN
x = 0 x = 1
6 of 119
We use Taylor Series to derivean approximation to u’’(x)
• We approximate the second derivative usingTaylor series:
• Summing and solving,
hOxuh
xuh
xuhxuxu )(+)(′′′+)(″+)(′+)(=)( iiiii +4
32
1 !3!2
hOxuh
xuh
xuhxuxu )(+)(′′′−)(″+)(′−)(=)( iiiii −4
32
1 !3!2
hOh
xuxuxuxu )(+
)(+)(−)(=)(″
iiii
−+ 22
11 2
7 of 119
We approximate the equationwith a finite difference scheme
• We approximate the BVP
with the finite difference scheme:
>σ,<<)(=)(σ+)(″− xxfxuxu 010uu =)(=)( 010
−...,,==σ+−+−
Nifvh
vvvii
iii +−2
11121
2
vv0 == N 0
8 of 119
The discrete model problem
• Letting and
we obtain the matrix equation whereis (N-1) x (N-1), symmetric, positive definite, and
A fv =
v ),...,,(= vvv 121T
N −
f ),...,,(= fff 121T
N −
A
ff
fff
f
vv
vvv
h
h
h
h
h
hA
=,
=,
σ+−
−σ+−
−σ+−
−σ+−
−σ+
=
21
121
121
121
12
1
1
2
3
2
1
12
321
2
2
2
2
2
2v
N
NNN
−
−−−
9 of 119
Solution Methods• Direct
– Gaussian elimination– Factorization
• Iterative– Jacobi– Gauss-Seidel– Conjugate Gradient, etc.
• Note: This simple model problem can be solvedvery efficiently in several ways. Pretend it can’t,and that it is very hard, because it shares manycharacteristics with some very hard problems.
10 of 119
A two-dimensional boundary valueproblem
• Consider the problem:
• Where the grid is given:
yyxxu >σ;=,=,=,=,= 010100
Nh
Mh yx ,=,=
11
),(=),( hjhiyx yxji
0 ≤≤ Mi0 ≤≤ Nj
x
y
z
<<,<<,),(=σ+−− yxyxfuuu yyxx 1010
11 of 119
Discretizing the 2D problem
• Let and . Again, using 2nd
order finite differences to approximate andwe obtain the approximate equation for theunknown , for i=1,2, … M-1 and j=1,2, …, N-1:
• Ordering the unknowns (and also the vector f )lexicographically by y-lines:
yxuv jiji ),(≈ yxff jiji ),(≈u xx u yy
yxu ),( ji
=σ+−+−
+−+−
fvh
vvv
h
vvvjiji
y
jijiji
x
jijiji +,−,,+,−
2
11
2
11 22
MjjMiiv ji =,=,=,=,= 000
v ),...,,,...,...,,,,...,,(= vvvvvvvvv 112111122212112111 −,−,−,−−,,,−,,,T
NNNNNN
12 of 119
Yields the linear system• We obtain a block-tridiagonal system Av = f :
where Iy is a diagonal matrix with on thediagonal and
=
−
−−
−−−−
−
f
fff
v
vvv
AIIAI
IAIIAI
IA
1
3
2
1
1
321
1
2
3
2
1
NNNy
yNy
yy
yy
y
−−−
−
1
hy2
hhh
hhhh
hhhh
hhh
A
yxx
xyxx
xyxx
xyx
i
σ++−
⋅⋅⋅⋅⋅⋅⋅⋅⋅
−σ++−
−σ++−
−σ++
=
111
1111
1111
111
222
2222
2222
222
13 of 119
Iterative Methods for LinearSystems
• Consider Au = f where A is NxN and let v be anapproximation to u.
• Two important measures:– The Error: with norms
– The Residual: with
vue ,−=
||=|||| ee ∞ xam i =|||| ee 212 i
Ni =
vAfr −=|||| r ∞ |||| r 2
14 of 119
Residual correction
• Since and , we can write as
which means that , which is theResidual Equation:
• Residual Correction:
vue ,−= vAfr −=fuA =
fevA =)+(
vAfeA −=
reA =
evu +=
15 of 119
Relaxation Schemes
• Consider the 1D model problem
• Jacobi Method (simultaneous displacement): Solvethe ith equation for holding other variablesfixed:
==−≤≤=−+− uuNifhuuu Niiii +− 02
11 0112
vi
Nifhvvv idlo
idlo
iwen
i)(
+)(
−)( −≤≤)++(= 11
21 2
11
16 of 119
In matrix form, the relaxation is
• Let where D is diagonal and L and Uare the strictly lower and upper parts of A.
• Then becomes
• Let , then the iteration is:
fuA =
fuULuD +)+(=fDuULDu +)+(= −− 11
ULDRJ )+(= −1
ULDA )−−(=
=)−−( fuULD
fDvRv −)()( dloJ
wen += 1
17 of 119
The iteration matrix and theerror
• From the derivation,
• the iteration is
• subtracting,
• or
• hence
fDuULDu +)+(= −− 11
fDuRu += J−1
fDvRv −)()( dloJ
wen += 1
e )()( dloJ
wen = eR
vRuRvu −=− )()( dloJJ
wen
fDvRfDuRvu )+(−+=− −)(−)( dloJJ
wen 11
18 of 119
Weighted Jacobi Relaxation• Consider the iteration:
• Letting A = D-L-U, the matrix form is:
.• Note that
• It is easy to see that if ,then
fhvvvv idlo
idlo
idlo
iwen
i)(
+)(
−)()( )++(
ω+)ω−(←
21 2
11
fDhvULDIv −)(−)( dlowen ω+)+(ω+)ω−(= 1 121
ω+= fDhvR −)(ω dlo 12
RIRω ]ω+)ω−([= 1 J
uue −≡ )()( xorppatcaxe
eRe )(ω)( dlowen =
19 of 119
Gauss-Seidel Relaxation (1D)• Solve equation i for ui and update immediately.• Equivalently: set each component of r to zero.• Component form: for set
• Matrix form:
• Let• Then iterate
• Error propagation:
Ni ,−...,,= 121
fhvvv iiii )++(←21
+−2
11
ULDA )−−(=+=)−( fuUuLD
fLDuULDu )−(+)−(= −− 11
ULDRG )−(= −1
fLDvRv −)()( dloG
wen )−(+← 1
eRe )()( dloG
wen ←
20 of 119
Red-Black Gauss-Seidel• Update the even (red) points
• Update the odd (black) points
fhvvv 22
12122 iiii )++(←21
+−
fhvvv 122
22212 iiii +++ )++(←21
x0 xN
21 of 1190 0 . 1 0 . 2 0 . 3 0 . 4 0 . 5 0 . 6 0 . 7 0 . 8 0 . 9 1
- 1
- 0 . 8
- 0 . 6
- 0 . 4
- 0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
Numerical Experiments• Solve ,• Use Fourier modes as initial iterate, with N =64:
uA = 0 =−+− uuu iii +− 11 02
vk
π
=)(=Nki
v ki nis 1111 −≤≤,−≤≤ NkNicomponent mode
k = 3
k = 1
k = 6
22 of 119
0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 00
0 . 1
0 . 2
0 . 3
0 . 4
0 . 5
0 . 6
0 . 7
0 . 8
0 . 9
1
Error reduction stalls• Weighted Jacobi on 1D problem.• Initial guess:
• Error plotted against iteration number:
Nj
Nj
Nj
v0
π
+
π
+
π
=23
nis6
nisnis31
=ω32
|||| e ∞
23 of 1190 20 40 6 0 8 0 1 00 1 20
0
0 .1
0 .2
0 .3
0 .4
0 .5
0 .6
0 .7
0 .8
0 .9
1
Convergence rates differ fordifferent error components
• Error, ||e||∞∞∞∞ , in weighted Jacobi on Au = 0 for100 iterations using initial guesses of v1, v3, and v6
k = 1
k = 3
k = 6
24 of 119
Analysis of stationary iterations
• Let . The exact solution isunchanged by the iteration, i.e., .
• Subtracting, we see that
• Letting e0 be the initial error and ei be the errorafter the ith iteration, we see that after niterations we have
gvRv )()( dlowen +=
guRu +=
eRe )()( dlowen =
eRe )()( nn = 0
25 of 119
A quick review of eigenvectorsand eigenvalues
• The number is an eigenvalue of a matrix B, and wits associated eigenvector, if Bw = w.
• The eigenvalues and eigenvectors are characteristicsof a given matrix.
• Eigenvectors are linearly independent, and if there isa complete set of N distinct eigenvectors for anNxN matrix, they form a basis; i.e., for any v, thereexist unique ck such that:
λλ
wcv = kkN
k = 1
26 of 119
“Fundamental theorem ofiteration”
• R is convergent (that is, as ) if andonly if , where
therefore, for any initial vector v(0), we see that if and only if .
• assures the convergence of the iterationgiven by R and is called the convergencefactor for the iteration.
R n→0 n ∞→<)(ρ R 1
|λ|,...,|λ|,|λ|=)(ρ R xam N21
ne )( n ∞→→ sa0 <)(ρ R 1
<)(ρ R 1)(ρ R
27 of 119
Convergence Rate• How many iterations are needed to reduce the
initial error by ?
• So, we have .
• The convergence rate is given:
01 −d
∼))(ρ(∼||||≤||||||||
RRee −
)(
)( dMMM
001
dM
=gol
101 )(ρ R
noitaretistigid
gol1
goletar ))(ρ(−=)(ρ
= 0101 RR
28 of 119
Convergence analysis forweighted Jacobi on 1D model
ULDIR −ω )+(ω+)ω−(= 1 1
ω−= ADI −1
IRω
−⋅⋅⋅⋅⋅⋅⋅⋅⋅
−−−−
−
ω−=
21
121121
12
2
)(λω
−=)(λ ARω 21
For the 1D model problem, he eigenvectors of the weightedJacobi iteration and the eigenvectors of the matrix A arethe same! The eigenvalues are related as well.
29 of 119
Good exercise: Find theeigenvalues & eigenvectors of A
• Show that the eigenvectors of A are Fouriermodes!
π
=,
π
=)(λ jkk Nkj
wN
kA nis
2nis4 2
,
0 1 0 2 0 3 0 4 0 5 0 6 0
-1
- 0 . 8
- 0 . 6
- 0 . 4
- 0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
0 1 0 2 0 3 0 4 0 5 0 6 0
- 1
- 0 . 8
- 0 . 6
- 0 . 4
- 0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
0 1 0 2 0 3 0 4 0 5 0 6 0
-1
- 0 . 8
- 0 . 6
- 0 . 4
- 0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
0 1 0 2 0 3 0 4 0 5 0 6 0
-1
-0 . 8
-0 . 6
-0 . 4
-0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
0 1 0 2 0 3 0 4 0 5 0 6 0
-1
-0 . 8
-0 . 6
-0 . 4
-0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
30 of 119
Eigenvectors of Rωωωω and A are thesame,the eigenvalues related
• Expand the initial error in terms of theeigenvectors:
• After M iterations,
• The kth mode of the error is reduced by λk at eachiteration
π
ω−=)(λk Nk
Rω 2nis21 2
wce−
=
)(1
1
0 = kkN
k
wcwRceR kMkk
N
kk
Mk
N
k
M−
=
−
=
)(1
1
1
1
0 λ==
31 of 119
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
k
Eig
enva
lue
/=ω 31
/=ω 21
/=ω 32=ω 1
N2
Relaxation suppresseseigenmodes unevenly
• Look carefully at
π
ω−=)(λk Nk
Rω 2nis21 2
Note that ifthen for
For ,
10 ≤ω≤<|)(λ| k Rω 1
Nk −,...,,= 121
10 ≤ω≤
π
ω−=λ 21 2
nis21N
π
ω−=2
nis21 2 h
≈)(−= 11 hO 2
32 of 119
Low frequencies are undamped
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
k
Eig
enva
lue
/=ω 31
/=ω 21
/=ω 32=ω 1
N2
• Notice that no value of will damp out the long(i.e., low frequency) waves.
ω
What value of givesthe best damping ofthe short waves ?
Choose such that
NkN2
≤≤
ω
)(λ−=)(λ NN2
RR ωω
ω
=ω
32
33 of 119
The Smoothing factor
• The smoothing factor is the largest absolute valueamong the eigenvalues in the upper half of thespectrum of the iteration matrix
• For Rω, with , the smoothing factor is ,since
and for .
• But, for long waves .
2rofxamrotcafgnihtooms ≤≤)(λ= k Nk
NR
=ω32
31
=λ=λ NN2
31
<λk 31
NkN2
<<
π−≈λk 32
1 hk 222
Nk
2«
34 of 119
Convergence of Jacobi on Au=0
• Jacobi method on Au=0 with N=64. Number ofiterations required to reduce to ||e||∞ < .01
• Initial guess :
0 10 20 30 40 50 600
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 600
10
20
30
40
50
60
70
80
90
100
=N
jkvkjπsin
Wavenumber, kWavenumber, k
Unweighted Jacobi Weighted Jacobi
35 of 119
Weighted Jacobi RelaxationSmooths the Error
• Initial error:
• Error after 35 iteration sweeps:0 0 . 5 1
- 2
- 1
0
1
2
0 0 . 5 1
- 2
- 1
0
1
2
Many relaxationschemes
have the smoothingproperty, where
oscillatorymodes of the error
areeliminated
effectively, butsmooth modes are
dampedvery slowly.
Nj
Nj
Nj
v jk
π
+
π
+
π
=23
nis2161
nis212
nis
36 of 1190 10 20 30 40 50 60
0
0.2
0.4
0.6
0.8
1
Other relaxations schemes maybe analyzed similarly
• Gauss-Seidel relaxation applied to the 3-pointdifference matrix A (1D model problem):
• Good exercise: Show thatULDRG )−(= −1
0 10 20 30 40 50 60
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
π
=)(λ Gk Nk
R soc 2Nkj
wj
kjk/
,
π
)λ(=2
nis
37 of 119
Convergence of Gauss-Seidel onAu=0
• Eigenvectors of RG are not the same as those of A.Gauss-Seidel mixes the modes of A.
0 10 20 30 40 50 600
10
20
30
40
50
60
70
80
90
100 Jacobi method on Au=0with N=64. Number ofiterations required toreduce to ||e||∞ < .01
Initial guess : (Modes ofA)
=N
jkvkjπsin
38 of 119
• Many relaxation schemes have the smoothingproperty, where oscillatory modes of the errorare eliminated effectively, but smooth modesare damped very slowly.
• This might seem like a limitation, but by usingcoarse grids we can use the smoothing property togood advantage.
• Why use coarse grids??
First observation towardmultigrid
x0 xN
x0 xN2
Ωh
Ω2h
39 of 119
Reason #1 for using coarsegrids: Nested Iteration
• Coarse grids can be used to compute an improvedinitial guess for the fine-grid relaxation. This isadvantageous because:
– Relaxation on the coarse-grid is much cheaper (1/2 asmany points in 1D, 1/4 in 2D, 1/8 in 3D)
– Relaxation on the coarse grid has a marginally betterconvergence rate, for example
instead of 1 )(− hO 241 )(− hO 2
40 of 119
Idea! Nested Iteration
• …• Relax on Au=f on to obtain initial guess• Relax on Au=f on to obtain initial guess• Relax on Au=f on to obtain … final solution???
• But, what is Au=f on , , … ?
• What if the error still has smooth componentswhen we get to the fine grid ?
Ω4h
Ω2h
Ωh
v2h
vh
Ω2h Ω4h
Ωh
41 of 119
Reason #2 for using a coarsegrid: smooth error is (relatively)
more oscillatory there!• A smooth function:
• Can be represented by linearinterpolation from a coarser grid:
0 0 . 5 1- 1
- 0 . 5
0
0 . 5
1
0 0 . 5 1- 1
- 0 . 5
0
0 . 5
1
On the coarse grid, the smooth error appears tobe relatively higher in
frequency: in the exampleit is the 4-mode, out ofa possible 16, on the finegrid, 1/4 the way up thespectrum. On the coarse grid, it is the 4-mode outof a possible 8, hence itis 1/2 the way up the
spectrum.
Relaxation will be moreeffective on this mode ifdone on the coarser grid!!
42 of 119
For k=1,2,…N/2, the kth mode ispreserved on the coarse grid
0 2 4 6 8 1 0 1 2
- 1
- 0 . 8
- 0 . 6
- 0 . 4
- 0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
0 1 2 3 4 5 6
- 1
- 0 . 8
- 0 . 6
- 0 . 4
- 0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
wNkj
Nkj
w hjk
hjk ,,
22 =
/
π
=
π
=2
nis2
nis
Also, note that
on the coarse grid
whN/2 → 0
k=4 mode, N=12 grid
k=4 mode, N=6 grid
43 of 119
For k > N/2, wkh is invisible on
the coarse grid: aliasing!!• For k > N/2, the kth mode on
the fine grid is aliased andappears as the (N-k)th modeon the coarse grid:
0 2 4 6 8 1 0 1 2
- 1
- 0 . 8
- 0 . 6
- 0 . 4
- 0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
0 1 2 3 4 5 6
- 1
- 0 . 8
- 0 . 6
- 0 . 4
- 0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
)−(π
−=2
nisN
kNj
π)(
=)(N
kjw
jhk 2
2nis
/
)−(π
−=2
nisN
jkN
)(−= w hkN −
2
k=9 mode, N=12 grid
k=3 mode, N=12 grid
44 of 119
Second observation towardmultigrid:
• Recall the residual correction idea: Let v be anapproximation to the solution of Au=f, where theresidual r=f-Av. The the error e=u-v satisfiesAe=r.
• After relaxing on Au=f on the fine grid, the errorwill be smooth. On the coarse grid, however, thiserror appears more oscillatory, and relaxation willbe more effective.
• Therefore we go to a coarse grid and relax on theresidual equation Ae=r, with an initial guess of e=0.
45 of 119
Idea! Coarse-grid correction
• Relax on Au=f on to obtain an approximation• Compute .• Relax on Ae=r on to obtain an approximation
to the error, .• Correct the approximation .
• Clearly, we need methods for the mappings and
Ω2h
Ωh vh
Ω2hΩh
vAfr −= h
e2h
evv hhh +← 2
ΩhΩ2h
46 of 119
1D Interpolation (Prolongation)
• Mapping from the coarse grid to the fine grid:
• Let , be defined on , . Then
where
vh v2h Ωh Ω2h
vv 22
hi
hi =
vvv 21
212
hi
hi
hi ++ )+(=
21
for 12
0 −≤≤N
i
I 22
hhhh Ω→Ω:
vvI 22
hhhh =
47 of 119
1D Interpolation (Prolongation)
Ωh
Ω2h
• Values at points on the coarse grid map unchangedto the fine grid
• Values at fine-grid points NOT on the coarse gridare the averages of their coarse-grid neighbors
48 of 119
The prolongation operator (1D)
• We may regard as a linear operator fromℜ N/2-1 ℜ N-1
• e.g., for N=8,
• has full rank, and thus null space
=
/
//
//
/ 21
12121
12121
121
177
6
5
4
3
2
1
1323
22
21
37
xh
h
h
h
h
h
h
xh
h
h
x v
v
v
v
v
v
v
v
v
v
φ
I2hh
I2hh
49 of 119
How well does v2h approximate u?
• Imagine that a coarse-grid approximation v2h hasbeen found. How well does it approximate theexact solution u ? Think u error!!
• If u is smooth, a coarse-grid interpolant of v2h
may do very well.
50 of 119
How well does v2h approximate u?
• Imagine that a coarse-grid approximation v2h hasbeen found. How well does it approximate theexact solution u ? Think u error!!
• If u is oscillatory, a coarse-grid interpolant of v2h
may not work well.
51 of 119
Moral of this story:
• If u is smooth, a coarse-grid interpolant of v2h
may do very well.
• If u is oscillatory, a coarse-grid interpolant of v2h
may not work well.
• Therefore, nested iteration is most effectivewhen the error is smooth!
52 of 119
1D Restriction by injection• Mapping from the fine grid to the coarse grid:
• Let , be defined on , . Then
where .
vh v2h Ωh Ω2h
vv hi
hi 22 =
I hhhh
22 Ω→Ω:
vvI hhhh
22 =
53 of 119
1D Restriction by full-weighting
• Let , be defined on , . Then
where
vh v2h Ωh Ω2h
vvvv hi
hi
hi
hi 122122 )++(= 2
41
+−
vvI hhhh
22 =
54 of 119
The restriction operator R (1D)
• We may regard as a linear operator fromℜ N-1 ℜ N/2-1
• e.g., for N=8,
• has rank , and thus dim(NS(R))
=
///
//////
412141
412141412141
v
v
v
v
v
v
v
v
v
v
23
22
21
7
6
5
4
3
2
1
h
h
h
h
h
h
h
h
h
h
∼N2
∼N2
I hh2
I hh2
55 of 119
Prolongation and restriction areoften nicely related
• For the 1D examples, linear interpolation and full-weighting are related by:
• A commonly used, and highly useful, requirement isthat
for c in ℜIcI 22
Thh
hh )(=
I2hh
=
1211
211
21
21 I h
h2
=121
121121
41
56 of 119
2D Prolongation
vv 222
hji
hji, =
vvv 12
212h
jihji
hji ,+,+ )+(=
21
vvv 12
122h
jihji
hji +,+, )+(=
21
vvvvv 11112
1212h
jih
jih
jihji
hji +,++,,++,+ )+++(=
41 4
121
41
21
121
41
21
41
We denote the operator byusing a “give to” stencil, ] [.Centered over a c-point, ,it shows what fraction of
the c-point’s value iscontributed to neighboring
f-points, .
57 of 119
2D Restriction (full-weighting)
We denote the operator byusing a “give to” stencil, [ ].Centered over a c-point, ,it shows what fractions of
the neighboring ( ) f-points’value is contributed to the
value at the c-point.
611
81
611
81
41
81
611
81
611
58 of 119
Now, let’s put all these ideastogether
• Nested Iteration (effective on smooth errormodes)
• Relaxation (effective on oscillatory error modes)
• Residual equation (i.e., residual correction)
• Prolongation and Restriction
59 of 119
Coarse Grid Correction Scheme
• 1) Relax times on on with
arbitrary initial guess .
• 2) Compute .
• 3) Compute .
• 4) Solve on .
• 5) Correct fine-grid solution .
• 6) Relax times on on with initial
guess .
fuA hhh = Ωh
vh
fuA hhh = Ωh
vh
fvGCv hhh )α,α,,(← 21
α1
α2
vAfr hhhh −=
reA 222 hhh = Ω2h
rIr 22 hhh
h =
eIvv hhh
hh +← 22
60 of 119
Coarse-grid Correction
Relax on fuA hhh =uAfr hhhh −=Compute
rIr 22 hhh
h =Restrict
Solve reA 222 hhh =rAe 2122 hhh )(= −
Correct
eIe hhh
h ≈ 22
Interpolate
euu hhh +←
61 of 119
What is ?
• For this scheme to work, we must have , acoarse-grid operator. For the moment, we willsimply assume that is “the coarse-gridversion” of the fine-grid operator .
• We will return to the question of constructinglater.
A 2h
A 2h
A 2h
A 2h
A h
62 of 119
How do we “solve” the coarse-grid residual equation? Recursion!
uIe hhh
h← 22
uIe 424
2 hhh
h←
uIe 848
4 hhh
h←
fAGu hhh ),(← ν
fAGu 222 hhh ),(← ν
fAGu 444 hhh ),(← ν
fAGu 888 hhh ),(← ν
uAfIf 22 hhhhh
h )−(←
uAfIf 22242
4 hhhhh
h )−(←
uAfIf 44484
8 hhhhh
h )−(←
euu 888 hhh +←
euu 444 hhh +←
euu 222 hhh +←
euu hhh +←
fAe HHH )(= −1
63 of 119
V-cycle (recursive form)
1) Relax times on , initial arbitrary
2) If is the coarsest grid, go to 4) Else:
3) Correct
4) Relax times on , initial guess
fvVMv hhhh ),(←
α1 fuA hhh = vh
Ωh
vAfIf 22 hhhh
hh )−(←
v2h ← 0
fvVMv 2222 hhhh ),(←
vIvv hhh
hh +← 22
α2 fuA hhh = vh
64 of 119
Storage Costs: and mustbe stored on each level
In 1-d, each coarse grid has about half the number of points as the finer grid.
In 2-d, each coarse grid has about one- fourth the number of points as the finer grid.
In d-dimensions, each coarse grid has about the number of points as the finer grid.
2− d
21
2222212
NN
d
ddMdddd
−<)+…++++(
−−−−− 32
Total storage cost: less than 2, 4/3, 8/7 the cost of storage on the fine grid for 1, 2, and 3-d problems, respectively.
vh f h
65 of 119
Computation Costs• Let 1 Work Unit (WU) be the cost of one
relaxation sweep on the fine-grid.• Ignore the cost of restriction and interpolation
(typically about 20% of the total cost).• Consider a V-cycle with 1 pre-Coarse-Grid
correction relaxation sweep and 1 post-Coarse-Grid correction relaxation sweep.
• Cost of V-cycle (in WU):
• Cost is about 4, 8/3, 16/7 WU per V-cycle in 1, 2,and 3 dimensions.
21
2222212
−<)+…++++(
−−−−−
ddMddd 32
66 of 119
Convergence Analysis
• First, a heuristic argument:– The convergence factor for the oscillatory modes of the
error (e.g., the smoothing factor) is small andindependent of the grid spacing h.
– The multigrid cycling schemes focus the relaxation onthe oscillatory components on each level.
2rofxamrotcafgnihtooms ≤≤)(λ= k Nk
NR
smoothk=1
oscillatoryk=Nk=N/2
Relax on fine gridRelax 1st coarse grid
The overall convergence factor for multigridmethods is small and independent of h!
67 of 119
Convergence analysis, a littlemore precisely...
• Continuous problem:• Discrete problem:
• Global error: (p=2 for model problem)
• Algebraic error:
• Suppose a tolerance is specified such thatmust satisfy
• Then this is guaranteed if
xuufuA )(=,= iiuvfuA hhhhh ≈,=
uxuE hiii −)(=
hKE ≤p
vue hi
hii −=
ε vh
vu ε≤− h
eEvuuu ε≤+=−+− hhh
68 of 119
We can satisfy the requirementby imposing two conditions
1) . We use this requirement to determine agrid spacing from
2) , which determines the number ofiterations required.
• If we iterate until on thenwe have converged to the level of truncation.
h ∗E
ε≤
2
Kh
/∗
ε
≤2
1 p
eε
≤2
hKe )(=ε
≤2
∗ pΩh ∗
69 of 119
Converging to the level oftruncation
• Use a MV scheme with convergence rateindependent of h (fixed and ).
• Assume a d-dimensional problem on an NxNx…xNgrid with .
• The V-cycle must reduce the error fromto
• We can determine θ, the number of V-cyclesrequired to accomplish this.
<γ 1α1 α2
Nh = −1
Oe )(∼ 1
NOhOe )(∼)(∼pp −
70 of 119
Work needed to converge to thelevel of truncation
• Since θ V-cycles at convergence rate γ arerequired, we see that
implying that .
• Since one V-cycle costs O(1) WU and one WU isO(Nd), we see that the cost of converging to thelevel of truncation using the MV method is
• which is comparable to fast direct methods (FFTbased).
)(∼γ−θ NO p
)(∼θ NO gol
NNO )( d gol
71 of 119
A numerical example
• Consider the two-dimensional model problem (withσ=0), given by
inside the unit square, with u=0 on the boundary.)−()−(+)−()−(=−− xxyyyxuu yyxx 1611612 222222
• The solution to this problem is
• We examine the effectiveness of MV cycling tosolve this problem on NxN grids [(N-1) x (N-1)interior] for N=16, 32, 64, 128.
yyxxyxu )−()−(=),( 2442
72 of 119
Numerical Results, MV cycling
Shown are the resultsof 16 V-cycles. Wedisplay, at the end ofeach cycle, the residualnorm, the error norm,and the ratio of thesenorms to their values atthe end of the previouscycle.
N=16, 32, 64, 128
73 of 119
Look again at nested iteration• Idea: It’s cheaper to solve a problem (i.e., takes
fewer iterations) if the initial guess is good.
• How to get a good initial guess:– Interpolate coarse solution to fine grid.– “Solve” the problem on the coarse grid first.– Use interpolated coarse solution as initial guess on fine
grid.
Now, let’s use the V-cycle as the solver on each grid level! This defines the Full Multigrid (FMG) cycle.
74 of 119
The Full Multigrid (FMG) cycle
• Initialize
• Solve on coarsest grid
• interpolate initial guess
• perform V-cycle
• interpolate initial guess
• perform V-cycle
ffff Hhhh ,...,,, 42
fAv HHH )(=−1
vIv 424
2 hhh
h←
vIv hhh
h← 22
fvVMv 2222 hhhh ),(←
fvVMv hhhh ),(←
fGMFv hh )(←
75 of 119
Full Multigrid (FMG)• Restriction• Interpolation• High-order Interpolation
76 of 119
FMG-cycle (recursive form)
1) Initialize
2) If is the coarsest grid, then solve exactly
Else:
3) Set initial guess
4) , η times
Ωh
ffff Hhhh ,...,,, 42
fGMFv hh )(←
vIv hhh
h← 22
fGMFv 22 hh )(←
fvVMv hhh ),(←
77 of 119
Cost of an FMG cycle• One V-cycle is performed per level, at a cost of WU per grid (where the WU is for the
size grid involved).• The size of the WU for coarse-grid j is times
the size of the WU on the fine grid (grid 0).• Hence, cost of the FMG(1,1) cycle is less than
• d=1: 8 WU; d=2: 7/2 WU d=3: 5/2 WU
21
1
− −d
2− dj
)−(=)...+++(
−
21
2221
21
2−
−−− d
ddd 2
2
78 of 119
How to tell if truncation error isreached with Full Multigrid (FMG)
• If truncation error is reached, for each gridlevel h. The norms of the errors at the “solution”points in the cycle should form a Cauchy sequence and
hOe )(∼ 2
e
e2
hj
hj
∼41
79 of 119
Cost to achieve convergence totruncation by the FMV method
• Consider using the FMV method, which solves theproblem on to the level of truncation beforegoing to , i.e.,
• We ask that which impliesthat the algebraic error must be reduced byon . Hence, V-cycles are needed, where
thus and computational cost of theFMV method .
Ω2h
Ωh
hKvue 222 phhh )(∼−= 2
ehKe hpph =∼ 2− 2
2−p
Ωh θ1
∼γ−θ1 2
p
)(∼θ1 O 1NO )( d
80 of 119
A numerical example
• Consider again the two-dimensional model problem(with σ=0), given by
inside the unit square, with u=0 on the boundary.)−()−(+)−()−(=−− xxyyyxuu yyxx 1611612 222222
• We examine the effectiveness of FMG cycling tosolve this problem on NxN grids [(N-1) x (N-1)interior] for N=16, 32, 64, 128.
81 of 119
FMG results• Shown are results for three FMG cycles, and a
comparison to MV cycle results.
82 of 119
What is ?• Recall the coarse-grid correction scheme:
– 1) Relax on on to get .
– 2) Compute .
– 4) Solve on .
– 5) Correct fine-grid solution .
• Assume that . Then theresidual equation can be written
for some
• How does act upon ?
fuA hhh = Ωh
reA 222 hhh = Ω2h
eIvv hhh
hh +← 22
A 2h
vAfIf 22 hhhhh
h )−(=
IegnaRe hh
h )(∈ 2
eAr hhh = = uIA hhh
h 22 u 22 hh Ω∈
A h IegnaR )( 2hh
vh
83 of 119
How does act on ?
• Therefore, the odd rows of are zero (in 1Donly) and . We therefore keep the even rowsof for the residual equations on . Theserows can be picked out by applying restriction!
A h IegnaR )( 2hh
u2h
uI 22
hhh
uIA hhh
h 22
r 12i + = 0IA h
hh
2
rIuIAI hhh
hhh
hhh
222
2 =
IA hh
h2
Ω2h
84 of 119
Building .
• The residual equation on the coarse grid is:
• Therefore, we identify the coarse-grid operatoras
• Next, we determine how to compute the entries ofthe coarse-grid matrix.
A 2h
rIuIAI hhh
hhh
hhh
222
2 =
A 2h
IAIA 222 h
hhh
hh =
85 of 119
Computing the ith row of .
• Compute where
A 2h
eA 22 hi
h e Thi2 ),...,,,,...,,(= 001000
ii −1 i +1
0 01 e hi2
0 01 21
21 eI 2
2h
ihh
0 0 −2
1
h2−
2
1
h21
h2 eIA hi
hh
h 22
2
1
h2−4
1
h2 −4
1
h2eIAI h
ihh
hhh
22
2
86 of 119
• The ith row of is ,
which is the version of .
• Therefore, IF relaxation on leaves only error inthe range of interpolation, then solving
determines the error exactly!
• In general, this will not be the case, but thisargument certainly leads to a plausible representationfor .
The ith row of looks alot like a row of !
A 2h
A h
A 2h 1212
1−−
)( h 2
A hΩ2h
A 2h
Ωh
87 of 119
The variational properties• The definition for that resulted from the
foregoing line of reasoning is useful for boththeoretical and practical reasons. Together withthe commonly used relationship betweenrestriction and prolongation we have the following“variational properties”:
IAIA 222 h
hhh
hh =
IcI 22
Thh
hh )(=
(Galerkin Condition )
for c in ℜ
A 2h
88 of 119
Properties of the Grid TransferOperators: Restriction
• Full Weighting: or
ℜ N-1 ℜ N/2-1
• For N=8,
• has rank and a null space withdimension .
I hhhh
22 Ω→Ω:I hh2 :
I hh
73
2 =121
121121
41
×
N1
2−I h
h2 IN )( h
h2
N2
89 of 119
Spectral properties of .
• How does act on the eigenvectors of ?
• Consider , 1 ≤ k ≤ N-1, 0 ≤ j ≤ N
• Good Exercise: show that
for 1 ≤ k ≤ N/2.
I hh2
I hh2 A h
Nkj
whjk,
π
= nis
π
=)( wN
kwI h
jkjhk
hh
2222
soc ,
≡ wc hjkk ,
2
90 of 119
Spectral properties of (cont.).
• i.e., [kth mode on ] = ck [ kth mode on ]
: N=8, k=2
: N=4, k=2
I hh2
I hh2 Ωh Ω2h
I hh2
Ωh
Ω2h
91 of 119
Spectral properties of (cont.).
• Let for , so that
• Then (another good exercise!)
I hh2
kNk −=′ NkN2
<′<2
1 <≤N
k
π
−=)( wN
kwI h
jkjhk
hh
222,′ 2
nis
≡ ws hjkk ,
2
92 of 119
Spectral properties of (cont.).
• i.e., [(N-k)th mode on ] = -sk [ kth mode on ]
• : N=8, k=6
• : N=4, k=2
I hh2
I hh2 Ωh Ω2h
I hh2
Ω2h
Ωh
93 of 119
Spectral properties of (cont.).
• Summarizing:
• Complementary modes:
I hh2
wcwI hkk
hk
hh
22 =
wswI hkk
hk
hh
22′ −=
wI hN
hh 22
/ = 0
21 <<
Nk
kNk −=′
wwW hk
hkk ,= naps ′
wWI hkk
hh
22 →
94 of 119
Null Space of .
• Observe that where i is oddand is the ith unit vector.
• Let .
• While the look oscillatory, they contain all ofthe Fourier modes of , i.e.,
• All the Fourier modes of are needed torepresent the null space of restriction!
I hh2
eAIN
=)( hi
hhh2 naps
ehi
≡η hi
hi eA
A hηi
=η kkN
ki
−
=
1
1wa ak ≠ 0
A h
0 000 −1 −12 0 0
95 of 119
Properties of the Grid TransferOperators: Interpolation
• Interpolation: or
: ℜ N/2-1 ℜ N-1
• For N=8,
• has full rank and null space .
I2hh
I 22
hhhh Ω→Ω:
I2hh =
1211
211
21
21
I2hh φ
96 of 119
Spectral properties of .
• How does act on the eigenvectors of ?
• Consider , 1 ≤ k ≤ N/2-1, 0 ≤ j ≤ N/2
• Good Exercise: show that the modes of areNOT preserved by , but that the space ispreserved:
I2hh
I2hh A 2h
/
π
=)(Nkj
wj
hk2
2nis
A 2h
I2hh W k
wN
kw
Nk
wI 2222
hk
hk
hk
hh
π
−
π
=2
nis2
soc ′
−= wswc hkk
hkk ′
97 of 119
Spectral properties of (cont.).
• Interpolation of smooth modes excitesoscillatory modes on .
• Note that if ,
• is second-order interpolation.
I2hh
wswcI2hkk
hkk
hh −= ′
Ω2h
Ωh
Nk
2«
wN
kOw
N
kOwI
2
2
2
22
2hk
hk
hk
hh
+
−
= 1 ′
≈ whk
I2hh
98 of 119
The Range of .
• The range of is the span of the columns of
• Let be the ith column of .
• All the Fourier modes of are needed torepresent Range( )
I2hh
I2hh I2
hh
ξi I2hh
:ξi
I2hh
≠,=ξ khkk
N
k
hi
−
=
1
1bwb 0
A h
99 of 119
Use all the facts to analyze thecoarse-grid correction scheme1) Relax times on :
2) Compute and restrict residual
3) Solve residual equation
4) Correct fine-grid solution .
• The entire process appears as
• The exact solution satisfies
Ωh vRv hh ← α1
vAfIf 22 hhhhh
h )−(←
fAv 2122 hhh )(=−
vIvv hhh
hh +← 22
vRAfIAIvRv hhhhh
hhh
hh )−()(+← α−α 2122
uRAfIAIuRu hhhhh
hhh
hh )−()(+← α−α 2122
α
100 of 119
Error propagation of the coarse-grid correction scheme
• Subtracting the previous two expressions, we get
• How does CG act on the modes of ? Assumeconsists of the modes and forand .
• We know how act on and .
eRAIAIIe hhhh
hhh
h )(−← 2122
α−
eGCe hh←
A h
whk′wh
k 21 /≤≤ NkkNk −=′
IAIAR−α ,)(,,, h
hhh
hh
2122
whk wh
k′
101 of 119
Error propagation of CG• For now, assume no relaxation . Then
is invariant under CG.
where
=α 0wwW h
khkk ,= naps ′
wswswGC hkk
hkk
hk += ′
wcwcwGC hkk
hkk
hk ′′ +=
Nk
ck
π
=2
soc 2N
ksk
π
=2
nis 2
102 of 119
CG error propagation for k << N
• Consider the case k << N (extremely smooth andoscillatory modes):
• Hence, CG eliminates the smooth modes but doesnot damp the oscillatory modes of the error!
wN
kOw
N
kOw kkk
+
→2
2
2
2
′
wN
kOw
N
kOw kkk
−
+
−
→ 112
2
2
2
′
103 of 119
Now consider CG with relaxation
• Next, include relaxation sweeps. Assume thatthe relaxation preserves the modes of(although this is often unnecessary). Letdenote the eigenvalue of associated with .For k < < N/2,
αR A h
λkR wk
wswsw kkkkkkk λ+λ→ ′αα
wcwcw kkkkkkk ′α′
α′′ λ+λ→
Small!
Small!
104 of 119
The crucial observation:
• Between relaxation and the coarse-grid correction, both smooth andoscillatory components of the errorare effectively damped.
• This is essentially the “spectral”picture of how multigrid works. Weexamine now another viewpoint, the“algebraic” picture of multigrid.
105 of 119
Recall the variational properties
• All the analysis that follows assumes that thevariational properties hold:
IAIA 222 h
hhh
hh =
IcI 22
Thh
hh )(=
(Galerkin Condition )
for c in ℜ
106 of 119
Algebraic interpretation ofcoarse-grid correction
• Consider the subspaces that make up andΩh Ω2h
IR )( 2hh IN )( h
h2Ωh
Ω2h IR )( hh2
I2hh I h
h2
For the rest of this talk, ‘R( )’refers to the Range of a linearoperator.
From the fundamental theoremof linear algebra:
orIRIN ))((⊥)( Thh
hh
22
IRIN )(⊥)( hh
hh 22
107 of 119
Subspace decomposition of .• Since has full rank, we can say, equivalently,
( where means that ).
• Therefore, any can be written as where and .
• Hence,
Ωh
A h
AINIR )(⊥)( 22
hhhA
hh h
yx ⊥A h yxA h =, 0
eh tse hhh +=IRs h
hh )(∈ 2 AINt hh
hh )(∈ 2
)(⊕)(=Ω hhh
hh
h AINIR 22
108 of 119
Characteristics of the subspaces
• Since for some , weassociate with the smooth components of .But, generally has all Fourier modes in it (recallthe basis vectors for ).
• Similarly, we associate with oscillatorycomponents of , although generally has allFourier modes in it as well. Recall that isspanned by therefore isspanned by the unit vectorsfor odd i, which “look” oscillatory.
qIs hhh
h = 22 q 22 hh Ω∈
sh eh
shI2
hh
t h
eh t h
IN )( hh2
=η ih
i eA AIN )( hhh2
e Thi ),...,,,,...,,(= 001000
109 of 119
qIAIAIIsGC hhh
hhh
hhh
h )(−= 22
2122
−
Algebraic analysis of coarse-gridcorrection
• Recall that (without relaxation)
• First note that if then .This follows since for someand therefore
• It follows that , that is, the nullspace of coarse-grid correction is the range ofinterpolation.
AIAIIGC )(−= 2122
hhh
hhh
−
IRs hh
h )(∈ 2 sGC h = 0qIs hh
hh = 2
2 q 22 hh Ω∈
= 0
IRGCN )(=)( 2hh
A 2h by Galerkin property
110 of 119
More algebraic analysis ofcoarse-grid correction
• Next, note that if then
• Therefore .
• CG is the identity on
AINt hhh
h )(∈ 2
tAIAIItGC hhhh
hhh
h )(−= 2122
−
ttGC hh =0
AIN )( hhh2
111 of 119
How does the algebraic picturefit with the spectral view?
• We may view in two ways:
=
that is,
or as
Ωh
Low frequency modes1 ≤ k ≤ N/2
High frequency modesN/2 < k < N Ωh ⊕
⊕=Ωh HL
)(⊕)(=Ω hhh
hh
h AINIR 22
112 of 119
Actually, each view is just partof the picture
• The operations we have examined work ondifferent spaces!
• While is mostly oscillatory, it isn’t .and while is mostly smooth, it isn’t .
• Relaxation eliminates error from .
• Coarse-grid correction eliminates error from
H
H
IR )( 2hh L
AIN )( hhh2
IR )( 2hh
113 of 119
How it actually works (cartoon)
IR )( 2hh
AIN )( hhh2
eh
sh
t h
H
L
114 of 119
Relaxation eliminates H, butincreases the error in .
IR )( 2hh
AIN )( hhh2
eh
sh
t h
H
L
IR )( 2hh
115 of 119
CG eliminates butincreases the error in H.
IR )( 2hh
AIN )( hhh2
eh
sh
t h
H
L
IR )( 2hh
116 of 119
Difficulties:anisotropic operators and grids
• Consider the operator :
• The same phenomenonoccurs for grids with muchlarger spacing in one directionthan the other:
),(=∂∂β−
∂∂
α−2
2
2
2yxf
yu
xu
-100 -50 0 50 1000
0.2
0.4
0.6
0.8
1
1.2 If then the GS-smoothingfactors in the x- and y-directions areshown at right.Note that GS relaxation does notdamp oscillatory components in the x-direction.
βα «
117 of 119
Difficulties: discontinuous oranisotropic coefficients
• Consider the operator :where
• Solutions: line-relaxation (where whole gridlinesof values are found simultaneously), and/or semi-coarsening (coarsening only in the strongly coupleddirection).
Again, GS-smoothing factors in the x- and y-directionscan be highly variable, and very often, GS relaxation doesnot damp oscillatory components in the one or bothdirections.
)∇),((•∇− uyxD
yxdyxd
yxdyxdyxD
),(),(
),(),(
=),(2212
2111
118 of 119
For nonlinear problems, use FAS (Full Approximation Scheme)
• Suppose we wish to solve: wherethe operator is non-linear. Then the linearresidual equation does not apply.
• Instead, we write the non-linear residual equation:
• This is transferred to the coarse grid as:
• We solve for and transfer theerror (only!) to the fine grid:
reA =
fuA =)(
ruAeuA =)(−)+(
uAfIeuA 2222 hhhhh
hhh ))(−(=)+(
euw 222 hhh +≡
uIwIuu hhh
hhh
hh )−(+← 222
119 of 119
Multigrid: increasingly, the righttool!
• Multigrid has been proven on a wide variety ofproblems, especially elliptic PDEs, but has also foundapplication among parabolic & hyperbolic PDEs,integral equations, evolution problems, geodesicproblems, etc.
• With the right setup, multigrid is frequently anoptimal (i.e., O(N)) solver.
• Multigrid is of great interest because it is one of thevery few scalable algorithms, and can be parallelizedreadily and efficiently!
Top Related