Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms....
Transcript of Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms....
![Page 1: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/1.jpg)
© Raymond W. Yeung 2014The Chinese University of Hong Kong
Chapter 9 The Blahut-Arimoto
Algorithms
![Page 2: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/2.jpg)
Single-Letter Characterization
• For a DMC p(y|x), the capacity
C = max
r(x)I(X;Y ),
where r(x) is the input distribution, gives the maximum asymptotically
achievable rate for reliable communication as the blocklength n!1.
• This characterization of C, in the form of an optimization problem, is
called a single-letter characterization because it involves only p(y|x) but
not n.
• Similarly, the rate-distortion function
R(D) = min
Q(x̂|x):Ed(X,X̂)D
I(X;
ˆ
X)
for an i.i.d. information source {Xk
} is a single-letter characterization.
![Page 3: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/3.jpg)
Single-Letter Characterization
• For a DMC p(y|x), the capacity
C = max
r(x)I(X;Y ),
where r(x) is the input distribution, gives the maximum asymptotically
achievable rate for reliable communication as the blocklength n!1.
• This characterization of C, in the form of an optimization problem, is
called a single-letter characterization because it involves only p(y|x) but
not n.
• Similarly, the rate-distortion function
R(D) = min
Q(x̂|x):Ed(X,X̂)D
I(X;
ˆ
X)
for an i.i.d. information source {Xk
} is a single-letter characterization.
![Page 4: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/4.jpg)
Single-Letter Characterization
• For a DMC p(y|x), the capacity
C = max
r(x)I(X;Y ),
where r(x) is the input distribution, gives the maximum asymptotically
achievable rate for reliable communication as the blocklength n!1.
• This characterization of C, in the form of an optimization problem, is
called a single-letter characterization because it involves only p(y|x) but
not n.
• Similarly, the rate-distortion function
R(D) = min
Q(x̂|x):Ed(X,X̂)D
I(X;
ˆ
X)
for an i.i.d. information source {Xk
} is a single-letter characterization.
![Page 5: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/5.jpg)
Single-Letter Characterization
• For a DMC p(y|x), the capacity
C = max
r(x)I(X;Y ),
where r(x) is the input distribution, gives the maximum asymptotically
achievable rate for reliable communication as the blocklength n!1.
• This characterization of C, in the form of an optimization problem, is
called a single-letter characterization because it involves only p(y|x) but
not n.
• Similarly, the rate-distortion function
R(D) = min
Q(x̂|x):Ed(X,X̂)D
I(X;
ˆ
X)
for an i.i.d. information source {Xk
} is a single-letter characterization.
![Page 6: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/6.jpg)
Numerical Methods
• When the alphabets are finite, C and R(D) are given as solutions of finite-dimensional optimization problems.
• However, these quantities cannot be expressed in closed-forms except forvery special cases.
• Even computing these quantities is not straightforward because the asso-ciated optimization problems are nonlinear.
• So we have to resort to numerical methods.
• The Blahut-Arimoto (BA) algorithms are iterative algorithms devised forthis purpose.
![Page 7: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/7.jpg)
Numerical Methods
• When the alphabets are finite, C and R(D) are given as solutions of finite-dimensional optimization problems.
• However, these quantities cannot be expressed in closed-forms except forvery special cases.
• Even computing these quantities is not straightforward because the asso-ciated optimization problems are nonlinear.
• So we have to resort to numerical methods.
• The Blahut-Arimoto (BA) algorithms are iterative algorithms devised forthis purpose.
![Page 8: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/8.jpg)
Numerical Methods
• When the alphabets are finite, C and R(D) are given as solutions of finite-dimensional optimization problems.
• However, these quantities cannot be expressed in closed-forms except forvery special cases.
• Even computing these quantities is not straightforward because the asso-ciated optimization problems are nonlinear.
• So we have to resort to numerical methods.
• The Blahut-Arimoto (BA) algorithms are iterative algorithms devised forthis purpose.
![Page 9: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/9.jpg)
Numerical Methods
• When the alphabets are finite, C and R(D) are given as solutions of finite-dimensional optimization problems.
• However, these quantities cannot be expressed in closed-forms except forvery special cases.
• Even computing these quantities is not straightforward because the asso-ciated optimization problems are nonlinear.
• So we have to resort to numerical methods.
• The Blahut-Arimoto (BA) algorithms are iterative algorithms devised forthis purpose.
![Page 10: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/10.jpg)
Numerical Methods
• When the alphabets are finite, C and R(D) are given as solutions of finite-dimensional optimization problems.
• However, these quantities cannot be expressed in closed-forms except forvery special cases.
• Even computing these quantities is not straightforward because the asso-ciated optimization problems are nonlinear.
• So we have to resort to numerical methods.
• The Blahut-Arimoto (BA) algorithms are iterative algorithms devised forthis purpose.
![Page 11: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/11.jpg)
Numerical Methods
• When the alphabets are finite, C and R(D) are given as solutions of finite-dimensional optimization problems.
• However, these quantities cannot be expressed in closed-forms except forvery special cases.
• Even computing these quantities is not straightforward because the asso-ciated optimization problems are nonlinear.
• So we have to resort to numerical methods.
• The Blahut-Arimoto (BA) algorithms are iterative algorithms devised forthis purpose.
![Page 12: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/12.jpg)
In this chapter
![Page 13: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/13.jpg)
• A general alternating optimization algorithm
• The Blahut-Arimoto algorithms for computing
1. the channel capacity C
2. the rate-distortion function R(D)
• Convergence of the alternating optimization algorithms
In this chapter
![Page 14: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/14.jpg)
• A general alternating optimization algorithm
• The Blahut-Arimoto algorithms for computing
1. the channel capacity C
2. the rate-distortion function R(D)
• Convergence of the alternating optimization algorithms
In this chapter
![Page 15: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/15.jpg)
In this chapter
• A general alternating optimization algorithm
• The Blahut-Arimoto algorithms for computing
1. the channel capacity C
2. the rate-distortion function R(D)
• Convergence of the alternating optimization algorithms
![Page 16: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/16.jpg)
9.1 Alternating Optimization
![Page 17: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/17.jpg)
A Double Supremum
Consider the double supremum
sup
u12A1
sup
u22A2
f(u1,u2).
• Ai is a convex subset of <nifor i = 1, 2.
• f : A1 ⇥A2 ! < is bounded from above, such that
– f is continuous and has continuous partial derivatives on A1 ⇥A2
– For all u2 2 A2, there exists a unique c1(u2) 2 A1 such that
f(c1(u2),u2) = max
u012A1
f(u01,u2),
and for all u1 2 A1, there exists a unique c2(u1) 2 A2 such that
f(u1, c2(u1)) = max
u022A2
f(u1,u02).
![Page 18: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/18.jpg)
A Double Supremum
Consider the double supremum
sup
u12A1
sup
u22A2
f(u1,u2).
• Ai is a convex subset of <nifor i = 1, 2.
• f : A1 ⇥A2 ! < is bounded from above, such that
– f is continuous and has continuous partial derivatives on A1 ⇥A2
– For all u2 2 A2, there exists a unique c1(u2) 2 A1 such that
f(c1(u2),u2) = max
u012A1
f(u01,u2),
and for all u1 2 A1, there exists a unique c2(u1) 2 A2 such that
f(u1, c2(u1)) = max
u022A2
f(u1,u02).
![Page 19: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/19.jpg)
A Double Supremum
Consider the double supremum
sup
u12A1
sup
u22A2
f(u1,u2).
• Ai is a convex subset of <nifor i = 1, 2.
• f : A1 ⇥A2 ! < is bounded from above, such that
– f is continuous and has continuous partial derivatives on A1 ⇥A2
– For all u2 2 A2, there exists a unique c1(u2) 2 A1 such that
f(c1(u2),u2) = max
u012A1
f(u01,u2),
and for all u1 2 A1, there exists a unique c2(u1) 2 A2 such that
f(u1, c2(u1)) = max
u022A2
f(u1,u02).
![Page 20: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/20.jpg)
A Double Supremum
Consider the double supremum
sup
u12A1
sup
u22A2
f(u1,u2).
• Ai is a convex subset of <nifor i = 1, 2.
• f : A1 ⇥A2 ! < is bounded from above, such that
– f is continuous and has continuous partial derivatives on A1 ⇥A2
– For all u2 2 A2, there exists a unique c1(u2) 2 A1 such that
f(c1(u2),u2) = max
u012A1
f(u01,u2),
and for all u1 2 A1, there exists a unique c2(u1) 2 A2 such that
f(u1, c2(u1)) = max
u022A2
f(u1,u02).
![Page 21: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/21.jpg)
A Double Supremum
Consider the double supremum
sup
u12A1
sup
u22A2
f(u1,u2).
• Ai is a convex subset of <nifor i = 1, 2.
• f : A1 ⇥A2 ! < is bounded from above, such that
– f is continuous and has continuous partial derivatives on A1 ⇥A2
– For all u2 2 A2, there exists a unique c1(u2) 2 A1 such that
f(c1(u2),u2) = max
u012A1
f(u01,u2),
and for all u1 2 A1, there exists a unique c2(u1) 2 A2 such that
f(u1, c2(u1)) = max
u022A2
f(u1,u02).
![Page 22: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/22.jpg)
A Double Supremum
Consider the double supremum
sup
u12A1
sup
u22A2
f(u1,u2).
• Ai is a convex subset of <nifor i = 1, 2.
• f : A1 ⇥A2 ! < is bounded from above, such that
– f is continuous and has continuous partial derivatives on A1 ⇥A2
– For all u2 2 A2, there exists a unique c1(u2) 2 A1 such that
f(c1(u2),u2) = max
u012A1
f(u01,u2),
and for all u1 2 A1, there exists a unique c2(u1) 2 A2 such that
f(u1, c2(u1)) = max
u022A2
f(u1,u02).
![Page 23: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/23.jpg)
• Let u = (u1,u2) and A = A1 ⇥ A2. Then the double supremum can be
written as
sup
u2Af(u).
• In other words, the supremum of f is taken over a subset of <n1+n2which
is equal to the Cartesian product of two convex subsets of <n1and <n2
.
• Let
f⇤= sup
u2Af(u).
A Double Supremum
![Page 24: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/24.jpg)
• Let u = (u1,u2) and A = A1 ⇥ A2. Then the double supremum can be
written as
sup
u2Af(u).
• In other words, the supremum of f is taken over a subset of <n1+n2which
is equal to the Cartesian product of two convex subsets of <n1and <n2
.
• Let
f⇤= sup
u2Af(u).
A Double Supremum
![Page 25: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/25.jpg)
A Double Supremum
• Let u = (u1,u2) and A = A1 ⇥ A2. Then the double supremum can be
written as
sup
u2Af(u).
• In other words, the supremum of f is taken over a subset of <n1+n2which
is equal to the Cartesian product of two convex subsets of <n1and <n2
.
• Let
f⇤= sup
u2Af(u).
![Page 26: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/26.jpg)
An Alternating Optimization
• Let u(k)= (u(k)
1 ,u(k)2 ) for k � 0, defined as follows.
• Let u(0)1 be an arbitrarily chosen vector in A1, and let u(0)
2 = c2(u(0)1 ).
• For k � 1, u(k)is defined by
u(k)1 = c1(u
(k�1)2 )
and
u(k)2 = c2(u
(k)1 ).
• Let
f (k)= f(u(k)
).
• Then
f (k) � f (k�1).
![Page 27: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/27.jpg)
An Alternating Optimization
• Let u(k)= (u(k)
1 ,u(k)2 ) for k � 0, defined as follows.
• Let u(0)1 be an arbitrarily chosen vector in A1, and let u(0)
2 = c2(u(0)1 ).
• For k � 1, u(k)is defined by
u(k)1 = c1(u
(k�1)2 )
and
u(k)2 = c2(u
(k)1 ).
• Let
f (k)= f(u(k)
).
• Then
f (k) � f (k�1).
![Page 28: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/28.jpg)
An Alternating Optimization
• Let u(k)= (u(k)
1 ,u(k)2 ) for k � 0, defined as follows.
• Let u(0)1 be an arbitrarily chosen vector in A1, and let u(0)
2 = c2(u(0)1 ).
• For k � 1, u(k)is defined by
u(k)1 = c1(u
(k�1)2 )
and
u(k)2 = c2(u
(k)1 ).
• Let
f (k)= f(u(k)
).
• Then
f (k) � f (k�1).
![Page 29: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/29.jpg)
An Alternating Optimization
• Let u(k)= (u(k)
1 ,u(k)2 ) for k � 0, defined as follows.
• Let u(0)1 be an arbitrarily chosen vector in A1, and let u(0)
2 = c2(u(0)1 ).
• For k � 1, u(k)is defined by
u(k)1 = c1(u
(k�1)2 )
and
u(k)2 = c2(u
(k)1 ).
• Let
f (k)= f(u(k)
).
• Then
f (k) � f (k�1).
![Page 30: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/30.jpg)
An Alternating Optimization
• Let u(k)= (u(k)
1 ,u(k)2 ) for k � 0, defined as follows.
• Let u(0)1 be an arbitrarily chosen vector in A1, and let u(0)
2 = c2(u(0)1 ).
• For k � 1, u(k)is defined by
u(k)1 = c1(u
(k�1)2 )
and
u(k)2 = c2(u
(k)1 ).
• Let
f (k)= f(u(k)
).
• Then
f (k) � f (k�1).
![Page 31: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/31.jpg)
An Alternating Optimization
• Let u(k)= (u(k)
1 ,u(k)2 ) for k � 0, defined as follows.
• Let u(0)1 be an arbitrarily chosen vector in A1, and let u(0)
2 = c2(u(0)1 ).
• For k � 1, u(k)is defined by
u(k)1 = c1(u
(k�1)2 )
and
u(k)2 = c2(u
(k)1 ).
• Let
f (k)= f(u(k)
).
• Then
f (k) � f (k�1).
![Page 32: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/32.jpg)
f(u(0)1 ,u(0)
2 )
f(u(1)1 ,u(0)
2 )
f(u(1)1 ,u(1)
2 )
f(u(2)1 ,u(1)
2 )
f(u(2)1 ,u(2)
2 )
f(u(0)) = f (0)
f(u(1)) = f (1)
f(u(2)) = f (2)
![Page 33: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/33.jpg)
f(u(0)1 ,u(0)
2 )
f(u(1)1 ,u(0)
2 )
f(u(1)1 ,u(1)
2 )
f(u(2)1 ,u(1)
2 )
f(u(2)1 ,u(2)
2 )
f(u(0)) = f (0)
f(u(1)) = f (1)
f(u(2)) = f (2)
![Page 34: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/34.jpg)
f(u(0)1 ,u(0)
2 )
f(u(1)1 ,u(0)
2 )
f(u(1)1 ,u(1)
2 )
f(u(2)1 ,u(1)
2 )
f(u(2)1 ,u(2)
2 )
f(u(0)) = f (0)
f(u(1)) = f (1)
f(u(2)) = f (2)
![Page 35: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/35.jpg)
f(u(0)1 ,u(0)
2 )
f(u(1)1 ,u(0)
2 )
f(u(1)1 ,u(1)
2 )
f(u(2)1 ,u(1)
2 )
f(u(2)1 ,u(2)
2 )
f(u(0)) = f (0)
f(u(1)) = f (1)
f(u(2)) = f (2)
f(u(1)1 ,u(0)
2 )
![Page 36: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/36.jpg)
f(u(0)1 ,u(0)
2 )
f(u(1)1 ,u(0)
2 )
f(u(1)1 ,u(1)
2 )f(u(1)1 ,u(1)
2 )
f(u(2)1 ,u(1)
2 )
f(u(2)1 ,u(2)
2 )
f(u(0)) = f (0)
f(u(1)) = f (1)
f(u(2)) = f (2)
f(u(1)1 ,u(0)
2 )
![Page 37: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/37.jpg)
f(u(0)1 ,u(0)
2 )
f(u(1)1 ,u(0)
2 )
f(u(1)1 ,u(1)
2 )f(u(1)1 ,u(1)
2 )
f(u(2)1 ,u(1)
2 )f(u(2)1 ,u(1)
2 )
f(u(2)1 ,u(2)
2 )
f(u(0)) = f (0)
f(u(1)) = f (1)
f(u(2)) = f (2)
f(u(1)1 ,u(0)
2 )
![Page 38: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/38.jpg)
• Since the sequence f (k)is non-decreasing, it must converge because f is
bounded from above.
• We will show that f (k) ! f⇤ if f is concave.
• Replacing f by �f , the double supremum becomes the double infimum
inf
u12A1inf
u22A2f(u1,u2).
• The same alternating optimization algorithm can be applied to compute
this infimum.
• The alternating optimization algorithm will be specialized for computing
C and R(D).
![Page 39: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/39.jpg)
• Since the sequence f (k)is non-decreasing, it must converge because f is
bounded from above.
• We will show that f (k) ! f⇤ if f is concave.
• Replacing f by �f , the double supremum becomes the double infimum
inf
u12A1inf
u22A2f(u1,u2).
• The same alternating optimization algorithm can be applied to compute
this infimum.
• The alternating optimization algorithm will be specialized for computing
C and R(D).
![Page 40: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/40.jpg)
• Since the sequence f (k)is non-decreasing, it must converge because f is
bounded from above.
• We will show that f (k) ! f⇤ if f is concave.
• Replacing f by �f , the double supremum becomes the double infimum
inf
u12A1inf
u22A2f(u1,u2).
• The same alternating optimization algorithm can be applied to compute
this infimum.
• The alternating optimization algorithm will be specialized for computing
C and R(D).
![Page 41: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/41.jpg)
• Since the sequence f (k)is non-decreasing, it must converge because f is
bounded from above.
• We will show that f (k) ! f⇤ if f is concave.
• Replacing f by �f , the double supremum becomes the double infimum
inf
u12A1inf
u22A2f(u1,u2).
• The same alternating optimization algorithm can be applied to compute
this infimum.
• The alternating optimization algorithm will be specialized for computing
C and R(D).
![Page 42: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/42.jpg)
• Since the sequence f (k)is non-decreasing, it must converge because f is
bounded from above.
• We will show that f (k) ! f⇤ if f is concave.
• Replacing f by �f , the double supremum becomes the double infimum
inf
u12A1inf
u22A2f(u1,u2).
• The same alternating optimization algorithm can be applied to compute
this infimum.
• The alternating optimization algorithm will be specialized for computing
C and R(D).
![Page 43: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/43.jpg)
u1
u2
![Page 44: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/44.jpg)
u1
u2
![Page 45: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/45.jpg)
u1
u2
![Page 46: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/46.jpg)
u1
u2
![Page 47: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/47.jpg)
u1
u2
![Page 48: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/48.jpg)
u1
u2
![Page 49: Chapter 9 - Chinese University of Hong Kong · Chapter 9 The Blahut-Arimoto Algorithms. Single-Letter Characterization • For a DMC p(y|x), the capacity C = max r(x) I(X; Y ), where](https://reader034.fdocuments.net/reader034/viewer/2022042218/5ec41ca28007a1383a027c8d/html5/thumbnails/49.jpg)
u1
u2