Jie_VariableRate
Transcript of Jie_VariableRate
![Page 1: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/1.jpg)
Variable Rate Channel Capacity
Jie Ren 2013/4/26
![Page 2: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/2.jpg)
Reference
• This is a introduc?on of Sergio Verdu and Shlomo Shamai’s paper.
• Sergio Verdu and Shlomo Shamai, “Variable-‐Rate Channel Capacity”, IEEE Transac?ons on Informa?on Theory, vol. 56, No. 6, June 2010.
![Page 3: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/3.jpg)
Outline (1st talk)
• Conven?onal Channel Coding • Setup of Variable Rate Coding
![Page 4: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/4.jpg)
Conven?onal Channel Capacity
P(y|x) X
![Page 5: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/5.jpg)
Conven?onal Channel Capacity
• What if Channel state varies?
P(y|x,m) X
! = inf!sup!!
!(!;!!)!
![Page 6: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/6.jpg)
Example: BSC
• Crossover Probability p
![Page 7: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/7.jpg)
Example: BSC
• Crossover Probability p=0.5
! = 1− ℎ 0.5 = 0!
![Page 8: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/8.jpg)
Example: BSC
• Crossover Probability p~Uni(0,0.5)
! = inf!1− ℎ ! = 0!
![Page 9: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/9.jpg)
Conven?onal Channel Capacity
• What if Channel state varies?
P(y|x,m) X
! = inf!sup!!
!(!;!!)!
![Page 10: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/10.jpg)
Outline
• Conven?onal Channel Coding • Setup of Variable Rate Coding • Variable-‐to-‐Fixed Capacity • Fixed-‐to-‐Variable Capacity • Variable-‐Blocklength Capacity • Examples
![Page 11: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/11.jpg)
Setup of Variable Rate Coding
• Fixed-‐to-‐fixed coding • Variable-‐to-‐fixed coding • Fixed-‐to-‐variable coding • Variable-‐blocklength coding
![Page 12: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/12.jpg)
Fixed-‐to-‐fixed
X1…Xn Channel B1…Bk B1…BK
![Page 13: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/13.jpg)
Fixed-‐to-‐fixed
X1…Xn Channel B1…Bk B1…BK
![Page 14: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/14.jpg)
Variable-‐to-‐Fixed
! :!: {0,1}!! → !!!
X1…Xn Channel B1…Bmk B1…Bm
!!::!! → {0,1}!! !
![Page 15: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/15.jpg)
Variable-‐to-‐Fixed
X1…Xn Channel B1…Bmk B1…Bm
![Page 16: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/16.jpg)
Fixed-‐to-‐variable
!!:: {0,1}! → !!!
X1…… Channel B1…Bk B1…Bk
![Page 17: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/17.jpg)
Fixed-‐to-‐variable
X1…… Channel B1…Bk B1…Bk
![Page 18: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/18.jpg)
Variable-‐blocklength
!!→: {0,1}! → !!!
!→!:!! → {0,1}! !
![Page 19: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/19.jpg)
Nota?ons
![Page 20: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/20.jpg)
Basic Rela?onship
![Page 21: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/21.jpg)
Example: BSC
![Page 22: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/22.jpg)
Example: BSC
![Page 23: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/23.jpg)
Variable Rate Channel Capacity
Jie Ren 2013/4/30
![Page 24: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/24.jpg)
Outline (2nd talk)
• Reviews • Theorems • Examples
![Page 25: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/25.jpg)
Review: Defini?ons
• Fixed-‐to-‐fixed capacity
![Page 26: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/26.jpg)
Review: Defini?ons
• Variable-‐to-‐fixed capacity
![Page 27: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/27.jpg)
Review: Defini?ons
• Fixed-‐to-‐variable & upper fixed-‐to-‐variable
![Page 28: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/28.jpg)
Review: Defini?ons
• ε capacity
lim!→!
!! ≤ !!
![Page 29: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/29.jpg)
Basic Rela?onship
![Page 30: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/30.jpg)
State-‐Dependent Channels
![Page 31: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/31.jpg)
State-‐Dependent Channels
• Finite alphabets
![Page 32: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/32.jpg)
Outline (2nd talk)
• Reviews • Theorems • Examples
![Page 33: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/33.jpg)
Theorem 7
• Two possible states
• Variable-‐to-‐fixed capacity in SDC
!!!|!! !! !! = !! !!(!!|!!)!
!!!+ !! !!(!!|!!)
!
!!!!
! = max!!"
{min ! !,!! , ! !,!!
+max{!!! !,!! ! ,!!! !,!! ! }}!
![Page 34: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/34.jpg)
Theorem 12
• Fixed-‐to-‐fixed capacity in SDC
![Page 35: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/35.jpg)
Theorem 13
• Fixed-‐to-‐variable capacity in SDC
![Page 36: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/36.jpg)
Theorem 14
• Suppose
• Then the fixed-‐to-‐variable capacity in SDC
![Page 37: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/37.jpg)
Theorem 15
• Upper-‐fixed-‐to-‐variable capacity in SDC
![Page 38: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/38.jpg)
Outline (2nd talk)
• Reviews • Theorems • Examples
![Page 39: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/39.jpg)
Example 1
• With probability q
• With probability 1-‐q
C0
![Page 40: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/40.jpg)
Example 1
• By defini?on
![Page 41: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/41.jpg)
Example 2
• With probability π0 : BSC(δ0) • With probability π1 : BSC(δ1), 0.5>δ1>δ0
![Page 42: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/42.jpg)
Example 2
! = !!!!!!!!!!!!!!!
![Page 43: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/43.jpg)
Example 2
• By theorem 12
![Page 44: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/44.jpg)
Example 2
• By theorem 7
![Page 45: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/45.jpg)
Example 2
• By theorem 14 & 15
![Page 46: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/46.jpg)
Example 3
• Channel alphabet {0…1023} • With probability 0.9: no error • With probability 0.1: yi=(xi>0)
![Page 47: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/47.jpg)
Example 3
• By theorem 13
![Page 48: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/48.jpg)
Example 3
• Subop?mal scheme
![Page 49: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/49.jpg)
Example 4
• Binary erasure channel BEC(E) – E random variable – Stay constant during the dura?on of the codeword
![Page 50: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/50.jpg)
Example 4
![Page 51: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/51.jpg)
Example 4
• E~Uni(0,1)
![Page 52: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/52.jpg)
Example 5
• Gaussian channel with nonergodic fading
![Page 53: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/53.jpg)
Example 5
• Gaussian channel with nonergodic fading
![Page 54: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/54.jpg)
Example 5
• Gaussian channel with nonergodic fading
![Page 55: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/55.jpg)
Example 5
• Gaussian channel with nonergodic fading
![Page 56: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/56.jpg)
Variable Rate Channel Capacity
Jie Ren 2013/5/7
![Page 57: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/57.jpg)
Outline (3rd talk)
• Theorems and proofs
![Page 58: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/58.jpg)
Theorems and Proofs
![Page 59: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/59.jpg)
Theorem 1
• For all channels
![Page 60: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/60.jpg)
Proof
• Consider a fixed-‐to-‐fixed code
• Let
![Page 61: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/61.jpg)
Theorem 2
• If the channel input alphabet is finite
![Page 62: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/62.jpg)
Proof
• Prove by contradic?on
![Page 63: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/63.jpg)
Theorem 3
• State-‐dependent channel, suppose each component sa?sfies strong converse
![Page 64: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/64.jpg)
Proof
• The bound would be ?ght if the encoder knows the channel state
• Hence it’s an upper bound in general
![Page 65: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/65.jpg)
Theorem 4
• Capacity region or 2-‐user DMBC with degraded message sets
![Page 66: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/66.jpg)
Proof
• Thomas M. Cover, “Elements of Informa?on Theory”, page 568
![Page 67: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/67.jpg)
Theorem 5
• In state-‐dependent channels
![Page 68: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/68.jpg)
Proof
• Assume iden?ty permuta?on • Show
![Page 69: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/69.jpg)
Theorem 6
• Theorem 5 holds with equality if the K-‐user broadcast channels sa?sfy the strong converse
![Page 70: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/70.jpg)
Proof
• Number the channel states in order of increasing expected number of recovered bits
![Page 71: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/71.jpg)
Proof
• For an arbitrarily small δ
![Page 72: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/72.jpg)
Proof
• Hence,
![Page 73: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/73.jpg)
Theorem 7
• Suppose the channel has finite input/output alphabets and is memoryless.
![Page 74: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/74.jpg)
Proof
• When the cons?tuent channels are discrete memoryless, the broadcast channel with degraded message sets sa?sfies the strong converse.
• Theorem 7 follows from 4 and 6
![Page 75: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/75.jpg)
Theorem 8
• For all channels
![Page 76: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/76.jpg)
Proof
• Consider a fixed-‐to-‐fixed code
• Define fixed-‐to-‐variable code
![Page 77: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/77.jpg)
Theorem 9
• Suppose the channel has finite memory
![Page 78: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/78.jpg)
Proof
• Similar proof as theorem 8
![Page 79: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/79.jpg)
Theorem 10
• Assume either the input or output alphabets are finite
![Page 80: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/80.jpg)
Proof
• Jensen’s inequality • Counterpart of Theorem 2
![Page 81: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/81.jpg)
Theorem 11
• For all channels
![Page 82: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/82.jpg)
Proof
• Assume
![Page 83: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/83.jpg)
Proof
• Non-‐an?cipatory sesng • Compa?ble variable-‐to-‐fixed encoder
![Page 84: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/84.jpg)
Proof
• Construct a sequence of rateless encoders – Let the ith component be equal to the ith component of variable-‐to-‐fixed code
– For n sufficiently large
![Page 85: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/85.jpg)
Proof
![Page 86: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/86.jpg)
Proof
!!! !≤ !!!
!!! !≤ !
!!!!!
!
![Page 87: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/87.jpg)
Theorem 12
• In state-‐dependent channel
![Page 88: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/88.jpg)
Proof
• Shannon capacity • Has to sa?sfy the worst channel
![Page 89: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/89.jpg)
Theorem 13
• In state-‐dependent channel
![Page 90: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/90.jpg)
Proof
• Achievability proof • Scheme: – Fixed-‐to-‐variable – Repi?on auer the first transmission
![Page 91: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/91.jpg)
Theorem 14
• In state-‐dependent channel, suppose
• Then theorem 13 holds with equality
! = min!!
!!!(!!,!!)
!
!!!
!!
!
![Page 92: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/92.jpg)
Proof
• Converse proof • Data-‐processing inequality
![Page 93: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/93.jpg)
Theorem 15
• In state-‐dependent channel
![Page 94: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/94.jpg)
Proof
• Achievability – Assume the decoder has knowledge of the ergodic mode
– Apply theorem 8
! ≥ ! = max!!
!!!(!!,!!)!
!!!!
![Page 95: Jie_VariableRate](https://reader034.fdocuments.net/reader034/viewer/2022042818/55c3f552bb61eb7c068b4859/html5/thumbnails/95.jpg)
Proof
• Converse