符号理論 ...coding theory
-
Upload
guinevere-mckinney -
Category
Documents
-
view
75 -
download
6
description
Transcript of 符号理論 ...coding theory
符号理論 ...coding theory
The official language of this course is Englishslides and talks are (basically) in EnglishI will accept questions and comments in Japanese also.
omnibus-style lecture ... collection of several subjects“take-home” test ... questions are given, solve in your home
1
The Name of the Class
Coding Theory?a branch of Information Theoryproperties/constructions of “codes”
source codes (for data compression)channel codes (for error-correction)and various codes for various purposes
2
middle-class of Information Theory,with some emphasis on the techniques of coding
this class =
relation to Information Theory
3
measuring ofinformation
source coding
channel coding
entropymutual information
Kraft’s inequalityHuffman code
linear codeHamming code
source coding theoremuniversal code
analysis of codes
channel coding theoremconvolutional code
Turbo & LDPC codes
codes for data recordingnetwork codingand more...
class plan
seven classes, one testOct. 8 review brief review of information theoryOct. 15compress arithmetic code, universal codesOct. 22analyze analysis of codes, weight distributionOct. 29struggle cyclic code, convolutional codeNov. 5 Shannon channel coding theoremNov. 12 frontier Turbo code, LDPC codeNov. 19 unique coding for various purposes
take-home testNov. 26 no classslides ... http://isw3.naist.jp/~kaji/lecture/
4
Information Theory
Information Theory (情報理論)is founded by C. E. Shannon in 1948focuses on mathematical theory of communicationgave essential impacts on today’s digital technology
wired/wireless communication/broadcastingCD/DVD/HDDdata compressioncryptography, linguistics, bioinformatics, games, ...
5
Claude E. Shannon1916-2001
the model of communication
A communication system can be modeled as;
6
C.E. Shannon, A Mathematical Theory of Communication,The Bell System Technical Journal, 27, pp. 379–423, 623–656, 1948.
engineering artifacts
Other components are “given” and not controllable.
the first step
precise measurement is essential in engineeringvs.
information cannot be measured
To handle information by engineering means,we need to develop a quantitative measure of
information.
7
𝑿Entropy makes it!
the model of information source
information source = a machinery that produces symbols.The symbol produced is determined probabilistically.Use a random variable to represent the produced symbol.
takes either one value in . denotes the probability that .
8
𝑋 𝑃 𝑋 (1 )=𝑃 𝑋 (2 )=…=𝑃 𝑋 (6)=1/6𝐷 ( 𝑋 )={1,2,3,4,5,6 }
(We mainly focus on memoryless & stationary sources.)
entropy
the entropy of :
9
the expected value of over all is sometimes called as a self information of . is sometimes called as an expected information of .
𝐻 (𝑋 )= ∑𝑥∈𝐷 (𝑋 )
−𝑃 𝑋 (𝑥) log 2𝑃 𝑋 (𝑥)(bit)
𝑋 bit
𝑃 𝑋 (1 )=𝑃 𝑋 (2 )=…=𝑃 𝑋 (6)=1/6
entropy and uncertainty ( 不確実さ)
10
𝑋 𝑃 𝑋 (1 )=0.9
bit
𝐻 ( 𝑋 )=−0.9 log20.9−0.02 log20.02− …−0.02 log 20.02 bit
𝑃 𝑋 (2 )=…=𝑃 𝑋 (6 )=0.02
cheat dice...easier to guess
More difficulty to guess the value of correctly,more entropy is.
entropy = the size of uncertainty
basic properties of entropy
... 【 nonnegative 】
... 【 smallest value 】when for one particular value in
... 【 largest value 】when for all
11
some more entropies
joint entropy
12
conditional entropy
𝐻 ( 𝑋 ,𝑌 )= ∑𝑥∈𝐷 (𝑋 )
∑𝑦∈𝐷 (𝑌 )
−𝑃 𝑋 ,𝑌 (𝑥 , 𝑦 ) log2𝑃 𝑋 ,𝑌 (𝑥 , 𝑦 ) .
𝐻 ( 𝑋|𝑌 )= ∑𝑦∈𝐷(𝑌 )
𝑃𝑌 (𝑦 ) ∑𝑥∈𝐷(𝑋 )
−𝑃 𝑋∨𝑌 (𝑥|𝑦 ) log2𝑃 𝑋∨𝑌 (𝑥|𝑦 )
if and are independent, then
mutual information
mutual information between and
13
𝐻 (𝑋 )
𝐻 (𝑌∨𝑋)𝐻 (𝑌 )
𝐻 (𝑋∨𝑌 )
𝐻 (𝑋 ,𝑌 )
𝐼 (𝑋 ;𝑌 )
𝐼 (𝑋 ;𝑌 )¿𝐻 ( 𝑋 )−𝐻 (𝑋|𝑌 )¿𝐻 (𝑌 )−𝐻 (𝑌|𝑋 )¿𝐻 ( 𝑋 )+𝐻 (𝑌 ) −𝐻 (𝑋 ,𝑌 )
if and are independent𝐻 (𝑋 )
𝐻 (𝑌∨𝑋 )𝐻 (𝑌 )
𝐻 (𝑋∨𝑌 )
𝐻 (𝑋 ,𝑌 )
𝐼 (𝑋 ;𝑌 )
if & are independent:
example
binary symmetric channel (BSC) is transmitted is received
14
𝑋 𝑌0
1
0
1
𝑝
1−𝑝𝑝
1−𝑝
compute , assuming
for simplicity, define a binary entropy function
example solved
15
compute , assuming
10
𝑝 (1−𝑞)(1−𝑝 )𝑞
(1−𝑝 )(1−𝑞)𝑝𝑞
0 1𝑋𝑌𝑞1−𝑞
𝑝+𝑞−2𝑝𝑞 1−𝑝−𝑞+2𝑝𝑞𝑃 𝑋 ,𝑌 (𝑥 , 𝑦) 𝑃𝑌 (𝑦 )𝑃 𝑋 (𝑥)
𝐻 ( 𝑋 )=ℋ (𝑞) 𝐻 (𝑌 )=ℋ (𝑝+𝑞−2𝑝𝑞)𝐻 ( 𝑋 ,𝑌 )=− (1−𝑝 )𝑞 log2 (1−𝑝 )𝑞−𝑝𝑞 log2𝑝𝑞
−𝑝 (1−𝑞) log 2𝑝 (1−𝑞 )−(1−𝑝 )(1−𝑞) log2(1−𝑝 )(1−𝑞)
𝐼 ( 𝑋 ;𝑌 )=𝐻 ( 𝑋 )+𝐻 (𝑌 ) −𝐻 ( 𝑋 ,𝑌 )=ℋ (𝑝+𝑞−2𝑝𝑞) −ℋ (𝑝 )¿ℋ (𝑝 )+ℋ (𝑞)
good input and bad input
is a channel-specific constant is a controllable parameter
... the channel works poorly for input with or
... the channel works finely for input with
16
𝑋 𝑌0
1
0
1
𝑝
1−𝑝𝑝
1−𝑝
00.
080.
160.
240.
32 0.4
0.48
0.56
0.64
0.72 0.
80.
880.
96
𝑞
1−ℋ (𝑝 )
0
𝐼 (𝑋 ;𝑌 )
channel capacity
channel capacity = maximum of with = input to the channel = output from the channel
the channel capacity of BSC =
the channel capacity of a binary erasure channel = ,where is the probability of erasure
Channel capacities of some practical channels are also studied.
17
source coding
A source coding is to give a representation of information.The representation must be as small (short) as possible.
18
121464253… 00110101…
encoder
codewords
problem formulation
source symbol construct a code ,
where is a sequence (over )that is called a codeword for
our goal is to construct C so that is immediately decodable, andthe average codeword length of ,
is as small as possible.
19
code sourcesymbols
Huffman code
Code construction by iterative tree operations
1. prepare isolated nodes, each attached witha probability of a symbol (node = size-one tree)
2. repeat the following operation until all trees are joined to onea. select two trees and having the smallest probabilitiesb. join and by introducing a new parent nodec. the sum of probabilities of and is given to the new tree
20
David Huffman1925-1999
construction example
21
ABCDE
prob.0.20.10.30.30.1
codewords
source coding theorem
Shannon’s source coding theorem:There is no immediately decodable code with .
proof by Kraft’s inequality and Shannon’s lemma
We can construct an immediately decodable code with for any small .
construction of a block Huffman code
22
... two faces of source coding
“block” coding
23
AAABACADAEBABBBCBDBECA:
prob.0.040.020.060.060.020.040.010.030.030.010.06
:
codewordsABCDE
prob.0.20.10.30.30.1
problems of block Huffman code
The optimum code is obtained by grouping several symbols into one, andapplying Huffman code construction
practical problems arise:we need much storagewe need to know the probability distribution in advance
...solutions to these problems are discussed in this class.
24
channel coding
Errors are unavoidable in communication.
25
ABCABC ABCADC
Some errors are correctable by adding some redundancy.
ABC Alpha, Bravo, Charlie
ABCAlpha, Bravo, Charlie
Channel coding gives a clever way to introduce the redundancy.
linear code
linear code: practical class of channel codes
the encoding is made by using a generator matrix codeword
the decoding is made by using a parity check matrix syndrome The syndrome indicates the position of errors.
26
Hamming code
To construct a one-bit error correcting code,let column vectors of parity check matrix all different.
Hamming codedetermine a parameter enumerate all nonzero vectors with length use the vectors as columns of
27
Richard Hamming1915-1998
𝐻=(111010011010101011001) 𝐺=(
1000111010011000101010001011
)transpose
parameters of Hamming code
Hamming codedetermine design to have different column vectors
has rows and columnscode length# of information symbols# of parity symbols
28
234567
37
153163
127
14
112657
120code rate =
code rate and performance
If code rate = is large...more information in one codewordless number of symbols for error correctionThe error-correcting capability is weak in general.
29
code ratesmall large
erro
rca
pabi
lity
weak
strong
To have good error-correcting capability,we need to sacrifice the code rate...
channel coding theorem
Shannon’s channel coding theorem:Let be the capacity of the communication channel.
Among channel codes with rate ,there exists a code which can correct almost all errors.
There is no such codes in the class of codes with rate .
30
... two faces of channel coding
two coding theorems
source coding theorem:constructive solution given by Huffman code almost finished work
channel coding theoremno constructive solutiona number of studies have been madestill under investigation
remarkable classes of channel codesproof of the theorem
31
summary
today’s talk ... not self-contained summary of Information Theory
measuring of informationsource codingchannel coding
Students are encouraged to review basics of Information Theory.
32