A semantic similarity metric combining features and intrinsic information content
Extremal Problems of Information Combining Alexei Ashikhmin Information Combining: formulation of...
-
Upload
dulcie-wiggins -
Category
Documents
-
view
221 -
download
2
Transcript of Extremal Problems of Information Combining Alexei Ashikhmin Information Combining: formulation of...
Extremal Problems of Information Combining
Alexei Ashikhmin
Information Combining: formulation of the problem Mutual Information Function for the Single Parity Check Codes More Extremal Problems of Information Combining Solutions (with the help of Tchebysheff Systems) for the Single Parity Check Codes
Joint work with Yibo Jiang, Ralf Koetter, Andrew Singer
Encoder
Channel
Channel
Channel
APPDecoder
Information Transmission
Density function of the channel is not known
We only know
Optimization Problem
We assume that
and that the channel is symmetric
Problem 1
Among all probability distributions such that
determine the probability distribution that maximizes (minimizes)
the mutual information at the output of the optimal decoder
Var
iabl
e no
des
proc
essi
ng
Che
ck n
odes
pro
cess
ing
Interleaver
Inpu
t fro
m c
hann
el
From variable nodes
To variable nodes Decoder of single parity check code
Problem is Solved Already
1. I.Land, P. Hoeher, S.Huettinger, J. Huber, 2003
2. I.Sutskover, S. Shamai, J. Ziv, 2003
erasure
Repetition code: The Binary Erasure Channel (BEC) is the best
The Binary Symmetric Channel (BSC) is the worst
Single Parity Check Code:
is Dual of Repetition Code The Binary Erasure Channel (BEC) is the worst
The Binary Symmetric Channel (BSC) is the best
Our Goals
We would like to solve the optimization problem for the Single Parity Check Codes directly (without using duality)
Get some improvements
Soft Bits
We call soft bit, it has support on
Channel
erasure
Binary symmetric channel,
Gaussian Channel:
Channel
Channel
Channel
Decoder Single ParityCheck Code
Encoder Single ParityCheck Code
E.Sharon, A. Ashikhmin, S. Litsyn Results:
Properties of the moments
Lemma 1. is nonnegative and nonincreasing
2. The ratio sequence is nonincreasing
Lemma
In the Binary Erasure Channel all moments are the same
Problem 2
Among all T-consistent probability distributions on [0,1]
such that
determine the probability distribution that maximizes
(minimizes) the second moment
Solution of Problem 2
Theorem
Among all binary-input symmetric-output channel
distributions with a fixed mutual information
Binary Symmetric Channel maximizes
and
Binary Erasure Channel minimizes
the second moment
Proof: We use the theory of Tchebysheff Systems
Lemma
Binary Symmetric, Binary Erasure and an arbitrary channel
with the same mutual information have the following layout of
Lemma
Let and
1)
2) if for and for
then
This is exactly our case
satisfy conditions of the previous lemma
Problem 1 on extremum of mutual information
and
Problem 2 on extremum of the second moment
are equivalent
Extrema of MMSE
It is known that the channel soft bit is the MMSE estimator fo
the channel input
Theorem Among all binary-input symmetric-output channels withfixed the Binary Symmetric Channel has the minimum MMSE: and the Binary Erasure Channel has the maximum MMSE:
Channel
How good the bounds are
Problem 3
1)
2)
Among all T-consistent channels find that maximizes
(minimizes)
Channel
Channel
Channel
Decoder Single ParityCheck Code
Encoder Single ParityCheck Code
Problem 4
Among all T-consistent probability distributions on [0,1]
such that
1)
2)
determine the probability distribution that maximizes
(minimizes) the fourth moment
Theorem The distribution with mass at , mass at
and mass at 0 maximizes
The distribution with mass at , mass at
and mass at 1 minimizes
Extremum densities
Maximizing
Minimizing:
Lemma
Channel with minimum and maximum and an arbitrary
channel with the same mutual information have the followin
layout of
Problem 3 on extremum of mutual information
and
Problem 4 on extremum of the fourth moment
are equivalent
Assume that
and is the same as in AWGN channel with this
Tchebysheff Systems
Definition
A set of real continues functions is called
Tchebysheff system (T-system) if for any real the linear
combination has at most distinct roots at
Definition
A distribution is a nondecreasing, right-continues function
The moment space, defined by
( is the set of valid distributions), is a closed convex cone.
For define
Problem For a given find
that maximizes (minimizes)
Theorem
If and are T-systems,
and then the extrema are attained uniquely with
distrtibutions and with finitely many mass points
Lower principalrepresentation
Upper principalrepresentation
Soft Bits
We call soft bit, it has support on
Lemma (Sharon, Ashikhmin, Litsyn)
If then
Channel
Random variables with this property are called T-consistent
Find extrema of
Under constrains
Theorem
Systems and are T-systems on [0,1].
---------------------------------------------------------------------------------
the distribution that maximizes
has only one mass point at :
has probability mass at
and at
This is exactly the Binary Symmetric Channel