High Lit Ed Errs Notes

8/8/2019 High Lit Ed Errs Notes

1/75

Information Theory and Coding

Computer Science Tripos Part II, Michaelmas Term11 Lectures by J G Daugman

1. Foundations: Probability, Uncertainty, and Information

2. Entropies Dened, and Why they are Measures of Information

3. Source Coding Theorem; Prex, Variable-, & Fixed-Length Codes

4. Channel Types, Properties, Noise, and Channel Capacity

5. Continuous Information; Density; Noisy Channel Coding Theorem

6. Fourier Series, Convergence, Orthogonal Representation

7. Useful Fourier Theorems; Transform Pairs; Sampling; Aliasing

8. Discrete Fourier Transform. Fast Fourier Transform Algorithms

9. The Quantized Degrees-of-Freedom in a Continuous Signal

10. Gabor-Heisenberg-Weyl Uncertainty Relation. Optimal Logons

11. Kolmogorov Complexity and Minimal Description Length


2/75


J G Daugman

Prerequisite courses: Probability; Mathematical Methods for CS; Discrete Mathematics

Aims

The aims of this course are to introduce the principles and applications of information theory.The course will study how information is measured in terms of probability and entropy, and therelationships among conditional and joint entropies; how these are used to calculate the capacityof a communication channel, with and without noise; coding schemes, including error correctingcodes; how discrete channels and measures of information generalise to their continuous forms;the Fourier perspective; and extensions to wavelets, complexity, compression, and efficient codingof audio-visual information.

Lectures Foundations: probability, uncertainty, information. How concepts of randomness,

redundancy, compressibility, noise, bandwidth, and uncertainty are related to information.Ensembles, random variables, marginal and conditional probabilities. How the metrics of information are grounded in the rules of probability.

Entropies dened, and why they are measures of information. Marginal entropy, joint entropy, conditional entropy, and the Chain Rule for entropy. Mutual informationbetween ensembles of random variables. Why entropy is the fundamental measure of infor-mation content.

Source coding theorem; prex, variable-, and xed-length codes. Symbol codes.The binary symmetric channel. Capacity of a noiseless discrete channel. Error correctingcodes.

Channel types, properties, noise, and channel capacity. Perfect communicationthrough a noisy channel. Capacity of a discrete channel as the maximum of its mutualinformation over all possible input distributions.

Continuous information; density; noisy channel coding theorem. Extensions of thediscrete entropies and measures to the continuous case. Signal-to-noise ratio; power spectraldensity. Gaussian channels. Relative signicance of bandwidth and noise limitations. The

Shannon rate limit and efficiency for noisy continuous channels. Fourier series, convergence, orthogonal representation. Generalised signal expan-

sions in vector spaces. Independence. Representation of continuous or discrete data bycomplex exponentials. The Fourier basis. Fourier series for periodic functions. Examples.

Useful Fourier theorems; transform pairs. Sampling; aliasing. The Fourier trans-form for non-periodic functions. Properties of the transform, and examples. NyquistsSampling Theorem derived, and the cause (and removal) of aliasing.

Discrete Fourier transform. Fast Fourier Transform Algorithms. Efficient al-gorithms for computing Fourier transforms of discrete data. Computational complexity.

Filters, correlation, modulation, demodulation, coherence.


3/75


4/75


Computer Science Tripos Part II, Michaelmas Term11 lectures by J G Daugman

1. Overview: What is Information Theory?

Key idea: The movements and transformations of information, just likethose of a uid, are constrained by mathematical and physical laws.

These laws have deep connections with: probability theory, statistics, and combinatorics

thermodynamics (statistical physics)

spectral analysis, Fourier (and other) transforms

sampling theory, prediction, estimation theory

electrical engineering (bandwidth; signal-to-noise ratio) complexity theory (minimal description length)

signal processing, representation, compressibility

As such, information theory addresses and answers thetwo fundamental questions of communication theory:

1. What is the ultimate data compression?(answer: the entropy of the data,H , is its compression limit.)

2. What is the ultimate transmission rate of communication?(answer: the channel capacity,C , is its rate limit.)

All communication schemes lie in between these two limits on the com-pressibility of data and the capacity of a channel. Information theory

can suggest means to achieve these theoretical limits. But the subjectalso extends far beyond communication theory.1


5/75

Important questions... to which Information Theory offers answers:

How should information be measured?

How much additional information is gained by some reduction inuncertainty?

How do the a priori probabilities of possible messages determinethe informativeness of receiving them?

What is the information content of a random variable?

How does the noise level in a communication channel limit its

capacity to transmit information? How does the bandwidth (in cycles/second) of a communication

channel limit its capacity to transmit information?

By what formalism should prior knowledge be combined withincoming data to draw formally justiable inferences from both?

How much information in contained in a strand of DNA?

How much information is there in the ring pattern of a neurone?

Historical origins and important contributions:

Ludwig BOLTZMANN (1844-1906), physicist, showed in 1877 that

thermodynamic entropy (dened as the energy of a statistical en-semble [such as a gas] divided by its temperature: ergs/degree) isrelated to the statistical distribution of molecular congurations,with increasing entropy corresponding to increasing randomness. Hemade this relationship precise with his famous formulaS = k logW where S denes entropy, W is the total number of possible molec-ular congurations, andk is the constant which bears Boltzmanns

name: k =1.38 x 10 16

ergs per degree centigrade. (The aboveformula appears as an epitaph on Boltzmanns tombstone.) This is2


6/75

equivalent to the denition of the information (negentropy) in anensemble, all of whose possible states are equiprobable, but with aminus sign in front (and when the logarithm is base 2,k=1.) Thedeep connections between Information Theory and that branch of

physics concerned with thermodynamics and statistical mechanics,hinge upon Boltzmanns work.

Leo SZILARD (1898-1964) in 1929 identied entropy with informa-tion. He formulated key information-theoretic concepts to solve thethermodynamic paradox known as Maxwells demon (a thought-experiment about gas molecules in a partitioned box) by showingthat the amount of information required by the demon about thepositions and velocities of the molecules was equal (negatively) tothe demons entropy increment.

James Clerk MAXWELL (1831-1879) originated the paradox calledMaxwells Demon which greatly inuenced Boltzmann and whichled to the watershed insight for information theory contributed bySzilard. At Cambridge, Maxwell founded the Cavendish Laboratory

which became the original Department of Physics. R V HARTLEY in 1928 founded communication theory with his

paper Transmission of Information . He proposed that a signal(or a communication channel) having bandwidth over a durationT has a limited number of degrees-of-freedom, namely 2T , andtherefore it can communicate at most this quantity of information.

He also dened the information content of an equiprobable ensembleof N possible states as equal to log2 N .

Norbert WIENER (1894-1964) unied information theory and Fourieranalysis by deriving a series of relationships between the two. Heinvented white noise analysis of non-linear systems, and made thedenitive contribution to modeling and describing the informationcontent of stochastic processes known asTime Series.

3


7/75

Dennis GABOR (1900-1979) crystallized Hartleys insight by formu-lating a general Uncertainty Principle for information, expressingthe trade-off for resolution between bandwidth and time. (Signalsthat are well specied in frequency content must be poorly localized

in time, and those that are well localized in time must be poorlyspecied in frequency content.) He formalized the Information Di-agram to describe this fundamental trade-off, and derived the con-tinuous family of functions which optimize (minimize) the conjointuncertainty relation. In 1974 Gabor won the Nobel Prize in Physicsfor his work in Fourier optics, including the invention of holography.

Claude SHANNON (together with Warren WEAVER) in 1949 wrotethe denitive, classic, work in information theory:Mathematical Theory of Communication. Divided into separate treatments forcontinuous-time and discrete-time signals, systems, and channels,this book laid out all of the key concepts and relationships that de-ne the eld today. In particular, he proved the famous Source Cod-ing Theorem and the Noisy Channel Coding Theorem, plus manyother related results about channel capacity.

S KULLBACK and R A LEIBLER (1951) denedrelative entropy (also calledinformation for discrimination, or K-L Distance. )

E T JAYNES (since 1957) developedmaximum entropy methodsfor inference, hypothesis-testing, and decision-making, based on thephysics of statistical mechanics. Others have inquired whether these

principles impose fundamental physical limits to computation itself. A N KOLMOGOROV in 1965 proposed that thecomplexity of a

string of data can be dened by the length of the shortest binaryprogram for computing the string. Thus the complexity of datais its minimal description length, and this species the ultimatecompressibility of data. The Kolmogorov complexityK of a stringis approximately equal to its Shannon entropyH , thereby unifyingthe theory of descriptive complexity and information theory.

4


8/75

2. Mathematical Foundations; Probability Rules;Bayes Theorem

What are random variables? What is probability?

Random variables are variables that take on values determined by prob-ability distributions. They may be discrete or continuous, in either theirdomain or their range. For example, a stream of ASCII encoded textcharacters in a transmitted message is a discrete random variable, witha known probability distribution for any given natural language. Ananalog speech signal represented by a voltage or sound pressure wave-form as a function of time (perhaps with added noise), is a continuousrandom variable having a continuous probability density function.

Most of Information Theory involves probability distributions of ran-dom variables, and conjoint or conditional probabilities dened overensembles of random variables. Indeed, the information content of asymbol or event is dened by its (im)probability. Classically, there aretwo different points of view about what probability actually means:

relative frequency: sample the random variable a great many timesand tally up the fraction of times that each of its different possiblevalues occurs, to arrive at the probability of each.

degree-of-belief: probability is the plausibility of a proposition orthe likelihood that a particular state (or value of a random variable)might occur, even if its outcome can only be decided once (e.g. theoutcome of a particular horse-race).

The rst view, the frequentist or operationalist view, is the one thatpredominates in statistics and in information theory. However, by nomeans does it capture the full meaning of probability. For example,the proposition that "The moon is made of green cheese" is one

which surely has a probability that we should be able to attach to it.We could assess its probability by degree-of-belief calculations which5


9/75

combine our prior knowledge about physics, geology, and dairy prod-ucts. Yet the frequentist denition of probability could only assigna probability to this proposition by performing (say) a large numberof repeated trips to the moon, and tallying up the fraction of trips on

which the moon turned out to be a dairy product....

In either case, it seems sensible that the less probable an event is, themore information is gained by noting its occurrence. (Surely discoveringthat the moon IS made of green cheese would be more informativethan merely learning that it is made only of earth-like rocks.)

Probability Rules

Most of probability theory was laid down by theologians: Blaise PAS-CAL (1623-1662) who gave it the axiomatization that we accept today;and Thomas BAYES (1702-1761) who expressed one of its most impor-tant and widely-applied propositions relating conditional probabilities.

Probability Theory rests upon two rules:

Product Rule:

p(A, B ) = joint probability of both A and B = p(A|B ) p(B )

or equivalently,= p(B |A) p(A)

Clearly, in caseA and B are independent events, they are not condi-tionalized on each other and so

p(A |B ) = p(A)and p(B |A) = p(B ),

in which case their joint probability is simplyp(A, B ) = p(A) p(B ).

6


10/75

Sum Rule:If event A is conditionalized on a number of other eventsB , then thetotal probability of A is the sum of its joint probabilities with allB :

p(A) = B p(A, B ) = B p(A|B ) p(B )

From the Product Rule and the symmetry that p(A, B ) = p(B, A ), itis clear that p(A |B ) p(B ) = p(B |A) p(A). Bayes Theorem then follows:

Bayes Theorem:

p(B |A) = p(A |B ) p(B ) p(A)

The importance of Bayes Rule is that it allows us to reverse the condi-tionalizing of events, and to compute p(B |A) from knowledge of p(A |B ), p(A), and p(B ). Often these are expressed asprior and posterior prob-abilities, or as the conditionalizing of hypotheses upon data.

Worked Example:Suppose that a dread disease affects 1/1000th of all people. If you ac-tually have the disease, a test for it is positive 95% of the time, andnegative 5% of the time. If you dont have the disease, the test is posi-tive 5% of the time. We wish to know how to interpret test results.

Suppose you test positive for the disease. What is the likelihood thatyou actually have it?

We use the above rules, with the following substitutions of dataDand hypothesis H instead of A and B :

D = data: the test is positive

H = hypothesis: you have the diseaseH = the other hypothesis: you do not have the disease7


11/75

Before acquiring the data, we know only that thea priori probabil-ity of having the disease is .001, which setsp(H ). This is called aprior .We also need to knowp(D ).

From the Sum Rule, we can calculate that thea priori probability p(D ) of testing positive, whatever the truth may actually be, is:

p(D ) = p(D |H ) p(H ) + p(D |H ) p(H ) =(.95)(.001)+(.05)(.999) = .051

and from Bayes Rule, we can conclude that the probability that you

actually have the disease given that you tested positive for it, is muchsmaller than you may have thought:

p(H |D ) =p(D |H ) p(H )

p(D )=

(.95)(.001)(.051)

= 0.019 (less than 2%).

This quantity is called theposterior probability because it is computedafter the observation of data; it tells us how likely the hypothesis is,given what we have observed. (Note: it is an extremely common humanfallacy to confoundp(H |D ) with p(D |H ). In the example given, mostpeople would react to the positive test result by concluding that thelikelihood that they have the disease is .95, since that is the hit rateof the test. They confoundp(D |H ) = .95 with p(H |D ) = .019, whichis what actually matters.)

A nice feature of Bayes Theorem is that it provides a simple mech-anism for repeatedly updating our assessment of the hypothesis as moredata continues to arrive. We can apply the rule recursively, using thelatest posterior as the new prior for interpreting the next set of data.In Articial Intelligence, this feature is important because it allows thesystematic and real-time construction of interpretations that can be up-

dated continuously as more data arrive in a time series, such as a owof images or spoken sounds that we wish to understand.

8


12/75

3. Entropies Dened, and Why They areMeasures of Information

The information content I of a single event or message is dened as thebase-2 logarithm of its probabilityp:

I = log2 p (1)

and its entropy H is considered the negative of this. Entropy can beregarded intuitively as uncertainty, or disorder. To gain informationis to lose uncertainty by the same amount, soI and H differ only insign (if at all):H = I . Entropy and information have units of bits.

Note that I as dened in Eqt (1) is never positive: it ranges between0 and as p varies from 1 to 0. However, sometimes the sign isdropped, and I is considered the same thing asH (as well do later too).

No information is gained (no uncertainty is lost) by the appearanceof an event or the receipt of a message that was completely certain any-way ( p = 1, so I = 0). Intuitively, the more improbable an event is,the more informative it is; and so the monotonic behaviour of Eqt (1)seems appropriate. But why the logarithm?

The logarithmic measure is justied by the desire for information tobe additive. We want the algebra of our measures to reect the Rules

of Probability. When independent packets of information arrive, wewould like to say that the total information received is the sum of the individual pieces. But the probabilities of independent eventsmultiply to give their combined probabilities, and so we must takelogarithms in order for the joint probability of independent eventsor messages to contribute additively to the information gained.

This principle can also be understood in terms of the combinatoricsof state spaces. Suppose we have two independent problems, one withn

9


13/75

possible solutions (or states) each having probabilitypn , and the otherwith m possible solutions (or states) each having probabilitypm . Thenthe number of combined states ismn , and each of these has probability pm pn . We would like to say that the information gained by specifying

the solution to both problems is thesum of that gained from each one.This desired property is achieved:

I mn = log2( pm pn ) = log2 pm + log2 pn = I m + I n (2)

A Note on Logarithms:

In information theory we often wish to compute the base-2 logarithmsof quantities, but most calculators (and tools like xcalc) only offerNapierian (base 2.718...) and decimal (base 10) logarithms. So thefollowing conversions are useful:

log2 X = 1.443 loge X = 3.322 log10 X

Henceforward we will omit the subscript; base-2 is always presumed.

Intuitive Example of the Information Measure (Eqt 1):

Suppose I choose at random one of the 26 letters of the alphabet, and weplay the game of 25 questions in which you must determine which let-ter I have chosen. I will only answer yes or no. What is the minimumnumber of such questions that you must ask in order to guarantee nd-

ing the answer? (What form should such questions take? e.g., Is it A?Is it B? ...or is there some more intelligent way to solve this problem?)

The answer to a Yes/No question having equal probabilities conveysone bit worth of information. In the above example with equiprobablestates, you never need to ask more than 5 (well-phrased!) questions todiscover the answer, even though there are 26 possibilities. Appropri-

ately, Eqt (1) tells us that the uncertainty removed as a result of solvingthis problem is about -4.7 bits.10


14/75

Entropy of Ensembles

We now move from considering the information content of a single event

or message, to that of anensemble. An ensemble is the set of outcomesof one or more random variables. The outcomes have probabilities at-tached to them. In general these probabilities are non-uniform, withevent i having probability pi , but they must sum to 1 because all possi-ble outcomes are included; hence they form a probability distribution:

i pi = 1 (3)

The entropy of an ensemble is simply the average entropy of all theelements in it. We can compute their average entropy by weighting eachof the log pi contributions by its probability pi :

H = I = i

pi log pi (4)

Eqt (4) allows us to speak of the information content or the entropy of a random variable, from knowledge of the probability distribution thatit obeys. (Entropy does not depend upon the actual values taken by the random variable! Only upon their relative probabilities.)

Let us consider a random variable that takes on only two values, onewith probability p and the other with probability (1 p). Entropy is aconcave function of this distribution, and equals 0 if p = 0 or p = 1:

11


15/75

Example of entropy as average uncertainty:

The various letters of the written English language have the followingrelative frequencies (probabilities), in descending order:

E T O A N I R S H D L C ....105 .072 .066 .063 .059 .055 .054 .052 .047 .035 .029 .023 ...

If they had been equiprobable, the entropy of the ensemble wouldhave been log2( 126) = 4.7 bits. But their non-uniform probabilities im-ply that, for example, an E is nearly ve times more likely than aC;surely this prior knowledge is a reduction in the uncertainty of this ran-dom variable. In fact, the distribution of English letters has an entropyof only 4.0 bits. This means that as few as only four Yes/No questionsare needed, in principle, to identify one of the 26 letters of the alphabet;not ve.

How can this be true?

That is the subject matter of Shannons SOURCE CODING THEOREM(so named because it uses the statistics of the source, thea priori probabilities of the message generator, to construct an optimal code.)

Note the important assumption: that the source statistics are known!

Several further measures of entropy need to be dened, involving the

marginal, joint, and conditional probabilities of random variables. Somekey relationships will then emerge, that we can apply to the analysis of communication channels.

Notation: We use capital letters X and Y to name random variables,and lower case lettersx and y to refer to their respective outcomes.These are drawn from particular setsA and B: x {a 1, a 2, ...a J }, and

y {b1, b2, ...bK }. The probability of a particular outcomep(x = a i)is denoted pi , with 0 pi 1 and i pi = 1.12


16/75

An ensemble is just a random variableX , whose entropy was denedin Eqt (4). A joint ensemble XY is an ensemble whose outcomesare ordered pairs x, y with x {a 1, a 2, ...a J } and y {b1, b2, ...bK }.

The joint ensembleXY denes a probability distributionp(x, y ) overall possible joint outcomesx, y .

Marginal probability: From the Sum Rule, we can see that the proba-bility of X taking on a particular value x = a i is the sum of the jointprobabilities of this outcome forX and all possible outcomes forY :

p(x = a i) =y

p(x = a i , y)

We can simplify this notation to: p(x) =y

p(x, y )

and similarly: p(y) =x

p(x, y )

Conditional probability: From the Product Rule, we can easily see that

the conditional probability that x = a i , given that y = b j , is: p(x = a i |y = b j ) =

p(x = a i , y = b j ) p(y = b j )

We can simplify this notation to: p(x |y) =p(x, y ) p(y)

and similarly: p(y|x) =p(x, y ) p(x)

It is now possible to dene various entropy measures for joint ensembles:

Joint entropy of XY

H (X, Y ) =x,y

p(x, y )log1

p(x, y )(5)

(Note that in comparison with Eqt (4), we have replaced the sign infront by taking the reciprocal of p inside the logarithm).13


17/75

From this denition, it follows that joint entropy is additive if X and Y are independent random variables:

H (X, Y ) = H (X ) + H (Y ) iff p(x, y ) = p(x) p(y)

Prove this.

Conditional entropy of an ensembleX , given that y = b j

measures the uncertainty remaining about random variable X after

specifying that random variableY

has taken on a particular valuey = b j . It is dened naturally as the entropy of the probability dis-tribution p(x |y = b j ):

H (X |y = b j ) =x

p(x |y = b j )log1

p(x |y = b j )(6)

If we now consider the above quantityaveraged over all possible out-

comes that Y might have, each weighted by its probabilityp(y), thenwe arrive at the...

Conditional entropy of an ensembleX , given an ensembleY :

H (X |Y ) =y

p(y)x

p(x |y)log1

p(x |y)(7)

and we know from the Sum Rule that if we move thep(y) term fromthe outer summation over y, to inside the inner summation overx , thetwo probability terms combine and become justp(x, y ) summed over allx, y . Hence a simpler expression for this conditional entropy is:

H (X |Y ) =x,y

p(x, y )log1

p(x |y)(8)

This measures the average uncertainty that remains aboutX , when Y is known.14


18/75

Chain Rule for Entropy

The joint entropy, conditional entropy, and marginal entropy for two

ensemblesX and Y are related by:H (X, Y ) = H (X ) + H (Y |X ) = H (Y ) + H (X |Y ) (9)

It should seem natural and intuitive that the joint entropy of a pair of random variables is the entropy of one plus the conditional entropy of the other (the uncertainty that it adds once its dependence on the rstone has been discounted by conditionalizing on it). You can derive the

Chain Rule from the earlier denitions of these three entropies.

Corollary to the Chain Rule:

If we have three random variablesX,Y,Z , the conditionalizing of the joint distribution of any two of them, upon the third, is also expressedby a Chain Rule:

H (X, Y |Z ) = H (X |Z ) + H (Y |X, Z ) (10)

Independence Bound on Entropy

A consequence of the Chain Rule for Entropy is that if we have manydifferent random variablesX 1, X 2,...,X n , then the sum of all their in-dividual entropies is an upper bound on their joint entropy:

H (X 1, X 2,...,X n ) n

i=1H (X i) (11)

Their joint entropy only reaches this upper bound if all of the randomvariables are independent.

15


19/75

Mutual Information betweenX and Y

The mutual information between two random variables measures the

amount of information that one conveys about the other. Equivalently,it measures the average reduction in uncertainty aboutX that resultsfrom learning aboutY . It is dened:

I (X ; Y ) =x,y

p(x, y )logp(x, y )

p(x) p(y)(12)

Clearly X says as much aboutY as Y says about X . Note that in caseX and Y are independent random variables, then the numerator insidethe logarithm equals the denominator. Then the log term vanishes, andthe mutual information equals zero, as one should expect.

Non-negativity: mutual information is always 0. In the event thatthe two random variables are perfectly correlated, then their mutualinformation is the entropy of either one alone. (Another way to saythis is: I (X ; X ) = H (X ): the mutual information of a random vari-able with itself is just its entropy. For this reason, the entropyH (X )of a random variableX is sometimes referred to as itsself-information .)

These properties are reected in three equivalent denitions for the mu-tual information betweenX and Y :

I (X ; Y ) = H (X ) H (X |Y ) (13)

I (X ; Y ) = H (Y ) H (Y |X ) = I (Y ; X ) (14)I (X ; Y ) = H (X ) + H (Y ) H (X, Y ) (15)

In a sense the mutual informationI (X ; Y ) is the intersection betweenH (X ) and H (Y ), since it represents their statistical dependence. In theVenn diagram given at the top of page 18, the portion of H (X ) thatdoes not lie withinI (X ; Y ) is just H (X |Y ). The portion of H (Y ) that

does not lie withinI (X ; Y ) is just H (Y |X ).

16


20/75

Distance D (X, Y ) between X and Y

The amount by which the joint entropy of two random variables ex-

ceeds their mutual information is a measure of the distance betweenthem:D (X, Y ) = H (X, Y ) I (X ; Y ) (16)

Note that this quantity satises the standard axioms for a distance:D (X, Y ) 0, D (X, X ) = 0, D (X, Y ) = D (Y, X ), andD (X, Z ) D (X, Y ) + D (Y, Z ).

Relative entropy, or Kullback-Leibler distance

Another important measure of the distance between two random vari-ables, although it does not satisfy the above axioms for a distance metric,is the relative entropy or Kullback-Leibler distance . It is also calledthe information for discrimination . If p(x) and q(x) are two proba-bility distributions dened over the same set of outcomesx , then theirrelative entropy is:

D KL ( p q) =x

p(x)logp(x)q(x)

(17)

Note that D KL ( p q) 0, and in casep(x) = q(x) then their distanceD KL ( p q) = 0, as one might hope. However, this metric is not strictly a

distance, since in general it lacks symmetry:D KL ( p q) = D KL (q p).

The relative entropy D KL ( p q) is a measure of the inefficiency of as-suming that a distribution is q(x) when in fact it isp(x). If we have anoptimal code for the distributionp(x) (meaning that we use on averageH ( p(x)) bits, its entropy, to describe it), then the number of additionalbits that we would need to use if we instead describedp(x) using an

optimal code forq(x), would be their relative entropyD KL ( p q).

17


21/75


22/75

! " #$ %& ') ( #1 02 3 45 ( $ 6

79 8A @C BE DF DG @C BE H IQ PS RU TV RX WY HS BE `) 8b ac Hd RX e) af TV 8V H1 @C BE P Ih gi 8b gp RX af q rG HS R9 @G 82 s$ DE HS Rt TV RX WY HS BE `) 8b a1 u9 s aS vw Ry x

af RY TV 8V HS HS 8V Hb e) a1 3 RX e) a1 8V x 8b W P af RX TV 8V Hd H 3 sU HS q) g R D BE H2 8b WY 8b ad s$ PS 8V ` RX WQ 8b s$ TV Ih 8V `) 8b 1 BF H

H IY R$ @ WA X aS s IY BF Tb sw DF DE q9 Pd R 8V P IY 8b a @C BE P IU s1 P @G R1 HS P s$ Pd 8 u9 s ad v$ R$ x af RX TV 8V Hd H 3 RX aC P IY 8 s$ D I) s ) 8V P

b

r r r) c r jG kl Bm W1 n Y e) af 8c ob ) 7U 8l Tb s W P IY 8b W1 HS R DE x 8 RX a P IY 8 HS P s$ Pd 8 RX TV Tb e s WY TV q2 eY Hd B WY

R$ @ 8V ) e) s$ PS BE RX WY H P IY BF H 8V ) s g DE 8l BE HC P az BF xY Bm s$ Dm V

D, 1/8

C, 1/8

B, 1/4

A, 1/2 A, 1/2

B, 3/8

C, 1/8

D, 1/8

B, 1/4

C, 1/8

A, 1/4

E, 1/4

{ BE X e) af 81 ob ) |& }~ aS s IY H af 8 af 8V HS 8b W PS Bm WY gp 8b gi RX af qY DE 8V HS H Hd RX e) af TV 8 s WY `2 Pf @G R Hd P s$ PS 8 u9 s aS vw Ry x af R$

TV 8V HS H

! W 8b WY 8b aS s$ D$ P IY 8b W @G 8& gc s$ q `) 8 n WY 8 s WY q n WY BF Pd 8 HS P s$ PS 8 af RX TV 8V Hd H @ BF P I HS P s$ PS 8V H

y ~ b S V V V V 3 &

kX r

@C BE P Ii P aS s WY HS BE PS BE RX W az RX ) s Y BE DE BF Pd BF 8V HG ) S E ) 8V Bm WY P IY 8 af RX ) s Y BE DE BE Pf qc R$ gi R$ xX Bm WY af RY g HS P s$ PS 8

PS R HS P s$ PS 8 @C BE P Ii P IY 8C 8b gi BE HS HS BE RX W1 R$ HS RX gi 8 HS q gi R Dm V { B az HS P& @G 8C Tb s Wp `) 8 n WY 8 P IY 8 8b W P af R q

R$ 8b s$ TV I1 HS P s$ PS 8 Bm W P IY 8i WY RX aS gc sw D g s W) WY 8b aV |

i 5 A z E ) DF R ) S E )

s WY `Q P IY 8b Wt P IY 82 8b W P af R qt R$ P IY 8 HS qY HS PS 8b g PS Rt ) 82 P IY 82 H e) g Rw P IY 8V Hd 82 Bm WY `) BE xX BE ` e) s$ D~ HS P s$ PS 8

8b W P af R q x$ s$ Dm eY 8V H @G 8V BF Y I PS 8V `Q @C BE P I P IY 82 HS P sw PS 8 RX TV Tb e s WY TV q Tb s$ DE Tb eY Dm s$ PS 8V `Q ! af RX g P IY 8 R$ @

8V ) e) s$ PS BE RX WY H V |

! i

E ) S E ) ) DE R ) S E ) X

G DE 8b s af DE q1 3 RX aC s~ HS Bm WY DE 8i HS P s$ Pd 8 rX @G 8l I) s x 8 P IY 8 8b W P af R qc Rw P IY 8l gp 8b gi RX af qY DE 8V HS H Hd RX e) af TV 8

1 4& ( % ') ( & ! w 4 ( (

3 PS 8b W @G 8c @C BE H IU PS R2 8 1 TV BF 8b W PS DE q af 8 az 8V HS 8b W P~ P IY 81 HS q gi R DE Hl 8b WY 8b aS sw PS 8V ` q- Hd RX gi 8c HS RY e) af TV 8

79 8 H I) sw DF D TV RX WY HS BE `) 8b a 8b WY TV RX `) Bm WY 1 P IY 8l Hd q gi ) R DF H s$ H Y Bm W) s af q `) BE BE PS Hb

ob


23/75

Symbols EncodingSourceencoder

{ BE X e) af 81 ob ) | BE HS Tb af 8V PS 8i gi 8b gi RX az qX DE 8V HS HC Hd RX e) af TV 8l s WY `A 8b WY TV RX `) 8b a

m " C ) h $ w " )

G RX WY Hd BF `) 8b a 8b WY TV RY `) B WY P IY 8l Hd q gi ) R DF H

y

! kX r 8b W P az R q1 l r sw H si n Y 8V `2 DE 8b WY P I 3 G Y DE RX TV v

Y Bm W) s af q `) BE BE PS Hb Rc 8b WY H e) af 8l @G 8 Tb s W1 `) 8V TV RY `) 8i P IY 8V HS 8 HS q) gi ) R DE H @ 8l WY 8V 8V ` |

3

3 5 s R$ @G 8b aG R$ &

3

3 S o R P IY 8b af @C BE HS 8

@ IY 8b af 8

BE H P IY 8i Dm s af 8V HS P Bm W PS 8V 8b a DE 8V HS H P I) s W i C IY 8 b 3 BE H P IY 8b W Y BE PS H 8b a

HS q) gi ) R D3 rX s WY `p sw H @ 8C vY WY R$ @h

3

3 P IY 8b W h i C IY 8C 8 TV BE 8b WY TV qc Rw P IY 8G TV RY `) B WY

BF H BE x 8b WA q |

7 IY 8b W9 P IY 8 HS q) g R DE H s az 8c 8V eY B af RY ) s Y DE 8 r& s WY ` BF Hi s Ry @G 8b a R$ C P @G R r

o1 s WY `

i G C R PS 8 P I) s$ PC B & @G 8 P af q1 PS R sw T IY BE 8V x 8i 5 h Bm W2 P IY BF H Tb s$ HS 8 @G 8l gi eY HS P sw DF DE RY Tb s$ PS 8c s$ P

DE 8b s$ HS P RX WY 8 8b WY TV RY `) Bm WY i PS R gi RY af 8G P I) s W RX WY 8C HS q) gi ) R DY i P IY BE H gp 8b s WY H& P I) sw P @ 8 s af 8C Bm WY Tb s s Y DE 8

R$ & e) WY BE eY 8V DE q2 `) 8V TV RY `) Bm WY

PS BE DE D @C BE P I 8V eY B af RY ) s Y DE 8 HS q gi R DE Hb r) ) eY P @ IY 8b W BF HC WY R PG s R$ @G 8b a R$ Pf @G R r P IY BE HG TV RY `) B WY

BE H& Bm WY 8 TV BE 8b W Pb ) PS R~ P af ql Pd Rl R$ x 8b af TV RX gp 8 P IY BF H @G 8C TV RX WY HS BE `) 8b a HS 8V ) eY 8b WY TV 8V H Rw HS q) gi ) R DE HG R$ DE 8b WY P I

9 s WY ` TV RX WY HS BE `) 8b a 8b WY TV RX `) Bm WY i 8b s$ TV I R Hd HS Bm Y DF 8 Hd 8V eY 8b WY TV 8 R$ DE 8b WY P Ic 9 sw H& sl Y DE RX TV vi R$ Y Bm W) s af q

) BE BE PS Hb r P IY 8b W1 @G 8 RX Y P s$ Bm W |

~ DE R - o

@ IY 8b af 81 BE Hp WY R$ @ P IY 8 s x 8b aS s$ 8p WY e) gi ) 8b a~ R$ C Y BF Pd H 8b a~ HS q gi R D3 R PS 8 P I) s$ Pl s$ Hi h 8V PS H

Dm s af 8 r

o

m " m l $ 9 m $ h $ )

! WU 8b WY 8b aS sw D @ 8 `) R- WY R Pi I) sb x 8 8V ) eY B af RX ) s Y DE 8 HS q) gi ) R DE Hb r s WY `U @ 8 @ RY eY DF ` IY R 8 PS R s$ TV IY BF 8V x 8

HS RX gp 8p gp RX af 8i TV RX g af 8V Hd HS 8V ` 3 RX aS g R$ G 8b WY TV RX `) Bm WY 9 q9 eY HS 8i R$ C x$ s af Bm s Y DE 8c DE 8b WY P IU TV RX `) 8V H - s W

8V ) s g DE 8 R$ H eY TV I s W 8b WY TV RX `) Bm WY 1 BE HC u RX af HS 8 TV RX `) 8 ` sw PS Bm WY i az RX g5 P IY 8 ` s qY HG R$ PS 8V DE 8V X aS s IY Hb

79 8 TV RX WY Hd BF `) 8b a s$ X s$ Bm W1 RX e) a HS Bm g DE 8 3 RX e) aG Hd q gi ) R D sw D I) s ) 8V P s WY ` Hd RX gi 8 R HS Hd B Y DE 8i xy s af Bm s Y DE 8

DE 8b WY P I TV RX `) 8V Hb |

G RY `) 8 o G RY `) 8 G RY `) 8

ob o

ob $ ob ) o

ob ) o o ob ) o o

ob ob o o o o o o


24/75

79 8 TV RX WY Hd BF `) 8b a 8b s$ TV I TV RY `) 8 Bm W2 P e) aS W |

o C C HS Bm WY i P IY BE HG 8b WY TV RX `) Bm WY r @ 8 n WY `1 P I) s$ P az 8V HS 8b W Pd 8V ` @C BE P I1 s~ HS 8V ) eY 8b WY TV 8 DF Bm v$ 8c o ) o r$ @G 8

) R WY R P vX WY R$ @ @ IY 8V P IY 8b a PS R `) 8V TV RY `) 8c s$ H RX a l C IY BE Hl gc s v$ 8V HC H eY TV I si TV RX `) 8

e) WY H s$ PS BE Hf ! s$ TV PS RX az q { e) af P IY 8b aV rY Bm W 8b WY 8b aS s$ D3 r) 8V x 8b W si TV RY `) 8l @ IY 8b af 8 H eY TV I s W s gi Y BE X eY BE P q

TV RX eY DE `2 ) 8l af 8V HS R DF x 8V ` e) WY BE ) eY 8V DF q q DF RY RX v Bm WY s$ P Y BE PS H e) af P IY 8b a s IY 8b s$ ` Bm W1 P IY 8 Hd P af 8b s g

3 s WY `2 ) s$ TV v P aS s$ TV v Bm WY X BE H e) WY H s$ PS BE Hf ! s$ TV PS RX af q

Y HS 8b af x 8 P I) s$ P 3 RX a P IY BF HG TV RY `) 8 rY P IY 8 TV RY `) Bm WY c aS s$ PS 8 r RX a& s x 8b aS sw 8 WY e) g 8b a R$ Y BF Pd H 8b a

HS q gi R D3 r) BE H BE x 8b W q |

!

~

o&

C h

@ IY BE T I BF H DF 8V Hd H P I) s W1 P IY 8l 8b W P af R q

C C IY BE H TV RX `) 8i BE Hl e) WY BE eY 8V DE q9 `) 8V TV RY ` s Y DE 8 " e) af P IY 8b aC P IY BE H TV RY `) 8c BE H Bm W PS 8b az 8V HS PS Bm WY Bm W P I) s$ P

@ 8 Tb s W1 `) 8V TV RX `) 8c V 3 3 p P I) s$ P& BE H WY Rc ) s$ TV v P aS s$ TV v Bm WY i BE H az 8V eY Bm af 8V ` RX WY TV 8

@ 8c I) s x 8c P IY 8 Y BE PS Hl R$ G P IY 8c 8b WY TV RY `) 8V ` Hd q gi ) R D& @G 8i Tb s W `) 8V TV RY `) 8 @C BE P IY RY eY Pl @C s$ BE PS Bm WY

RX a gi RX az 8 l { e) af P IY 8b a P IY BE Hl s$ DE HS RA H s$ PS BE Hf n 8V H P IY 8 f l TV RX WY `) BE PS BE RX W r P I) sw P BE Hl P IY 8b af 8i BF H

WY R TV RX `) 8 @G RX af ì @ IY BE T I BE H af 8 n 2 B3 8 H s gi 8C Y BE P s$ PS PS 8b ad W) R$ s DE RX WY 8b a TV RY `) 8C @G RX af `

! W1 P IY BE H Tb s$ HS 8 P IY 8~ TV RX `) Bm WY aS s$ PS 8 BE H 8V ) e) s$ D PS R P IY 8 8b W P af R q

G 7 IY BE DE 8i P IY BE H BE H `) 8 ! TV RX ` s Y DF 8 3 s WY `2 TV RY `) Bm WY ad s$ PS 8l BE H 8V e) sw D PS R1 P IY 8l 8b W P af R q s$ X s$ Bm W) V r

RX Y HS 8b af x 8 P I) s$ PC BE P `) RY 8V H WY R P I) s x 8l P IY 8 af 8 n af R 8b af Pf q s WY `2 BE H WY R P s W1 B WY Hd P s W P s

WY 8V RX eY HC TV RY `) 8

I) s W) WY RX W HG n af HS PC P IY 8V RX af 8b g BE H P IY 8 X f ! b " # $X f & %5 @ IY BF TV I BE Hb |

{ RX a1 s9 `) BE HS Tb az 8V PS 8 gi 8b gi RY af qX DE 8V HS H1 HS RX e) az TV 8- @ BF P I n WY BE PS 8U 8b W P az R qh t 3 RX a s W q

R Hd BF Pd BF x 8b ( ' BE P BE H R Hd HS Bm Y DF 8 PS R 8b WY TV RY `) 8c P IY 8i HS q) gi ) R DE H s$ P s W sb x 8b aS s$ 8l aS sw PS 8

i rX H eY T I P I) s$ Pb |

) '

{ RX a af RX Rw HS 8V 8

I) s W) WY RX W1 0 79 8b s x 8b aV i C IY BE HC BE H s$ DE HS Rc HS RX gp 8V PS Bm gi 8V H Tb sw DF DE 8V À P IY 8 WY R BE HS 8V DE 8V HS H

TV RY `) B WY i P IY 8V RY af 8b g s$ H BE P `) 8b sw DF HG @C BE P I TV RY `) B WY i @C BE P IY RY eY P& TV RX WY HS BE `) 8b aS sw PS BE RX W1 R$ WY R BE HS 8 af RX TV 8V Hd HS 8V H

B3 8 Y BE P TV RX aS aS e Pd BF RY W1 8V PS Tb V

C IY 8G 8b W P af R q ! e) WY TV PS BE RX Wi P IY 8b W af 8 af 8V HS 8b W PS H& s ! e) WY ` s gi 8b W P s$ DY DF Bm gi BE PG RX W~ P IY 8C WY e) g 8b a R$ Y BE PS H

RX W s x 8b aS sw 8 az 8V eY Bm af 8V À PS R af 8 af 8V Hd 8b W PC P IY 8 HS q gi R DE H R$ P IY 8 HS RY e) af TV 8

m " m 2 l 3 $ "

79 8i I) sb x 8 s$ Dm af 8b s$ `) q2 gp 8b W PS BE RX WY 8V À P IY 8 af 8 n af R 8b af Pf q @ 8 n WY ` P I) s$ PG 3 RX aC s af 8 n TV RY `) 8

PS Ri 8V Y BF Hd Pb rY BF PC gi eY HS P H s$ PS BE Hf 3 qc P IY 85 4l f 7 6V 9 8 ! 8 " E l Bm WY 8V ) e) s$ DE BE P q G C I) s$ PG BE Hb r si WY 8V TV 8V HS H s af q- 3 WY R P

H e TV BE 8b W P TV RY WY `) BF Pd BF RY Wi 3 RX a s TV RY `) 8 I) s xX Bm WY Y Bm W) s az q TV RY `) 8C @G RX af `) H @ BF P Il DE 8b WY P IA @ B @

C& CD C

E @G F PS Ri H s$ Pd BF Hz q1 P IY 8 af 8 n TV RX WY `) BE PS BE RX WA BE Hb |

F

I H

o

Q P

o

) o


25/75

R9 #$ ') () (T S ( $ 6 U ( #$ #1 4 & (V U

79 82 I) s x 8 TV RX WY Hd BF `) 8b az 8V ` P IY 82 `) BE HS Tb af 8V PS 8A HS RX e) af TV 8 rG WY R$ @ @G 8 TV RX WY HS BE `) 8b a s TV I) s W) WY 8V DC P I) az RX eY X I

@ IY BE TV Ic @G 8C @C BE H I PS R s$ Hd H& HS q) gi ) R DE HG 8b WY 8b aS s$ Pd 8V ` qi H eY TV I1 s HS RX e) az TV 8 qi HS RY gi 8 s

)

af R af Bm s$ PS 8

8b WY TV RY `) B WY 1 gi 8V T I) s WY BF H g ) @G 8 s$ DE HS Ri Bm W P af RX ` eY TV 8 P IY 8 BE `) 8b sl Rw " WY R BF Hd 8 B W PS R~ P IY 8 Hd qX HS Pd 8b g c P I) s$ P

BE Hi @ 81 TV RX WY HS BE `) 8b ai P IY 8 TV I) s W) WY 8V DG PS R- gp RX `) B 3 q9 P IY 81 B W eY P TV RX `) Bm WY s WY ` R HS HS Bm Y DE q 8b WY 8b ad s$ PS 8

HS RX gp 8l gi RX `) B n 8V `A x 8b af HS BE RX W

79 8G H IY RX eY DE `c `) BE HS Pd B WY Y eY BF H I 8V P @G 8V 8b W~ HS qY HS PS 8b gc sw PS BE T gp RX `) B n Tb s$ PS BE RX Wp R$ P IY 8G 8b WY TV RX `) 8V ` HS q gi R DE Hb r

B3 8 `) BE HS Pd RX af PS BE RX W rY s WY ` WY R BE HS 8 BE HS Pd RX af PS BE RX W~ BE H& @ IY 8b W s Wi Bm W eY PG TV RX `) 8 sw DF @C s qY H& af 8V H eY DE PS H& Bm W P IY 8

P IY 8 H s gi 8 RX eY P eY PC TV RY `) 8 P IY BE H af RY TV 8V HS HC Tb s W TV DE 8b s af DE q2 ) 8i af 8V x 8b af Hd 8V ` & C R BF Hd 8l RX W1 P IY 8 R P IY 8b a

I) s WY `A B W P af RY ` eY TV 8V H P IY 8 8V DE 8b gi 8b W P R$ & aS s WY `) RX g WY 8V HS HC Bm W Pd Rc P IY 8l af 8V H eY DE PS Bm WY RX eY P eY P TV RY `) 8

Symbols Sourceencoder Decoder

Symbols

ChannelX Y

{ BE X e) af 8c ob |& G RY `) B WY s WY ` `) 8V TV RY `) Bm WY R$ HS q) gi ) R DE H RX a P ad s WY Hf 3 8b aG R$ x 8b aG si T I) s W) WY 8V D

79 8 TV RX WY Hd BF `) 8b a1 s Wt Bm W eY P s$ D I) s ) 8V P1 W

Q X b b S Y XV `

k- s WY ` RX eY P eY P s$ D I) s ) 8V Pb a

Q c b b b S d cD e

kl s WY ` ad s WY `) RX g xy s az B s Y DF 8V H

s WY `b f @ IY BE TV I aS s WY 8 R$ x 8b a P IY 8V HS 8 sw D I) s ) 8V PS Hb

R PS 8G P I) s$ PG - s WY `h g WY 8V 8V ` WY R P& 8 P IY 8 H s gi 8 l 3 RX a 8V ) s g DE 8 @ 8C gc s qi I) sb x 8 P IY 8 Y Bm W) s af q

Bm W eY P sw D I) s ) 8V P o$ k s WY ` P IY 81 RX eY P eY P s$ D I) s 8V P o

! i

kY r @ IY 8b af 8

i

az 8 af 8V HS 8b W PS H

P IY 8~ `) 8V TV RX `) 8b a BE `) 8b W PS B 3 qX Bm WY HS RY gi 8 8b aS af RX a & C IY 8l ) BE HS Tb az 8V PS 8 gi 8b gp RX af qY DF 8V Hd HC T I) s W) WY 8V D Tb s W P IY 8b W

) 8i af 8 af 8V HS 8b W PS 8V ` s$ HC si HS 8V PC R$ P aS s WY HS BE PS BE RX W af RX ) s Y BE DE BE PS BE 8V Hb |

cq ps rt X

f5

cq pu r X

C I) s$ Pi BE Hi P IY 8 af RX ) s Y BF DE BE Pf q P I) s$ Pi B

X

BE Hi Bm WD v 8V TV PS 8V `Q Bm W PS R P IY 81 TV I) s W) WY 8V D3 r

cq p

BE Hc 8b gp BF Pd PS 8V `

cp

rt XY

1 BE H P IY 8 TV RX WY `) BE PS BE RX W) sw D af RX ) s Y BF DE BE Pf q IY BF HA Tb s W ) 8 @ af BE PS Pd 8b W sw H P IY 8 ! $X V D

%i 3 I $ |

w

y x

c rt X c rt X b b

ce

rt X

c Q rt X c$ q rt X b b

cD e rt X

c & rt XG ` c$ q rt XG ` b b cD e r XG `

7

R PS 8 P I) s$ PG @G 8 I) sb x 8 P IY 8 af R 8b af Pf qc P I) s$ PG 3 RX a 8V x 8b af q1 Bm W eY P Hd q gi ) R D rY @G 8 @C BE DE D 8V P Hd RX gi 8

P IY Bm WY 1 RX eY Pb |

e

p

H

cq ps rt X

o


26/75

8V X P @G 8C P s v$ 8& P IY 8 RX eY P eY P R$ s `) BE HS Tb af 8V PS 8 gi 8b gi RX af qY DE 8V HS H& Hd RX e) af TV 8C s$ H P IY 8C Bm W eY PG PS R s TV I) s W

WY 8V D3

Rl @G 8 I) s x 8 s$ HS HS RY TV Bm s$ PS 8V `c @C BE P Ip P IY 8C Bm W eY PC s$ D I) s ) 8V PG R$ P IY 8C TV I) s W) WY 8V D P IY 8 az RX ) s Y BE D

BE Pf qc `) BE HS P af Bm ) eY PS BE RX W R$ RX eY P eY P ! af RX g si gi 8b gi RX az qX DE 8V HS HG HS RX e) az TV 8

X

l 5 o

b V

kY 79 8

P IY 8b WU RX Y P s$ Bm W P IY 8 vf R Bm W P af RX ) s Y BE DE BE Pf q `) BE HS P az B ) eY Pd BF RY Wt R$ C P IY 8 aS s WY `) RY g xy s af Bm s Y DE 8V H

s WY `

f |

X Y cq p

X

f5

cq p

cq pu r X

3

X

79 8 Tb s W P IY 8b W n WY ` P IY 8l g s af Bm W) s$ D af RX ) s Y BF DE BE Pf q `) BE HS P af Bm ) eY PS BE RX WA R$ i rX P I) s$ PG BF H P IY 8 az RX ) s

Y BE DE BF Pf qA R$ RX eY P eY P HS q gi R D

cq p

s

)

8b s az B WY |

cq p

f5

cq p

`

H

cp

rt X

3

XY

G @G 8p I) s x 8c Q gt r @G 8 Rw PS 8b WA BE `) 8b W PS B q 8b s$ TV I RX eY P eY P Hd q gi ) R D s$ Hl 8V Bm WY 2 P IY 8c `) 8V Hd B az 8V `

af 8V H eY DE PC Rw HS RX gi 8 Bm W eY PC HS q) gi ) R D3 & aG @G 8 gc s qc HS 8V DE 8V TV PC HS RY gi 8 H e) Y HS 8V PC R$ RY eY P eY PG HS q gi R DE Hb r

3 RX a 8V ) s g DE 8G Bm W~ P IY 8G B W eY P o$ kG s WY `i RX eY P eY P o

7 i

kX 7 8 P IY 8b W~ `) 8 n WY 8C P IY 8 s x 8b ad s$ 8

af RX ) s Y BF DE BE Pf q R$ HS q gi R D 8b aS af RY a sw Hb |

e

p

H

pq

H

f

cQ p r X

e

p

H

`

H H

p

cq p rt X

3

X

u

s WY ` TV RX ad af 8V H RY WY `) B WY DF q r P IY 8l sb x 8b aS s$ 8 af RX ) s Y BE DE BE Pf q2 R$ TV RX aS af 8V TV PC af 8V TV 8 Pd BF RY W2 s$ Hl o

E i 3 D h ! X b $ )

C IY 8 Y Bm W) s az q Hd q gc gp 8V P af BE TG T I) s W) WY 8V D I) s$ H Pf @G R Bm W eY P s WY `c RX eY P eY P HS q) gi ) R DE H 3 eY H e) s$ DE DE qi @ af BE Pf

PS 8b W o$ k$ s WY ` s TV RY gc gi RX W az RX ) s Y BE DE BF Pf q r " r Rw d B WY TV RY aS af 8V TV P A `) 8V TV RX `) Bm WY 9 R$ s W B W eY P

s$ P P IY 8i RX eY P eY Pb P IY BE Hl TV RX eY DE ` 8 s1 HS Bm g DE BE HS PS BE T2 gi RY `) 8V D R$ s1 TV RX gc gi e) WY BE Tb s$ PS BE RX WY H~ DE B W) v r" n $

e) af 8l s)

Ry @G 8V x 8b a r PS R e) WY `) 8b af HS P s WY `9 P IY 8c s x 8b aS s$ B WY af R 8b af Pf q R$ P IY 8 8b ad af RX a aS s$ PS 8 ) 8V HS Tb af Bm ) 8V `

s Ry x 8 rb TV RX WY Hd BF `) 8b a P IY 8 n Y e) af 8C r @ IY 8b af 8G @G 8 I) s x 8 o q HS q) g R DE Hb r$ R$ @ IY BE TV Ii P IY 8 n af HS P I) s$ H

s af RX ) s Y BE DE BE P q R$ 8V Bm WY 2 af 8V TV 8V BE x 8V `9 Bm WA 8b aS af RX ai R$ & ob r s WY ` P IY 8c af 8b gc s$ Bm WY `) 8b a s az 8 s$ DE @C s qX H

af 8V TV 8V BE x 8V ` 8b a 3 8V TV PS DE q C IY 8b WU RX Y HS 8b af xY Bm WY 2 P I) s$ Pl gp R HS P R$ C P IY 8c Pd 8b aS gi H Bm W9 P IY 8c H e) g RY W P IY 8

af BE X I PC R$ 8V e) sw PS BE RX W1 u c s az 85 jV 8b af R |

c & rt X k

3

X k

o

ob s l

ob lG m

X


27/75


28/75

Sourceencoder Decoder

X Y XCorrection

ObserverCorrection data

{ BF Y e) af 8l ) o |& RX aS af 8V TV PS BE RX W1 Hd qX HS Pd 8b g

s$ HS Hd 8V `- s$ DE RX WY c P IY 8l TV RX aS az 8V TV PS BE RX W TV I) s W) WY 8V D PS R1 af 8V TV RX WY HS P ad eY TV PC P IY 8l Bm W eY Pb P IY BE H P e) aS WY HC RX eY P PS R

) 8i

B r

f

79 8 Tb s W e) af P IY 8b a af 8V @ af BE PS 8A P IY 8 8b W P az R qU ! e) WY TV PS BE RX W W1 c s$ H s9 H e) g R$ x 8b a P IY 8 v R Bm W P

af RX ) s Y BF DE BE Pf q `) BE HS P af Bm ) eY PS BE RX W |

W1

`

H

X

DE R E

X

S

o

`

H

X

DE R E

X

S

e

p

H

cq pu rt X

`

H

e

p

H

cq pu rt X

3

X

) DE R

X

S

`

H

e

p

H

X d cQ p

) DE R

X

S

8b WY TV 8l @G 8 RX Y P s$ Bm W s W 8V az 8V HS HS BE RX W1 3 RX a P IY 8l gi eY P e) s$ D" B W 3 RX aS g s$ PS BE RX W |

WA a

`

H

e

p

H

X Y cq p

DF R

X rt cq p

X

79 8 Tb s W `) 8V ` eY TV 8l x$ s af BE RX eY H af R 8b az PS BE 8V H R$ P IY 8 gi eY P e) s$ D Bm W 3 RX aS gc s$ Pd BF RY W |

o WA aC y zh )

R1 H IY R$ @5 P IY BE Hb r @G 8c WY R PS 8i P I) s$ PG

X rt cq p

3

cq p

C

X Y cq p

s WY ` H e) Y HS PS BE P eY Pd 8c P IY BE H

B W P IY 8 8V e) s$ Pd BF RY Wc s Y R xY 8 |

W2 { a

e

p

H

`

H

X d cQ p

) DE R

X Y cq p

X

cq p

z X

q eY HS 8 R$ P IY 8 Bm WY 8V e) sw DF BE Pf q2 @G 8 8V HS P s Y DE BF H IY 8V ` az 8V xX BE RX eY HS DE q


29/75

WA aC ~ B

s WY `| f s af 8 HS P s$ PS BE HS PS BE Tb s$ DE DE q2 Bm WY `) 8 8b WY `) 8b W Pb

s WY `T f s az 8 Bm WY `) 8 8b WY `) 8b W Pb r"

X Y cq p

C

X

3

cq p

V r IY 8b WY TV 8c P IY 8c DE R Pd 8b aS g

) 8V TV RX gi 8V Hy jV 8b af R

BE H HS q) gc gi 8V P af BE T r W2 { a a d Wc V

C HS Bm WY l

X r cQ p

3

cq p

&

cQ ps rt X

3

X

r) @G 8 RX Y P s$ Bm W |

W2 { a

`

H

e

p

H

XY y d c p

) DE R

XY rt cp

X

`

H

e

p

H

X d cq p

) DE R b }

cq pu r X

cq p

~

ai Y Wi 3

C C IY 8 af 8V TV 8V `) Bm WY DE 8b s$ `) Hl PS R P IY 8c RX xX BE RX eY H~ HS q g gi 8V P af BE Tc `) 8 n WY BE PS BE RX W RY a P IY 8 gi eY P e) s$ D

B W 3 RX aS g s$ PS BE RX W1 Bm W PS 8b ad gi HC R$ P IY 8l 8b W P af R qc R$ P IY 8 RX eY P eY P |

WA aC & a 9 a

r

Wc

G 79 8 `) 8 n WY 8l P IY 8 vf R Bm W PC 8b W P af R qc R$ Pf @G Rc aS s WY `) RX g x$ s af Bm s Y DE 8V H B W P IY 8 RX xX BE RX eY H gc s W

WY 8b aV r ) s$ HS 8V ` RY W1 P IY 8 vf R Bm W P af RX ) s Y BE DE BE P q `) BE HS P af Bm ) eY PS BE RX W |

W a 5

`

H

e

p

H

X Y cq p

DE R E

X Y cq p

d

C IY 8l gi eY P e) s$ D Bm W RX ad gc s$ PS BE RX W BF H P IY 8 gi RY af 8 W) sw P e) aS s$ DE DE q @ af BE PS PS 8b W |

WA a W h a 9 W a

I " i 3 D h )

G RX WY Hd BF `) 8b a P IY 8 TV RY WY `) BF Pd BF RY W) s$ D& 8b W P af R q2 s WY `9 gi eY P e) s$ D Bm W RX ad gc s$ PS BE RX W 3 RX aC P IY 8p Y Bm W) s az q2 HS q) g~

gi 8V P az BF T TV I) s W) WY 8V D3 C IY 8 Bm W eY P2 HS RX e) az TV 8 I) sw H2 s$ D I) s ) 8V P| W o$ k s WY ` s$ HS HS RY TV Bm s$ PS 8V `

af RX ) s Y BF DE BE PS BE 8V H ob ob Y k s WY ` P IY 8 T I) s W) WY 8V D gc s$ P af BE BE Hb |

oC

o

C IY 8b W P IY 8 8b W P af R q rX TV RY WY `) BF Pd BF RY W) s$ D 8b W P af R q s WY `2 gi eY P e) s$ D Bm W RY aS gc s$ PS BE RX WA s af 8 BE x 8b W q |

W1 o

W

r

a C DE R E f oG 2 DF R f oG 2

W2 { aC o& DF R E z oC ) DE R f oC

{ BE X e) af 82 s H IY R$ @C Hi P IY 8 Tb s s$ TV BE Pf q9 R$ P IY 8 TV I) s W) WY 8V DC s$ X s$ Bm WY HS P P aS s WY HS BE PS BE RX W az RX ) s Y BE DE BF Pf q

R PS 8 P I) s$ PG P IY 8 Tb s s$ TV BE Pf q R$ P IY 8 TV I) s W) WY 8V D ` az R HG PS R jV 8b af Ri @ IY 8b W1 P IY 8 P aS s WY HS BE PS BE RX W az RX ) s

Y BE DE BF Pf q1 BE Hl o& ) rY s WY `1 BF HC gc sw X Bm gi B jV 8V ` @ IY 8b W P IY 8 P aS s WY HS BE PS BE RX W af RX ) s Y BF DE BE Pf q BE H X RY aC oG c B @G 8

af 8V DE Bm s Y DE q9 P aS s WY H R Hd 8 P IY 8c HS q) g R DE Hl RX WA P IY 8i @C Bm af 8 @ 8c s$ DE HS RA 8V P P IY 8c gc s$ Y Bm gi e) g s gi RY e) W P

R$ Bm W 3 RX aS gc sw PS BE RX W1 P I) af RX eY Y I { BE X e) af 8 1 H IY R$ @C HG P IY 8 8d 8V TV PG RX W gi eY P e) s$ D Bm W RY aS gc s$ PS BE RX W 3 RX a

s$ HS q) gc gi 8V P af BE T B W eY P s$ D I) s 8V PS Hb


30/75

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0 .1 0 .2 0 .3 0 .4 0 .5 0 .6 0 .7 0 .8 0 .9 1P(transition)

I(X;Y)

I(X;Y)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.1

0.20.3

0.40.5

0.60.7

0.80.9

0

0.5

1

P(transition)

P(X=0)

{ BE X e) af 8l |& C s s$ TV BE Pf q1 R$ Y Bm W) s af q TV I) s W) WY 8V D s HS q g gi 8V P af BE T r ) C sw HS q g gi 8V P af BE T

4 & (V U ') o ') 3 V 6

79 81 @C BE H IU PS R2 `) 8 n WY 81 P IY 81 Tb s s$ TV BE Pf q- Rw sA TV I) s W) WY 8V D3 r& eY HS Bm WY P IY 8 gi RX `) 8V D R$ s1 ! af 8V 81 B W eY P

s$ D I) s 8V Pp s WY ` `) 8 8b WY `) 8b W Pi RX eY P eY Pc s$ D I) s 8V P2 BE x 8b W9 q P IY 8 TV I) s W) WY 8V DG gc s$ P af BE ) V 1 79 8

WY R PS 8~ P I) s$ PG 3 RX aC sp BE x 8b WA T I) s W) WY 8V D3 r B @G 8 @C BE H I Pd R gc sw X Bm gi B jV 8l P IY 8 gi eY P e) s$ D Bm W 3 RX aS gc sw PS BE RX W r

@G 8 gi eY HS P 8b a RX ad g5 sl gc s$ Y Bm gi B jb s$ PS BE RX W R$ x 8b a s$ DE D af RX ) s Y BE DE BE Pf qc `) BE HS P af Bm ) eY PS BE RX WY HG 3 RX a P IY 8 B W eY P

s$ D I) s 8V P i ) 79 8 `) 8 n WY 8i P IY 8c $X G & d $ r) `) 8b WY R PS 8V `2 qb r s$ Hb |

g s$

& d

WA a 3 ) ob

7 IY 8b WQ @ 8 s$ TV IY BF 8V x 8- P IY BF H2 ad s$ PS 8 @G 82 `) 8V HS Tb az B 89 P IY 8 HS RX e) az TV 8- s$ H ) 8V Bm WY g s$ PS TV IY 8V `h PS Rt P IY 8

TV I) s W) WY 8V D3

4 & (V U ') & !

" R R$ x 8b af TV RY gi 8 P IY 8 az RX Y DE 8b g R$ WY R BF Hd 8- Bm WQ P IY 8A HS qX Hd PS 8b g rG @G 82 gi BE X I P TV RX WY Hd BF `) 8b a sw `) `) B WY

af 8V ` e) WY ` s WY TV q ` e) af Bm WY P IY 8i 8b WY TV RY `) Bm WY af RY TV 8V HS H PS R1 R$ x 8b af TV RX gi 8 R Hd HS Bm Y DF 8 8b aS af RX af H IY 8 8V

s g DE 8V Hi P I) s$ Pl s af 8 eY HS 8V ` IY 8b af 8 s az 8 af 8V HS P az BF TV Pd 8V `9 PS R2 Hd RX e) af TV 8V Hl @ IY BF TV IU @ RY eY DF ` W) s$ P e) ad s$ DE DF q9 8

8b WY TV RY `) 8V `9 Bm W- s1 WY R BE HS 8V DE 8V HS H~ 8b W xY Bm af RX W) gi 8b W P sw HC n Y 8V ` HS B jV 8 Y DE RX TV v TV RY `) 8V H B3 8 s HS RX e) af TV 8i s$ D

I) s 8V P WA rY @ IY BE T IA I) s$ H 8V ) eY B af RX ) s Y DE 8i HS q) gi ) R DE Hb IY R$ @G 8V x 8b aV rY P IY 8 `) BE HS Tb eY Hd HS BE RX W2 s

)

DE BE 8V H

PS R gp RX af 8 8b WY 8b aS s$ D Hd RX e) af TV 8V H s WY `A xy s az B s Y DF 8~ DE 8b WY P I TV RY `) Bm WY Hd T IY 8b gi 8V H

WY 8 s af PS BE Tb eY Dm s a s$ H 8V TV P PS R ) 8 TV RX WY HS BE `) 8b af 8V `c Bm W~ af 8b s$ D) eY HS 8V H Rw T I) s W) WY 8V DY TV RY `) B WY BE H P I) s$ P g s W q


31/75

HS RX e) az TV 8V H @ IY BF TV I @ 8 s af 8 B W PS 8b af 8V Hd PS 8V ` Bm W 8b WY TV RX `) Bm WY RY a P aS s WY H gi BE HS HS BE RX WY HG I) sb x 8 s HS BE X WY B n Tb s W P

s gi RY e) W P R$ az 8V ` e) WY ` s WY TV q s$ Dm af 8b s$ `) q i G RX WY Hd BF `) 8b a HS 8b WY `) Bm WY - s BE 8V TV 81 R$ HS q) W P s$ TV PS BE Tb s$ DE DE q TV RX a

af 8V TV P s WY ` HS 8b g s W Pd BF Tb sw DF DE q2 gi 8b s WY B WY w eY D j WY DE BE H I RX a TV RX g eY PS 8b a af R Y aS s g PS 8V Y P P I) af RX eY X I1 s

TV I) s W) WY 8V D @ IY BE TV Ic aS s WY `) RY gi DE q TV RY aS aS e PS 8V `c RY W s x 8b ad s$ 8 oG Bm W ob TV I) s aS s$ TV PS 8b az H H eY TV Ic s$ H& gi BE X I P

) 8~ Bm W P az RX ` eY TV 8V ` q1 P aS s WY H gi BE HS HS BE RX W2 sw Tb af R HS HC sc aS sw P IY 8b aG HS BE TV v DE q2 " 8V DF 8V Hd qX HS Pd 8b gc V 8 m |

o C az B WY 1 af 8V Bm W RX az TV 8b gi 8b W Pd Hb r @G 8 az 8 R B WY PS R s$ `) x$ s WY TV 8

C 3 P& HG 8b s$ Hd q PS R1 af 8V TV R X WY BE HS 8~ H 8V 8V TV I

G 8V TV RX WY Hd P aS eY TV PS BE RX W~ ! af RX g P IY 8G R DF DE R$ @C Bm WY l ` eY 8 PS R TV RX aS aS e Pd BF RY Wi R$ oG Bm W o T I) s ad s$ TV PS 8b af H @G RX eY DE `

) 8~ TV RX g s ad s$ PS BE x 8V DE q Hd P aS s$ BE X I P RX az @ s af ` |

o C az B jV 1 af 8V Bm W RX az TV 8l 8b W PS Hb r) @G 8 af 8 R Bm WY 1 PS Rc s$ `) xy s WY TV 8

C 3 P& HG 8b s$ Hd q gi R1 af 8V TV R X WY BE HS 8~ H 8V 8V TV I

Ry @G 8V x 8b a r& @ IY BE DE 8 P IY 82 af 8V ` e) WY ` s WY TV qQ R$ P IY BE H HS RY e) af TV 8 af R PS 8V TV PS H s$ Y s$ Bm WY HS Pc H eY TV Ih aS s WY `) RX g

TV I) s aS s$ TV PS 8b a 8b aS az RX aV r) TV RX WY Hd BF `) 8b a P IY 8 8b aS az RX aG ` eY 8 PS R s IX e) gc s W gi BE Hf IY 8b s af Bm WY |

o C az B WY P I) af 8V 8 s WY `1 RX e) a 8b WY TV 8 r @ 8 af 8 R Bm WY PS Rc si ` s WY TV 8

C 3 P& HG 8b s$ Hd q PS R @ af 8V TV v sc WY BE TV 8 8b s$ TV I

C IY 8c TV RY `) Bm WY 9 WY 8V 8V `) H Pd R TV RX WY HS BE `) 8b a~ P IY 8c 8b aS af RY a T I) s ad s$ TV PS 8b af BE HS PS BE TV Hi R$ G P IY 8c TV I) s W) WY 8V D s WY `U `) 8

TV RY `) 8b aV r s WY ` P af q9 PS R9 s$ TV IY BE 8V x 82 s HS BE X WY B n Tb s W PT S `) BE HS P s WY TV 8! ) 8V Pf @G 8V 8b W Dm s eY HS Bm Y DE 89 8b WY TV RY `) 8V `

gi 8V HS H s$ 8V Hb

Y b m b l "

WY 8 R$ P IY 82 HS Bm g DE 8V HS PA TV RX `) 8V H1 BE H2 sh f b 3 3 b 79 8A P s v$ 82 st Y Bm W) s af qQ HS q g gi 8V P af BE T

TV I) s W) WY 8V Dl @C BE P I st P aS s WY HS BE PS BE RX W af RX ) s Y BE DE BE P q ~ P IY BE H2 BF x 8V H- st TV I) s W) WY 8V D Tb s s$ TV BE Pf q

oG DE R E " z o DF R z o V ~ C IY 8c W) s$ P e) aS s$ D Y Bm W) s af q 8b WY TV RY `) B WY RY a P IY BE H BF H~ P IY 8b W

o$ k P IY 8 af 8 8V Pd BF Pd BF RY Wc TV RY `) 8 @ BF DE D af 8 8b s$ P& P IY 8V Hd 8 `) BE BE PS HC s Wi RY `) ` WX e) gi ) 8b a R$ PS Bm gi 8V HC s WY `

8b a RX ad g gc sq vf RX af BE Pf qc x R PS Bm WY

8b WY TV 8C @G 8C P aS s WY H gi BE P @ oC Y BF Pd H 8b a Hd q gi ) R D rY s WY `c @C BE DE D RY Y P s$ Bm Wc s W~ 8b aS af RY a B o

RX aC gp RX af 8 Y BF Pd H s az 8l af 8V TV 8V BE x 8V `2 Bm W 8b ad af RX aV rY P I) s$ P BF H |

"

I H

"

5 o

z oC

7 "

l

G RX WY Hd BF `) 8b a sp P aS s WY HS BE PS BE RX W af RX ) s Y BE DE BE Pf q2 R$ & ) o & C IY 8l TV I) s W) WY 8V D Tb s s$ TV BE Pf q2 s$ HC BE x 8b WA q

8V ) e) s$ PS BE RX Wl o BE H ) ob n X e) af 8 s V C IY 8 TV RX `) 8G aS s$ PS 8 Rw P IY 8G af 8 8V Pd BF Pd BF RY Wl Pd 8V T I) WY BE ) eY 8

s$ X sw B WY Hd PC P IY 8l af 8V HS BE ` e) s$ D af RX ) s Y BF DE BE Pf q R$ 8b aS af RX a BE H `) 8b gp RX WY HS P aS sw PS 8V ` Bm W1 n X e) af 8l


32/75

0.9

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1

1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1

C a p a c

i t y

P(transition)

I(X;Y)(p)I(X;Y)(0.01)

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

0.01 0.1 1

R e s

i d

u a

l

e r r o r

r a t e

Code rate

Repetitiion codeH(0.01)

{ BE X e) af 8c ) |C s C s s$ TV BE P q Rw Y B W) s af q2 HS q) gc gi 8V P af BE T TV I) s W) WY 8V D& s$ P DE R$ @ DE R HS Hl ad s$ PS 8V Hb r ) j c

TV BE 8b WY TV q2 R$ af 8 8V PS BE PS BE RX WA TV RY `) 8 3 RX aG si P aS s WY HS BE PS BE RX W af RY ) s Y BE DF BE Pf q R$ & ) o

4 & (V U~ & ! 1 4& ( (

79 8l s ad af BE x 8l s$ P

I) s W) WY RY W HC HS 8V TV RX WY `A P IY 8V RX af 8b g r) P IY 8c ! $Y G & # $Y f D % |

{ RX aC s1 TV I) s W) WY 8V D R$ & Tb s s$ TV BE Pf q| s WY `9 s HS RX e) af TV 8i R$ & 8b W P af R q t B C 1 r

P IY 8b W1 3 RX aC s ad Y BF P aS s af BE DE q2 H g s$ DE D 'V r P IY 8b af 8~ 8V X BE HS Pd Hl si TV RX `) Bm WY HS TV IY 8b gi 8i H eY TV I P I) sw P

P IY 8 HS RX e) af TV 8 BE H af 8 af RY ` eY TV 8V `2 @C BE P IA sc af 8V HS BE ` e) s$ D" 8b aS af RX aC ad s$ PS 8 DF 8V Hd H P I) s W1 'V

I) s W) WY RX W H af RX Rw R$ G P IY BE H P IY 8V RX af 8b g BE H s W2 8V Y BE HS PS 8b WY TV 8 az RX R$ C aS s$ P IY 8b a P I) s W s gi 8b s WY H PS R

TV RX WY HS P aS eY TV Pi H eY T It TV RY `) 8V H1 B Wt P IY 8 8b WY 8b aS s$ DG Tb s$ HS 8 t ! W s af PS BE Tb eY Dm s ai P IY 8 TV IY R BE TV 8 R$ l sA RX RY `

TV RY `) 8 BE HC `) BE TV P s$ Pd 8V ` qi P IY 8 TV I) s aS sw TV PS 8b af BE HS PS BE TV HG R$ P IY 8 T I) s W) WY 8V D WY R BE HS 8 ! W s aS sw DF DE 8V D @C BE P I P IY 8

WY R BE HS 8V DE 8V HS H Tb s$ HS 8 r$ ) 8V PS PS 8b a TV RY `) 8V H& s af 8 R$ 3 PS 8b Wl s$ TV IY BE 8V x 8V `c q TV RY `) Bm WY gi eY DE PS B DE 8C Bm W eY P HS q gi R DE Hb

I " A h m c $ 3

G RX WY Hd BF `) 8b a s ad s$ P IY 8b a s af Pd B n TV Bm s$ D) TV I) s W) WY 8V D @ IY BE T Ic g sb ql ad s WY `) RX gi DE ql TV RY aS aS e P RX WY 8C Y BE P& Bm Wi 8b s$ TV I

Y DE RX TV vA R$ & HS 8V x 8b WU eY HS 8V ` PS R 8b WY TV RX `) 8 HS q gi R DE H B W P IY 8i T I) s W) WY 8V D @G 8i P s vw 8l P IY 8 az RX ) s Y BE D

BE Pf q R$ l st Y BF P1 TV RX aS ad e PS BE RX Wt 8V x 8b W Pc BE H P IY 8A H s gi 82 sw Hc TV RX aS af 8V TV P1 af 8V TV 8 PS BE RX W 79 8 Bm WD vf 8V TV P2

8V ) eY B az RX ) s Y DE 8l Bm W eY P HS q gi R DE H TV DE 8b s af DE q h

m

3 RX aG e) WY BE eY 8~ `) 8V TV RX `) Bm WY X V 7 I) s$ PG BE HC P IY 8

Tb s s$ TV BE Pf q1 R$ P IY BF H TV I) s W) WY 8V DI

79 8c I) s x 8

m

Bm W eY Pi s WY ` RX eY P eY P s$ PS PS 8b aS WY H RX a s1 BE x 8b WA Bm W eY P

X

@C BE P I9 Y Bm W) s af qA `) BF BF P

af 8 af 8V Hd 8b W P sw PS BE RX W

!

m

r) @G 8l I) s x 8 8V BE X I P 8V eY B af RX ) s Y DF 82 B3 8 @C BE P I9 o& az RX ) s

Y BE DE BF Pf q) C RY eY P eY PC HS q) gi ) R DE Hc 3 s WY `2 WY Rc R P IY 8b af H V |


33/75

!

m

!

m

!

m

!

m

m

!

m

!

m

!

m

C IY 8b W TV RX WY HS BE `) 8b af Bm WY P IY 8l Bm W RY aS gc s$ PS BE RX W1 Tb s s$ TV BE P q 8b a HS q) gi ) R D3 |

1

X

3 a a

r

W d

o

t

p

cq pu r X

DF R

o

cq pu r X

X

o

C

o

DE R

o

o

o

}C h

DE R

o

o

) ~

C IY 8 Tb s s$ TV BE Pf qh Rw l P IY 8 T I) s W) WY 8V D BE H X 9 Bm W 3 RX aS gc s$ Pd BF RY Wh Y BE PS H 8b a Y Bm W) s af qh `) BE BE PA R$ l P IY 8

TV I) s W) WY 8V D& TV RY `) Bm WY c C s W @G 8~ n WY ` s gi 8V TV I) s WY BE H g PS R 8b WY TV RX `) 8 B W 3 RX aS g s$ PS BE RX W9 Y BE PS H~ B WU

TV I) s W) WY 8V D Y BE PS H H e) s vf 8V TV PC PS Rc P IY 8 8b ad af RX a af R 8b af Pf q `) 8V HS Tb az B 8V `9 s ) R$ x 8&

C IY 8 3 $ X s g gi Bm WY G RY `) 8 af R$ xX BE `) 8V H s1 ! D 3 D %i 3 TV RY `) 8l PS R 8b a RX ad g P IY BE H s HS qY Hf

PS 8b gc sw PS BE TC TV RX `) 8 BE HG RX WY 8C Bm Wc @ IY BF TV I1 P IY 8C RY xY BE RX eY H Y Bm W) s af q 8b WY TV RX `) Bm WY c R$ P IY 8 HS RX e) af TV 8 HS q) g R DE H

BE H af 8V HS 8b W PG Bm W1 P IY 8 TV I) s W) WY 8V D 8b WY TV RY `) 8V ` 3 RX aS g " { RX a RX e) a HS RX e) az TV 8 @ IY BE Tb I1 8b gi BE PS HC s$ P 8b s$ TV I1 PS Bm gi 8

Bm W Pd 8b af xy s$ DG o~ R$ oD i HS q) g R DE Hb r @G 8 P s v$ 8 P IY 8 Y Bm W) s af q af 8 af 8V HS 8b W P s$ PS BE RX W R$ P IY BE Hl s WY `2 TV R q BE P

PS R Y BE PS Hl

r ! r s WY `-

m

Rw P IY 8~ 8b WY TV RX `) 8V ` Y DE RY T v P IY 8p az 8b gc s$ Bm WY Bm WY 2 Y BF Pd H s af 8~ BE x 8b W q

r s WY ` q % q

|

! - -Q

m

s WY ` r

- ! -Q -Q

m

- -Q

m

s WY ` r

-

-Q -Q

m

- ! -Q

m

s WY ` r

w

-

-Q ! -Q

m

W1 af 8V TV 8 PS BE RX W B P IY 8 Y Bm W) s af q WY e) gi ) 8b a

V $

P IY 8b Wi P IY 8b af 8 BE HG WY R 8b ad af RX aV r$ 8V DE HS 8l ! 7

BE H P IY 8l Y BF P Bm W 8b aS az RX aV

C IY BE H2 s gc gi Bm WY t TV RX `) 89 eY HS 8V H2 t Y BE PS H Pd R TV RX ad af 8V TV P 3

5 ob i 8b aS az RX a sw PS PS 8b aS WY H s WY `

P aS s WY Hf 8b a eY HS 8 ! eY D Y BE PS Hb ! W 8b WY 8b aS s$ D si s g gi Bm WY l TV RY `) 8 eY HS 8V H Y BE PS H& PS R~ TV RX aS af 8V TV PG o

8b aS af RY a s$ PS Pd 8b aS WY Hl s WY `t P aS s WY Hf 3 8b a o ) eY HS 8 ! eY D Y BE PS Hb 2 IY 8 s gc gi Bm WY 2 TV RY `) 8V Hc s af 8

Tb s$ DE DE 8V ` b 6V s$ H P IY 8V q2 eY HS 8 Y BE PS H PS Ri TV RX aS af 8V TV P h o 8b ad af RX af Hb


34/75

C IY 8l s gc gi Bm WY c TV RY `) 8V HC 8V Y BE HS PG 3 RX aG s$ DE D s$ Bm af H 3 t o

l

s WY ` `) 8V PS 8V TV PC RX WY 8l Y BE PC 8b aS az RX af Hb

DF Hd R P IY 8 } R Dm s q TV RY `) 82 BE H2 sh 3 ob ~ Y DE RX TV v TV RY `) 8 @ IY BE T I TV RX aS af 8V TV PS H e Pd R9 P I) af 8V 8 Y BF P

8b aS af RY af Hb r) s W e) W) W) s gi 8V `A TV RX `) 8~ 8V X BE HS Pd H s$ P 3 @ IY BE TV I TV RX aS az 8V TV PS HC e PS Rc Pf @G R Y BE P 8b aS af RX az Hb

1e-20

1e-10

1

1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1

R e s

i d

u a

l

e r r o r

r a t e

Mean error rate

{ BE X e) af 8i $ |C 3 w X G s g gi Bm WY c TV RX `) 8i af 8V HS BE ` e) s$ D 8b aS af RX aG aS sw PS 8

79 8~ Tb s W1 P IY 8b W TV RY WY HS BE `) 8b a P IY 8 gi RY af 8 8b WY 8b aS s$ D" Tb s$ HS 8 r) @ IY 8b az 8l @G 8l I) s x 8l aS s WY `) RX g Y BE P 8b ad af RX af H

e) WY B RY aS gi DE qc `) BE HS P af Bm ) eY PS 8V `9 B3 8 $ @G 8 s$ DE DE R$ @h P IY 8 R HS Hd B Y BE DE BE Pf qc R$ Pf @G R RX a gi RY af 8C Y BF P 8b aS af RX az H 8b a

l Y BE PC Y DE RX TV vY V X s$ Bm Wi @G 8 RY Y P s$ Bm Wi P IY 8 af RX ) s Y BE DE BE Pf qc R$ af 8V HS BE ` e) s$ D 8b aS af RY a& s$ H P IY 8 af 8b gc sw B WY `) 8b a

R$ P IY 8l Y Bm WY RX gi Bm s$ D HS 8b af BE 8V Hb |

m

I H

f oC { md l

d 5 D

79 8l WY R$ @ TV RX WY Hd BF `) 8b a P IY 8 Tb s$ Hd 8 B W @ IY BE T I P IY 8 HS BE X W) s$ DE H RX aC gi 8V Hd H s$ 8V HC @G 8 @C BE H I Pd Rc P aS s WY Hz 8b a

s af 8 TV RY W PS Bm WY eY RX eY HS DE q xy s af Bm s Y DE 8 P I) s$ Pi BF H ) R P It P IY 8 HS q) gi ) R DE H1 @ 81 @C BE H I PS R9 P aS s WY H gi BE Pc s af 8

TV RX W PS Bm WX eY RY eY H1 e) WY TV PS BE RX WY H r s WY `Q P IY 8- s$ Hd HS RX TV Bm s$ Pd 8V ` az RX ) s Y BE DE BF Pd BF 8V HA R$ l P IY 8V HS 8 e) WY TV Pd BF RY WY H2 s af 8

BE x 8b WA q si TV RX W PS Bm WX eY RY eY H af RX ) s Y BE DE BE Pf q2 `) BE HS P af Bm ) eY PS BE RX W

79 8C TV RY eY DF `1 P af qi s WY `c RX Y P s$ Bm Wi P IY 8C af 8V H eY DE PS HG s$ H s DE Bm gi BE PS Bm WY af RY TV 8V HS H ! af RX g P IY 8C `) BE HS Tb af 8V PS 8C Tb sw HS 8

{ RY a 8V s g DF 8 r TV RX WY Hd BF `) 8b ai sA aS s WY `) RX g xy s af Bm s Y DE 8

@ IY BF TV IU P s v$ 8V H RX W xy s$ Dm eY 8V H

X p

X

r

h

Y

o

Y

V V

r& @C BE P I az RX ) s Y BE DE BF Pf q9

XG p

{

X

r B3 8 af RY ) s Y BE DF BE Pf qh `) 8b WY Hd BF Pf qU e) WY TV Pd BF RY W

XV p

V 79 8i I) s x 8 P IY 8 s$ HS Hd RX TV Bm s$ PS 8V ` az RX ) s Y BE DE BF Pf q WY RX aS gc s$ DE B jb s$ PS BE RX W |

p

XV p

X

o

) o


35/75

HS Bm WY RX e) a RX ad g eY Dm s~ 3 RX a `) BE HS Tb af 8V PS 8~ 8b W P af R q s WY ` P s v Bm WY c P IY 8 DF Bm gi BE Pb |

W1 DE B g

&

k

p

XG p

X

DE R }

o

XG p

X

~

DE B g

&

k5

p

XG p

DE R}

o

XG p

~

X

9 DE R

X

p

X

Xs

l

X

) DE R }

o

X

D ~h

X

}~ DF Bm g

&

k

DE R

X

~

l

X

X

W1 DF Bm g

&

k

DE R

X

3

C IY BE Hl BE Hl aS sw P IY 8b a @G RX aS af qY Bm WY s$ H P IY 8 Dm s$ Pd PS 8b a DE B gp BF P~ `) RX 8V Hl WY R P 8V X BE HS Pb i Ry @G 8V x 8b a r s$ H @G 8i s af 8

R$ 3 PS 8b W Bm W PS 8b af 8V HS PS 8V `U B WU P IY 81 `) B 8b az 8b WY TV 8V Hp 8V Pf @ 8V 8b W 8b W P az R BE 8V H B 8 Bm W9 P IY 8c TV RX WY HS BE `) 8b aS s$ Pd BF RY W

R$ gi eY P e) s$ D" 8b W P az R q RY aC Tb s sw TV BF Pf q) V r @ 8~ `) 8 n WY 8c P IY 8 af RX Y DE 8b g s @ s q q eY HS Bm WY P IY 8 n az HS P

PS 8b aS g RX WY DE q s$ H sc gi 8b s$ H e) az 8 R$ f & 3 & 3 f d $ |

W1

l

X

) DE R }

o

X

~b

X

3

79 8i Tb s W 8V Y PS 8b WY ` P IY BE Hl Pd R2 sc TV RX W PS Bm WY eY RX eY H aS s WY `) RX g x 8V TV PS RX a R$ G `) B gp 8b WY HS BE RX W @U TV RX WY TV 8b s$ DE Bm WY

P IY 8 @ 3 3 R DE `2 Bm W Pd 8V X aS s$ D 8b IY Bm WY `- x 8V TV PS RX aC WY R P s$ PS BE RX W1 s WY `- R DE `2 Pf q 8 |

l

) DE R

}

o

~

3 $ X

79 8~ Tb s W1 P IY 8b W2 sw DF Hd Rc `) 8 n WY 8i P IY 8 v R B W P s WY ` TV RX WY `) BE PS BE RX W) s$ D 8b W P af R BE 8V HG 3 RX aG TV RY W PS Bm WY eY RX eY H `) BF Hz

P af Bm ) eY PS BE RX WY H |

{

Q

) DE R}

o

q ~

r

Q

) DE R }

q ~h

r

) DE R}

q ~

@C BE P I |

X

h

c

h

{ Bm W) s$ DE DE q r gi eY P e) s$ D) Bm W RY aS gc s$ PS BE RX Wp R$ TV RX W Pd B WY eY RX eY HG aS s WY `) RX g xy s af Bm s Y DE 8V H

s WY `h f BE HG `) 8 n WY 8V `

eY HS Bm WY 1 P IY 8 `) RX e) Y DE 8i B W PS 8V X ad s$ D3 |

{ f

X d c

DE R

}

X rt c

X

~

X

c

@C BE P I P IY 8 Tb s s$ TV BE Pf q R$ P IY 8 T I) s W) WY 8V D BE x 8b WA q gc s$ Y Bm gi B jV Bm WY P IY BE H gi eY P e) s$ D" B W 3 RX aS g s$ PS BE RX W

R$ x 8b aG s$ DE D R HS HS Bm Y DE 8 Bm W eY P `) BE HS P az B ) eY Pd BF RY WY HC 3 RX a


36/75

n c oC ( w w ( #

79 8c RY Y P s$ Bm W xy s af BE RX eY H af R 8b af PS BE 8V Hc s W) s$ Dm s$ RX eY H PS R P IY 8c `) BE HS Tb az 8V PS 8 Tb sw HS 8 W P IY 8i 3 R DE DE Ry @ B WY

@G 8i ` af R P IY 8c R DE ` Pf q 8 r ) eY P 8b sw T I `) BE HS P af Bm ) eY PS BE RX W r xy s af Bm s Y DE 81 8V PS Tc H IY RY eY DF `t ) 81 P s v$ 8b WA PS R

) 8~ R$ @2 `) Bm gi 8b WY Hd BF RY WY Hb

o C j& W P af R q9 BE H g s$ X Bm gi B jV 8V `Q @C BE P I 8V ) eY B af RX ) s Y DE 8 S HS q) gi ) R DE H

X

BE H DE Bm gi BE PS 8V ` PS R

HS RX gi 8 x R Dm e) gi 8 B3 8 ) BE HC RX WY DE q WY RX W 9 jV 8b af R1 @C BE P IY Bm W P IY 8 x R Dm e) gi 8b G P IY 8b W1

X

BE H gc s$

B gp B jV 8V ` @ IY 8b Wi

X

5 o

X" Y c

& B

X

" )

c

G 7 I) sw P BE H2 P IY 8 g s$ X Bm gi e) g `) B 8b af 8b W Pd B sw D 8b W P az R q RY ac H 8V TV B n 8V ` x$ s af Bm s WY TV 8 h @G 8

T IY RY R HS 8c P IY BF Hc s$ H~ P IY 81 xy s af Bm s WY TV 81 BE Hc s2 gi 8b s$ H e) af 8 Rw s x 8b ad s$ 8 R$ @G 8b aV C 8b WY TV 82 sA af 8

HS P s$ PS 8b gp 8b W PG R$ P IY 8 af RX Y DE 8b g BE H PS R~ n WY `2 P IY 8l g s$ X Bm gi e) g `) B 8b af 8b W PS Bm s$ D 8b W P af R qc 3 RX a

si H 8V TV B n 8V `9 gi 8b s W R$ @ 8b a

G RX WY HS BE `) 8b aG P IY 81 o ! `) Bm gi 8b WY HS BE RX W) s$ D aS s WY `) RY g xy s az B s Y DF 8

rX @ BF P I P IY 8 TV RX WY Hd P aS s$ Bm W PS Hb |

Q

X

X

o

X

X

X

@ IY 8b af 8 |

X

X

X

3

C IY BE H R PS Bm gi B jb s$ PS BE RX W af RX Y DE 8b g BE HG HS R DE x 8V `2 eY HS Bm WY A s$ X ad s WY 8 gi eY DE PS B DE BF 8b az H s WY ` gc s$

B gp B jV Bm WY |

b

X

DF R

X

" E

X

b

X

E

X

7

X

@ IY BE T I BF H RX Y P s$ Bm WY 8V `- q1 HS R DE xX Bm WY |

oC 9 DE R

X

" E

X

HS Ri P I) s$ PC @C BE P I1 ` eY 8i af 8V X s az `1 RX a P IY 8 TV RX WY HS P ad s$ Bm W PS H RX W1 P IY 8 HS qY HS PS 8b g |

X

o

l lG

Y

7

IY 8b WY TV 8 |

o

DF R 3

3

79 8l RX Y Hd 8b af x 8l P I) s$ Pb |G Bm G 3 RX a s W q2 aS s WY `) RX g xy s az B s Y DF 8 f @ BF P I2 x$ s af Bm s WY TV 8A r fi

V r BE B ~ P IY 8 `) B 8b af 8b W PS Bm s$ D 8b W P az R qA BE Hl ) 8 8b WY `) 8b W Pi RX WY DE q9 RX WA P IY 8c x$ s af Bm s WY TV 8 s WY `

BF H Bm WY `) 8 8b WY `) 8b W Pl R$ & P IY 8 gi 8b s W r IY 8b WY TV 8 BE BE Bm P IY 8~ X af 8b s$ Pd 8b aG P IY 8 R$ @ 8b a r) P IY 8l Y af 8b s$ PS 8b a

P IY 8 `) B 8b az 8b W PS Bm s$ D" 8b W P az R q

C IY BE HC 8V X Pd 8b WY `) H PS R1 g eY DE PS B DE 8i ) Bm gi 8b WY HS BE RX WY H B WA P IY 8 RY xY BE RX eY H gc s W) WY 8b a


37/75

n i & #$ ( B U ( #

! W P IY 8C Tb sw HS 8 Rw TV RY W PS Bm WY eY RX eY H& Hd BF Y W) s$ DE H s WY ` TV I) s W) WY 8V DE HG @ 8 s af 8 B W PS 8b af 8V Hd PS 8V `c Bm W e) WY TV Pd BF RY WY H R$

H s qi PS Bm gi 8b V r T IY R Hd 8b Wi ! af RX g5 s HS 8V P& R$ R HS HS Bm Y DE 8 e) WY TV Pd BF RY WY Hb r) s$ H& Bm W eY PG PS Rl RY e) a TV I) s W) WY 8V D3 r) @C BE P I

HS RX gp 8 8b af P e) aS 8V ` x 8b af HS BE RX W R$ G P IY 8i ! e) WY TV PS BE RX WY Hc ) 8V Bm WY 2 `) 8V Pd 8V TV PS 8V ` sw P P IY 8c RX eY P eY P c C IY 8V HS 8

! e) WY TV PS BE RX WY HC P IY 8b W1 P s v$ 8 P IY 8 Dm s$ TV 8~ R$ P IY 8 Bm W eY P s WY ` RY eY P eY PC s$ D I) s ) 8V PS H R$ P IY 8 `) BE HS Tb af 8V Pd 8

Tb s$ HS 8

79 8 gi eY HS P s$ DE HS R P IY 8b WU Bm WY TV Dm eY `) 82 P IY 8 af RX ) s Y BE DE BE Pf q R$ sA BE x 8b W e) WY TV Pd BF RY W ) 8V Bm WY 9 Bm WD v 8V TV Pd 8V `

Bm W Pd R P IY 8 T I) s W) WY 8V D" @ IY BE T I ) af Bm WY H eY H Pd R P IY 8 BF `) 8b sp R$ & D %A ! E ! c P IY BE HC 8b WY 8b aS sw D s af 8b s R$

@G RX aS vi BE H vX WY R$ @ W sw H %i q ! Y i $X d $

{ RY aG 8V s g DE 8 r) TV RX WY HS BE `) 8b a P IY 8l 8b WY HS 8b gi Y DE 8V Hb |

o q

Hd B W ) $

j& s$ TV I xy s$ Dm eY 8 R$ `) 8 n WY 8V H& sG `) B 8b az 8b W P ! e) WY TV PS BE RX W r s WY ` Pd R 8V P IY 8b a @ BF P Il s af RY ) s Y BE DF BE Pf q

) BF Hd P af Bm ) eY PS BE RX W r H s q $ V r @G 8 I) s x 8 s W 8b WY HS 8b gi Y DE 8 9 C R PS 8c P I) s$ P 2 IY 8b af 8 gc s q9 ) 8

) BF Hd Tb af 8V PS 8l RY a& TV RY W PS Bm WY eY RX eY H TV RX WY HS BE `) 8b a I) s$ HS 8l H IY B 3 P v$ 8V qY Bm WY

C G RX WY HS BE `) 8b a sU HS 8V P R$ aS s WY `) RY g xy s af Bm s Y DE 8V H f |Y

d

o

Y

b b V

k9 @ IY 8b af 8 8b s$ TV I

P s v$ 8V Hc RX W st aS s WY `) RX g xy s$ Dm eY 89 s$ TV TV RX af `) Bm WY PS Rt s9 }l s eY Hd HS Bm s W `) BE HS P az B ) eY Pd BF RY W @C BE P I

HS P s WY ` s az ` `) 8V xY Bm s$ PS BE RX W

9 P IY 8b W |

! k Hd B WY T) 3 T

BF HG P IY 8 S @ IY BE PS 8 WY R BE HS 8! i 8b WY HS 8b gi Y DF 8 r ) s WY `Y DE Bm gi BE PS 8V ` PS R 8b af P j s WY `1 @C BE P I1 s x 8b aS s$ 8

R$ @G 8b a

G u- RY af 8 8b WY 8b aS s$ DE DE q r @G 8l I) s x 8 3 RX aC aS s WY `) RX g x$ s af Bm s Y DE 8V H

Q X

! kl P IY 8l 8b WY HS 8b gi Y DE 8i R$ ) s WY `Y

DF Bm gi BE PS 8V `A ! e) WY TV PS BE RX WY Hb |

Q X

k

X

HS Bm WY T) 3 T !

@ IY 8b af 8 R$ TV RX e) af HS 8 @G 8l af 8b gp 8b g 8b a ! af RX g P IY 8 H s g DE Bm WY P IY 8V RY af 8b g P I) s$ P |

X

}

~

c @G 8 s$ DE HS Rh TV RY WY HS BE `) 8b aA e) WY TV Pd BF RY WY H- DE Bm gi BE PS 8V `5 PS R PS Bm gi 8U B W PS 8b af x$ s$ D l r~ P IY 8b W @G 89 RY

P s$ Bm Wt RX WY DE q ( WY RY W 9 jV 8b af R9 TV RY 8 TV BE 8b W PS H1 s WY `h @G 81 Tb s Wt TV RX WY HS BE `) 8b a P IY 8 8b WY HS 8b gi Y DE 8

PS Rc ) 8 az 8 af 8V HS 8b W PS 8V `- q s W1 @) ! `) Bm gi 8b WY Hd BF RY W) s$ D @1 q ( af RX ) s Y BF DE BE Pf q `) BE HS P af Bm ) eY PS BE RX W

X V Y X b b S Y X

G u- RY af 8c H 8V TV B n Tb sw DF DE qY rC B @G 8 TV RX WY HS BE `) 8b ai P IY 81 8b WY HS 8b gi Y DE 82 R$ DF Bm gi BE PS 8V ` R$ @G 8b a 3 q V r

) s WY `Y DE Bm gi BE PS 8V `h PS R

s WY ` Pd B gp 8 & DF Bm gi BE PS 8V `A HS BE X W) s$ DE H 3 WY RY W 9 jV 8b af Rc RX WY DE q1 B W1 Bm W PS 8b af x$ s$ D

3 S V r @ 8A n WY ` P I) s$ P P IY 8- 8b WY Hd 8b g Y DE 89 BE H af 8 af 8V HS 8b W PS 8V ` q s W @ ! `) B gp 8b WY HS BE RX W) s$ D

af RX ) s Y BE DE BE P qQ `) BE HS P af Bm ) eY PS BE RX W @ IY BE T IQ BE H1 jV 8b af R RX eY Pd HS BE `) 8- P IY 8 @ ! H IY 8b az 8 ad s$ `) Bm eY H

$


38/75

q TV RY WY HS BE `) 8b af Bm WY P IY 81 Dm s$ PS PS 8b a~ Pf q 8V Hc R$ 8b WY HS 8b gi Y DF 8V H r& @G 8 Tb s W n Pc P IY 8b g Bm W Pd R- P IY 81 n WY BF Pd 8

) Bm gi 8b WY HS BE RX W) s$ D TV RX W PS Bm WX eY RY eY H `) B 8b af 8b W PS Bm s$ D 8b W P af R q `) 8 n WY BE PS BE RX WY H BF x 8b W Bm W HS 8V TV Pd BF RY W )

n 4 & (V U~ o ') 3 V 6

79 8C TV RY WY HS BE `) 8b aG T I) s W) WY 8V DF HG Bm W1 @ IY BE TV I WY R BE HS 8 BE HG B WD vf 8V TV PS 8V ` Bm WY `) 8 8b WY `) 8b W PS DE q R$ P IY 8C Hd BF Y W) s$ D3 ) P IY 8

s af Pd BF Tb eY Dm s a WY R BE HS 8C HS RY e) af TV 8C R$ B W PS 8b af 8V Hd P& BE H& P IY 8 HS R$ Tb s$ DE DE 8V `- 3 b 5 $Y 3 C d V b I V b

{ e) az P IY 8b aG @G 8l af 8V HS P az BF TV P TV RX WY HS BE `) 8b aS s$ PS BE RX WY H PS Rc P IY 8 n W) s$ D TV Dm s$ HS H R$ 8b WY HS 8b gi Y DE 8

79 89 I) s x 8- sU HS BE X W) s$ D @ BF P I s x 8b ad s$ 8 R$ @G 8b a rC Pd B gp 8 p DE Bm gi BE PS 8V ` Pd R) s WY ` ) s WY `) @ BF `) P I

DE Bm gi BE PS 8V ` PS Rh

79 8A P IY 8b WQ TV RX WY HS BE `) 8b a1 P IY 8 @5 H s g DE 8V H

h p

s WY `E f

p

i P I) sw P Tb s WQ ) 89 eY HS 8V ` PS R

TV I) s aS s$ TV PS 8b az BF Hd 8l ) R P I1 P IY 8 B W eY P s WY ` RX eY P eY P Hd BF Y W) s$ DE Hb IY 8b W1 P IY 8l af 8V Dm s$ PS BE RX WY H IY B ) 8V Pf @G 8V 8b W

P IY 8 Bm W eY P s WY `2 RX eY P eY PC BE H BE x 8b W q |

f

p h p

p

rG c 5 o

b V

@

@ IY 8b af 8c

p

BE H ! af RX g P IY 8c ) s WY `Y C DF Bm gi BE PS 8V ` }l s eY HS HS Bm s WA `) BE HS P af Bm ) eY PS BE RX W @ BF P I| jV 8b az R2 gi 8b s W s WY `

xy s af Bm s WY TV 8 |

k

@ IY 8b af 8l

k

BE H P IY 8 R$ @G 8b aG H 8V TV P aS s$ D `) 8b WY HS BE Pf q

Hp BE Hc Bm WY `) 8 8b WY `) 8b W P R$

@G 81 RX Y P s$ Bm Wt P IY 8 TV RY WY `) BF Pd BF RY W) s$ DC 8b W P af R q9 s$ Hc ) 8V Bm WY Hd R DE 8V DF q

) 8 8b WY `) 8b W P RX W1 P IY 8l WY R BF Hd 8l HS RX e) az TV 8 r s WY `1 ! af RX g 8V ) e) s$ PS BE RX W n WY `2 BE PS HC x$ s$ Dm eY 8 |

f

r

3

o

DE R

k

8b WY TV 8 |

{ f fc T -

C IY 8 Tb s sw TV BF Pf q1 R$ P IY 8 TV I) s W) WY 8V D BE HC P IY 8b W1 RX Y P s$ Bm WY 8V `2 q gc sw X Bm gi B jV Bm WY P IY BF H Ry x 8b a sw DF D" B W eY P

) BE HS P af Bm ) eY PS BE RX WY H~ 2 P IY BE Hi BE H s$ TV IY BE 8V x 8V ` q- gc sw X Bm gi B jV Bm WY 9 @C BE P I9 az 8V H 8V TV P~ PS R P IY 8i ) BE HS P af Bm ) eY PS BE RX W

R$

H e) s vf 8V TV PC PS R P IY 8l s x 8b aS s$ 8 R$ @G 8b aG DE Bm gi BE P s$ PS BE RX W |

w

p

x

H @G 8 I) s x 8 Hd 8V 8b W2 P IY BF H BE H sw T IY BE 8V x 8V ` @ IY 8b WA @G 8l I) s x 8 sc }~ s eY HS HS Bm s W `) BE HS P af Bm ) eY PS BE RX W r IY 8b WY TV 8

@G 8 n WY `9 ) R P I

s WY ` f gi eY HS P I) s x 8 }~ s eY HS HS Bm s W `) BE HS P af Bm ) eY PS BE RX WY Hb W s af PS BE Tb eY Dm s a

I) s$ H

xy s af Bm s WY TV 8 s WY `1 f I) s$ HC xy s af Bm s WY TV 8

k

79 8 Tb s W P IY 8b W1 8V xy sw D e) sw PS 8l P IY 8 TV I) s W) WY 8V D Tb s s$ TV BE Pf q r s$ Hb |

fc

o

DE R G

k

3

o

DE R G

3

k

o

DE R b } o&

k

~

3


39/75

C IY BE HG Tb s s$ TV BE Pf qc BE H& Bm W Y BE PS H 8b a T I) s W) WY 8V D HS q) gi ) R D3 rY HS R~ @ 8 Tb s Wi RX Y P sw B W1 sl ad s$ PS 8 8b a& Hd 8V TV RX WY ` r

q gi eY DE PS B DE BF Tb sw PS BE RX W9 q @ q l r B 8 Y az RX g @ l r gi eY DE PS B DF BE Tb s$ Pd BF RY W- q2 |

DE R

}

o

k

~

Y BE PY $ H

R

I) s W) WY RY W H P IY B az ` P IY 8V RX af 8b g r P IY 8A ! & f ! $X G D d ) b 1 $X D %i |

C IY 8c Tb s s$ TV BE Pf q9 R$ s T I) s W) WY 8V DC ) s WY `) DE Bm gi BE PS 8V `Q PS R 8b af P j r 8b af P e) aS 8V ` q

s$ `) `) BE PS BE x 8G @ IY BE PS 8 }l s eY HS HS Bm s Wl WY R BF Hd 8C R$ R$ @G 8b a" H 8V TV P ad s$ D) `) 8b WY HS BE Pf q

k

s WY `c ) s WY `Y

DF Bm gi BE PS 8V ` PS RA BE H BE x 8b W q |

DE R } o

k

~

Y BE PY $ H

@ IY 8b af 8 BE HC P IY 8 s x 8b aS s$ 8 P ad s WY H gi BE PS PS 8V ` R$ @ 8b a

m " V

C IY 8G HS 8V TV RX WY `i PS 8b ad g @C BE P IY Bm Wi P IY 8 DE R Bm W~ 8V ) e) s$ PS BE RX Wi BF H P IY 8 Hd BF Y W) s$ D) PS R WY R BE HS 8C aS s$ PS BE Rc

( V

o C Y HS 8b af x 8c P I) s$ Pl P IY 8 Tb s s$ TV BE Pf q9 Bm WY Tb af 8b s$ HS 8V Hc gi RX WY R PS RX WY BE Tb s$ DE DE q s WY `t @C BE P IY RX eY P RX eY WY `

s$ HG P IY 8

Bm WY Tb af 8b sw HS 8V Hb

B gp BF Dm s az DF q P IY 8 Tb s sw TV BF Pf q1 Bm WY Tb af 8b s$ HS 8V H gp RX WY R PS RX WY BE Tb s$ DE DE q s$ HG P IY 8l ) s WY `) @C BE `) P I1 Bm WY Tb af 8b s$ Hd 8V H

) eY PC PS R s~ DE Bm gi BE Pb C C Hd B WY sb qY DE RX a! HG 8V s WY HS BE RX W RX a D W" |

D W f o B

CD C& C

@ 8 RY Y P s$ Bm W |

k

DE R

C C IY BE HC BE H R$ 3 PS 8b W1 af 8V @ af BE PS Pd 8b W1 B W PS 8b aS gi HG R$ C & b #u y ) b V 3 r ( r) @ IY BF TV I1 BE H `) 8 n WY 8V ` q

C IY 8 DE B gp BF Pd B WY A xy s$ Dm eY 8~ BF H P IY 8b W |

k

DF R l

C IY BE HC BE H Tb s$ DE DE 8V ` P IY 8

I) s W) WY RX W Bm gi BE Pb

C C IY 8& Tb s sw TV BF Pf q~ R$ P IY 8G T I) s W) WY 8V D) BF H& sw T IY BE 8V x 8V `i @ IY 8b W~ P IY 8G HS RX e) az TV 8

d DF RY RX v H DE Bm v$ 8 WY R BF Hd 8!

C IY BE Hc BE H P IY 82 ) sw HS BE Hi RY ai H af 8b s$ ` H 8V TV P aS e) g PS 8V T I) WY BE ) eY 8V H Rw gi RY ` eY Dm s$ PS BE RX WQ s WY ` Bm W

s af PS BE Tb eY Dm s aC G RY `) 8l BE xY BE HS BE RX WA u eY DE PS B DE 8 TV TV 8V Hd H 3 G u V


40/75


41/75

Introduction to Fourier Analysis, Synthesis, and Transforms

It has been said that the most remarkable and far-reaching relationship in all of mathemat-ics is the simple Euler Relation,

ei + 1 = 0 (1)which contains the ve most important mathematical constants, as well as harmonic analysis.

This simple equation unies the four main branches of mathematics: {0,1}represent arithmetic, represents geometry, i represents algebra, and e = 2 .718... represents analysis, since one wayto dene e is to compute the limit of (1 + 1n )

n as n .Fourier analysis is about the representation of functions (or of data, signals, systems, ...) interms of such complex exponentials. (Almost) any function f (x) can be represented perfectly asa linear combination of basis functions:

f (x) =k

ckk(x) (2)

where many possible choices are available for the expansion basis functions k(x). In the caseof Fourier expansions in one dimension, the basis functions are the complex exponentials:

k(x) = exp( i kx) (3)where the complex constant i = 1. A complex exponential contains both a real part and animaginary part, both of which are simple (real-valued) harmonic functions:

exp( i ) = cos( ) + i sin() (4)which you can easily conrm by using the power-series denitions for the transcendental functionsexp, cos, and sin:

exp() = 1 +1!

+2

2!+

3

3!+ +

n

n !+ , (5)

cos() = 1 2

2!+

4

4! 6

6!+ , (6)

sin() = 3

3!+

5

5! 7

7!+ , (7)

Fourier Analysis computes the complex coefficients ck that yield an expansion of some functionf (x) in terms of complex exponentials:

f (x) =n

k= n

ck exp( i kx) (8)

where the parameter k corresponds to frequency and n species the number of terms (whichmay be nite or innite) used in the expansion.

Each Fourier coefficient ck in f (x) is computed as the (inner product) projection of the functionf (x) onto one complex exponential exp( i kx) associated with that coefficient:

ck =1T

+ T/ 2

T/ 2f (x)exp(i kx)dx (9)

where the integral is taken over one period ( T ) of the function if it is periodic, or from to+

if it is aperiodic. (An aperiodic function is regarded as a periodic one whose period is

).

For periodic functions the frequencies k used are just all multiples of the repetition frequency;for aperiodic functions, all frequencies must be used. Note that the computed Fourier coefficientsck are complex-valued.

38


42/75

! # "$ ! & %' ( 0 )1

2 4 36 58 7@ 9B AD C' E4 EF C' GI HD PQ GR HT S8 CU AW VD C' EX 9B 38 AD 9` Y ab SD 58 YU c@ 9B 38 58 7e d fh gp iU qR 9h rs Ct ru ah 36 Eu 7 36 vw Cy x Gt 58 A g qD d fh g4 W x8 i

d fh g i r0 9` c 8 36 S8 c( HB 3 7 7y 3I a t C' 58 C' E@ GR H` 9B cF 4 C c Gt R C! x t r

C 36 8 7@ C' EF Pt C c 8 Cy 38 EF c 8 3t t 38 5D GR H` 9B cF VD EX 36 VD C' EF c@ 9B CU 7( 3R a0 7 9T 50 fb $ g i4 Gt 58 A$ YU 3t 7Q fb gp i aj 36 E 9T 5t c CU C' EF 7y

Gt 58 AW u k

ln mX o

YU 3t 7Q fh & g i YU 3t 7' fh $ g i@ 6 g

9 a $

z | { 3t c 8 C' EX } 9` 7 C

l

mX o

7 9T 5 fh & g i 7 9T 5 fh $ g i@ 6 g

9 a $

| {3t c 8 C' EX } 9` 7 C

l

mX o

7 9T 50 fh & g i YU 3t 7' fh $ g i@ 6 g ~ fF ' i

C! S8 7 C c 8 C! EF 36 58 CU YU R C' E} a SD 58 YU c 9` 36 5$ c@ 3 v C' Gt 5& k

z | {

3t c 8 C' EX } 9` 7 C

8 C' 5 c 8 C 36 SD EX 9B C' E( 8 C' EF 9` CU 7 aj 36 E4 d fh gp i4 9` 7' k

x

{Q

fb xt { YU 3t 7Q fh & g i z {| 7 9T 50 fb gp i i fh t i

y 8 C' EF C cz 8 C! 38 SD EF 9` C' E} 2 4 38 Cz YU 9` C' 5t c 7 Gt EF Ct k

x {

l

mF o

d fh g i YU 3t 7' fh & g i 8 g fh t i

z {

lW mX o

d fh g i 7 9T 5 fh & g i 8 g f 6 i

C 8 36 VD C c D GR c} c 8 C p 36 SD EF 9` C' E 8 C' EF 9` CU 7! VD EF 3R P6 9` AD CU 7 Gt 5 CU VD G 58 7 9` 36 5 3R a cz 8 C a SD 58 YU c 9` 36 5W d fh gp i4 9T 5

c C' E vw 7} 3R a0 YU 3t 7@ 9 58 C Gt 58 AW 7@ 9 58 C c C' E v 7 r

| t D | p t - & W R | & t -p

CU cy

fh gp i4 D C G 5t 7 SD v 3R a 7 9T 58 C G 58 AW YU 3t 7 9T 58 C c C' E v 7' k

fh g i

x

{R

fh x

{

YU 3t 7Q fh & gp i

{

7 9T 5u fh & g i i fh t i

Gt 58 AW

fh gp iU q8 c 8 C c E SD 58 Y' GR c@ CU A$ 36 SD EF 9` C' E 8 C' EF 9` CU 7} ah 36 E} G ab SD 58 YU c 9` 36 5$ d fh gp iU k

fh g i

x

{R

fh xt { YU 3t 7Q fh & gp i z {| 7 9T 5u fh & g i i

t

High Lit Ed Errs Notes

Documents

Transcript of High Lit Ed Errs Notes