Exam #1 Reviewbazuinb/ECE3800/Exam1_Review.pdf1.3 Venn Diagrams 1.4 Random Variables 1.5 Basic...
Transcript of Exam #1 Reviewbazuinb/ECE3800/Exam1_Review.pdf1.3 Venn Diagrams 1.4 Random Variables 1.5 Basic...
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 1 of 32 ECE 3800
Exam #1 Review
What is on an exam? Read through the homework and class examples…
5 multipart questions. Points assigned based on complexity. (170 pts. Sp. 2019, 175 pts. Fa 2019)
Set theory based probability (switches - union, intersection, mutually exclusive events) Elementary probability determination (coin or coins flipped, die or dice rolled, etc.) Discrete probability pmf and CDF Expected Value Operator Moments, Variance, Moment Generating Function Selection or Failure related questions (combinatorial based questions)
Previous homework problem solutions as examples – Dr. Severance’s Skill Examples
Skills #1
Skills #2
Exam and homework like problems:
HW#1
HW#2
HW#3
HW#4
Read through previous exam ….
And now for a quick chapter review … the highlights …!
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 2 of 32 ECE 3800
The Chapter Content: 1-4
1 PROBABILITY BASICS 1.1 What Is Probability? 1.2 Experiments, Outcomes, and Events 1.3 Venn Diagrams 1.4 Random Variables 1.5 Basic Probability Rules 1.6 Probability Formalized 1.7 Simple Theorems 1.8 Compound Experiments 1.9 Independence 1.10 Example: Can S Communicate With D?
1.10.1 List All Outcomes 1.10.2 Probability of a Union 1.10.3 Probability of the Complement
1.11 Example: Now Can S Communicate With D? 1.11.1 A Big Table 1.11.2 Break Into Pieces 1.11.3 Probability of the Complement
1.12 Computational Procedures
2 CONDITIONAL PROBABILITY 2.1 Definitions of Conditional Probability 2.2 Law of Total Probability and Bayes Theorem 2.3 Example: Urn Models 2.4 Example: A Binary Channel 2.5 Example: Drug Testing 2.6 Example: A Diamond Network
3 A LITTLE COMBINATORICS 3.1 Basics of Counting 3.2 Notes on Computation 3.3 Combinations and the Binomial Coefficients 3.4 The Binomial Theorem 3.5 Multinomial Coefficient and Theorem 3.6 The Birthday Paradox and Message Authentication 3.7 Hypergeometric Probabilities and Card Games
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 3 of 32 ECE 3800
4 DISCRETE PROBABILITIES AND RANDOM VARIABLES 4.1 Discrete Random Variable and Probability Mass Functions 4.2 Cumulative Distribution Functions 4.3 Expected Values 4.4 Moment Generating Functions 4.5 Several Important Discrete PMFs
4.5.1 Uniform PMF 4.5.2 Geometric PMF 4.5.3 The Poisson Distribution
4.6 Gambling and Financial Decision Making3.4 The Binomial Theorem
Chapter 1: PROBABILITY BASICS
An understanding of Probability and Statistics is necessary in most if not all work related to science and engineering.
Statistics: the study of and the dealing with data.
Probability: the study of the likeliness of result, action or event occurring. Often based on prior knowledge or the statistics of similar or past events!
Definitions of Probability Experiment Possible Outcomes Trials Event Equally Likely Events/Outcomes Objects Attribute Sample Space With Replacement and Without Replacement
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 4 of 32 ECE 3800
Probability as the Ratio of Favorable to Total Outcomes (Classical Theory)
N
NAr A
ArAN
limPr
Where APr is defined as the probability of event A.
1. 1Pr0 A 2. Pr(A)+Pr(B)+Pr(C)+ … =1, for mutually exclusive events 3. An impossible event, A, can be represented as 0Pr A .
4. A certain event, A, can be represented as 1Pr A .
Sets, Fields and Events Set Subset Space Null Set or Empty Set Venn Diagram Equality
ABandBAiffBA Sum or Union
ABBA
AAA
AA
SSA
ABifABA , Products or Intersection
ABBA
AAA
A
ASA
ABifBBA , Mutually Exclusive or Disjoint Sets
BA Complement
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 5 of 32 ECE 3800
AA and SAA
S
S
AA
ABifBA ,
ABifBA , Differences
BAABABA
BBBA
AAA
AAAA
AA
SA
AAS Proofs of Set Algebra
DeMorgan’s Law
BABA
BABA
Union of two sets
GFGFGF PrPrPrPr
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 6 of 32 ECE 3800
Equalities in Set Algebra
from: Robert M. Gray and Lee D. Davisson, An Introduction to Statistical Signal Processing, Cambridge University Press, 2004. Appendix A, Set Theory. Pdf file version found at http://www-ee.stanford.edu/~gray/sp.html
The Axiomatic Approach
For event A
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 7 of 32 ECE 3800
1Pr0 A (Axiom 1 & 2, Theorem 1.3)
1Pr S (Axiom 2)
0Pr (Theroem 1.1)
Disjoint Sets
BABAthenBAIf PrPrPr,
Complement (complementary sets)
1PrPrPrPr, SAAAAthenAAIf
1Pr1Pr AA
Not a Disjoint Sets (solution)
BABABA PrPrPrPr
in general (three elements)
1
1
2
1
3
1
2
1
3
1
3
1321
Pr
Pr
PrPr
i ij jk kji
i ij ji
i i
EEE
EE
EEEE
Can you recognize a pattern …
“+”singles … “-“doubles … “+”triples … “-“quads … etc
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 8 of 32 ECE 3800
Joint, Conditional, and Total Probabilities; Independence
BBABA Pr|PrPr , for 0Pr B
B
BABA
Pr
Pr|Pr
, for 0Pr B
Joint Probability
𝑃𝑟 𝐴|𝐵 𝑃𝑟 𝐴 when A follows B
Marginal Probabilities: Total Probability (for Ai disjoint sets)
nAAAAS 321
jiforAA ji ,
iii AABAB Pr|PrPr , for 0Pr iA
nn ABABABABAAAABSBB 321321
nn AABAABAABB Pr|PrPr|PrPr|PrPr 2211
Independence BAABBA PrPr,Pr,Pr
or BABA PrPrPr
Independence can be extended to more than two events, for example three, A, B, and C. The conditions for independence of three events is
BABA PrPrPr CBCB PrPrPr CACA PrPrPr
CBACBA PrPrPrPr
For multiple events, every set of events from n down must be verified. This implies that 12 nn equations must be verified for n independent events.
Bayes Theorem
B
AABBA ii
i Pr
Pr|Pr|Pr
, for 0Pr B
nn
iii AABAABAAB
AABBA
Pr|PrPr|PrPr|Pr
Pr|Pr|Pr
2211
BA,Pr
AABBBAABBA Pr|PrPr|Pr,Pr,Pr
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 9 of 32 ECE 3800
Can S Communicate with D? (Switch problems)
Figure 1.4: Two-path Network
213 LLLDSComm , for all links independent!
32Pr pppDS
213213
213213
213
PrPrPrPrPrPr
PrPrPrPr
PrPr
LLLLLL
LLLLLL
LLLDS
32
213213 PrPrPrPrPrPrPr
ppp
pppppp
LLLLLLDS
Applying negative logic
DsDS Pr1Pr
213PrPr LLLDS
21213213 PrPrPrPrPrPrPrPr LLLLLLLLDS
2213 1111PrPrPr ppppLLLDS
2213 21221PrPrPr ppppLLLDS
322213 111PrPrPr pppppLLLDS
and
323211Pr1Pr ppppppDSDs
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 10 of 32 ECE 3800
CBADSComm
CBACBCABACBA
CBACBCABACBA
CBADS
PrPrPrPrPrPrPrPrPrPrPrPr
PrPrPrPrPrPrPr
PrPr
Alternately consider
CBA
CBADS
PrPrPr
PrPr
Now consider unequal probabilities
Remember the initial derivation?
213 LLLDSComm , for all links independent!
213213
213213
213
PrPrPrPrPrPr
PrPrPrPr
PrPr
LLLLLL
LLLLLL
LLLDS
And finally,
213213
213213 PrPrPrPrPrPrPr
pppppp
LLLLLLDS
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 11 of 32 ECE 3800
Experiment 1: A bag of marbles, draw 1
Experiment 2: A bag of marbles, draw 2
Experiment 3: A bag of marbles, draw 2 without replacement
Can you compute the probabilities related to bag of marbles questions?
Chapter 2: CONDITIONAL PROBABILITY
Defining the conditional probability of event A given that event B has occurred.
Using a Venn diagram, we know that B has occurred … then the probability that A has occurred given B must relate to the area of the intersection of A and B …
BBABA Pr|PrPr , for 0Pr B
Therefore
B
BABA
Pr
Pr|Pr
, for 0Pr B
For elementary events,
B
BA
B
BABA
Pr
,Pr
Pr
Pr|Pr
, for 0Pr B
Special cases for BA , AB , and AB .
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 12 of 32 ECE 3800
Special cases for BA , AB , and AB .
If A is a subset of B, then the conditional probability must be
B
A
B
BABA
Pr
Pr
Pr
Pr|Pr
, for BA
Therefore, it can be said that
AB
A
B
BABA Pr
Pr
Pr
Pr
Pr|Pr
, for BA
If B is a subset of A, then the conditional probability becomes
1
Pr
Pr
Pr
Pr|Pr
B
B
B
BABA , for AB
If A and B are mutually exclusive,
0
Pr
0
Pr
Pr|Pr
BB
BABA , for AB
Conditional probabilities are generally not symmetric!
BBABA Pr|PrPr , for 0Pr B
AABBA Pr|PrPr , for 0Pr A
and
B
BABA
Pr
Pr|Pr
, for 0Pr B
A
BAAB
Pr
Pr|Pr
, for 0Pr A
A
BBA
A
B
B
BA
B
B
A
BAAB
Pr
Pr|Pr
Pr
Pr
Pr
Pr
Pr
Pr
Pr
Pr|Pr
Therefore
BAAB |Pr|Pr
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 13 of 32 ECE 3800
Continuing Concepts
Conditional Probability
When the probability of an event depends upon prior events. If trials are performed without replacement and/or the initial conditions are not restored, you expect trial outcomes to be dependent on prior results or conditions.
ABA Pr|Pr when A follows B
The joint probability is.
AABBBAABBA Pr|PrPr|Pr,Pr,Pr
Applicable for objects that have multiple attributes and/or for trials performed without replacement.
Resistor Example: Joint and Conditional Probability
Assume we have a bunch of resistors (150) of various impedances and powers… Similar to old textbook problems (more realistic resistor values)
50 ohms 100 ohms 200 ohms Subtotal
¼ watt 40 20 10 70
½ watt 30 20 5 55
1 watt 10 10 5 25
Subtotal 80 50 20 150
Joint Probability (multiple dimensions)
Marginal Probability (for one of the dimensions only)
Conditional Probability (based on the condition, what is the probability)
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 14 of 32 ECE 3800
A Priori and A Posteriori Probability (Sec. 2.2 Bayes Theorem)
The probabilities defined for the expected outcomes, iAPr , are referred to as a priori
probabilities (before the event). They describe the probability before the actual experiment or experimental results are known.
After an event has occurred, the outcome B is known. The probability of the event belonging to one of the expected outcomes can be defined as
BAi |Pr
or from before
BBAAABBA iiii Pr|PrPr|PrPr
B
AABBA ii
i Pr
Pr|Pr|Pr
, for 0Pr B
Using the concept of total probability
nn AABAABAABB Pr|PrPr|PrPr|PrPr 2211
We also have the following forms
nn
iii AABAABAAB
AABBA
Pr|PrPr|PrPr|Pr
Pr|Pr|Pr
2211
or
B
AAB
AAB
AABBA jj
n
i
jjj Pr
Pr|Pr
Pr|Pr
Pr|Pr|Pr
111
This probability is referred to as the a posteriori probability (after the event).
It is also referred to as Bayes Theorem.
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 15 of 32 ECE 3800
2.6 Diamond Network
More Law of total Probability work (LTP)
The vertical link is a new consideration in out paths! By conditioning on the vertical link we can derive two separate problems and then find a general solution.
If we consider Link3 to be broken and connected, we can establish the total probability
𝑃𝑟 𝑆 → 𝐷 𝑃𝑟 𝑆 → 𝐷|𝐿 1 ∙ 𝑃𝑟 𝐿 1 𝑃𝑟 𝑆 → 𝐷|𝐿 0 ∙ 𝑃𝑟 𝐿 0
For equal probability of links
The diamond network solution then becomes
𝑃𝑟 𝑆 → 𝐷 𝑃𝑟 𝑆 → 𝐷|𝐿 1 ∙ 𝑃𝑟 𝐿 1 𝑃𝑟 𝑆 → 𝐷|𝐿 0 ∙ 𝑃𝑟 𝐿 0
𝑃𝑟 𝑆 → 𝐷 4 ∙ 𝑝 4 ∙ 𝑝 𝑝 ∙ 𝑝 2 ∙ 𝑝 𝑝 ∙ 1 𝑝
𝑃𝑟 𝑆 → 𝐷 4 ∙ 𝑝 4 ∙ 𝑝 𝑝 2 ∙ 𝑝 𝑝 2 ∙ 𝑝 𝑝
𝑃𝑟 𝑆 → 𝐷 2 ∙ 𝑝 2 ∙ 𝑝 5 ∙ 𝑝 2 ∙ 𝑝
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 16 of 32 ECE 3800
Chapter 3: A LITTLE COMBINATORICS
(i) Sampling with replacement is the easy way …
Possible combinations rnnnnn
(ii) Sampling without replacement
Possible combinations !!
121rn
nrnnnn
Next considerations … how many ways can the r things be selected …
Possible selections !121 rrrr
Now we can consider the “unique” combinations
Unique combinations !!
!
rrn
n
lectionPossibleSe
mbinationPossibleCo
We have now defined an operator to determine unique values for “n choose r”
!!
!
rrn
n
r
nC n
r
also sometimes shown as !!
!,
rrn
nrnCCrn
Binomial Probability of Bernoulli Trials
The binomial distribution arises from the summation of Independent and Identically Distributed (IID) Bernoulli R.V.
𝑝 𝑘1 𝑝 𝑞, 𝑋 0, 𝑘 0
𝑝, 𝑋 1, 𝑘 1
Then, 𝑆 𝑋 𝑋 𝑋 ⋯ 𝑋
𝑝 𝑆 𝑘 𝑃𝑟 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒𝑠 𝑤𝑖𝑡ℎ 𝑘 1 𝑠 𝑎𝑛𝑑 𝑛 𝑘 0′𝑠. " "
As described previously, the number of sequences with exactly k ones in N trials is based on the combinatorial computation of the binomial coefficient.
𝑝 𝑆 𝑘 𝑝 𝑘𝑛𝑘 ∙ 𝑝 ∙ 𝑞
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 17 of 32 ECE 3800
5-Card Draw Combinatorial
How many ways can 5 cards be drawn from a deck of 52 playing cards?
!!
!
rrn
n
r
nC n
r
!5!552
!52
5
522,598,960
3.1 Basics of Counting
1. Selections are ordered or in an arbitrary order.
2. Selections are made with replacement or not with replacement.
Generalized Formula:
a. Ordered, with replacement of n elements r time 𝑛 ∙ 𝑛 ∙ 𝑛 ⋯ 𝑛 𝑛
b. Ordered without replacement of n elements r times
𝑛 ∙ 𝑛 1 ∙ 𝑛 2 ⋯ 𝑛 𝑟 1𝑛!
𝑛 𝑟 !𝑛
where the textbook has defined new notations..
d. Unordered, without replacement 𝑛 ∙ 𝑛 1 ∙ 𝑛 2 ⋯ 𝑛 𝑟 1
𝑟!𝑛!
𝑛 𝑟 ! ∙ 𝑟!𝑛𝑟!
𝑛𝑟
c. Unordered, with replacement 𝑛 𝑟 1 !𝑛 1 ! ∙ 𝑟!
𝑛 𝑟 1𝑟
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 18 of 32 ECE 3800
3.7 Hypergeometric Probabilities and Card Games
From: http://en.wikipedia.org/wiki/Hypergeometric_distribution
In probability theory and statistics, the hypergeometric distribution is a discrete probability distribution (probability mass function) that describes the number of successes in a sequence of n draws from a finite population without replacement.
A typical example is the following: There is a shipment of N objects in which D are defective. The hypergeometric distribution describes the probability that in a sample of n distinctive objects drawn from the shipment exactly x objects are defective.
n
N
xn
DN
x
D
nDNXx ,,,Pr
for DnxNnD ,min,0max
The equation is derived based on a non-replacement Bernoulli Trials …
Where the denominator term defines the number of trial possibilities, the 1st numerator term defines the number of ways to achieve the desired x, and the 2nd numerator term defines the filling of the remainder of the set.
The text extends this to multiple selections
𝑃𝑟 𝑘 , 2, ⋯ , 𝑘 ,
𝑛𝑘 ∙
𝑛𝑘 ∙ ⋯ ∙
𝑛𝑘
𝑛𝑘
where nj or subsets of n and kj selected from nj
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 19 of 32 ECE 3800
Quality Control Example
A batch of 50 items contains 10 defective items. Suppose 10 items are selected at random and tested. What is the probability that exactly 5 of the items tested are defective?
The number of ways of selecting 10 items out of a batch of 50 is the number of combinations of size 10 from a set of 50 objects:
!40!10
!50
10
505010
C
The number of ways of selecting 5 defective and 5 nondefective items from the batch of 50 is the product N1 x N2 where N1 is the number of ways of selecting the 5 items from the set of 10 defective items, and N2 is the number of ways of selecting 5 items from the 40 nondefective items.
!35!5
!40
!5!5
!10405
105 CC
Thus the probability that exactly 5 tested items are defective is the desired ways the selection can be made divided by the total number of ways selection can be made, or
0.01614 01027227817
658008252
!40!10
!50!35!5
!40
!5!5
!10
5010
405
105
C
CC
Suppose 10 items are selected at random and tested. What is the probability that exactly 1 of the items tested are defective?
𝐶 ∙ 𝐶
𝐶
10!9! ∙ 1! ∙ 40!
31! ∙ 9!50!
40! ∙ 10!
10 ∙ 27343888010272278170
0.2662
nchoosek(10,1)*nchoosek(40,9)/nchoosek(50,10)
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 20 of 32 ECE 3800
Textbook box with Marbles Example
A box contains 9 marbles (4 red and 5 blue). Six marbles are blindly selected from the box. What is the probability of two red marbles and four blue marbles?
𝐶 ∙ 𝐶𝐶
4!2! ∙ 2! ∙ 5!
4! ∙ 1!9!
6! ∙ 3!
6 ∙ 59 ∙ 8 ∙ 7/6
6 ∙ 53 ∙ 4 ∙ 7
514
0.3571
nchoosek(10,1)*nchoosek(40,9)/nchoosek(9,6)= 0.3571
Textbook Poker Full House in 5 cards
Draw 5 cards from a deck of 52. For s specific full house, assume 3 queens and 2 eights.
The probability becomes
𝐶 ∙ 𝐶
𝐶
4!3! ∙ 1! ∙ 4!
2! ∙ 2!52!
47! ∙ 5!
4 ∙ 4 ∙ 32
52 ∙ 51 ∙ 50 ∙ 49 ∙ 48120
𝐶 ∙ 𝐶
𝐶
242598960
1108290
9.2345𝑒 06
The number of possible unique full houses is … 13 3-card groups and 12 2 card groups for a total of 13 x 12 = 156.
Therefore the probability of being dealt a full house of any kind is
13 ∙ 12 ∙𝐶 ∙ 𝐶
𝐶
156108290
1.4406𝑒 03
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 21 of 32 ECE 3800
Chapter 4: DISCRETE PROBABILITIES AND RANDOM VARIABLES
Probability Mass Functions (pmf)
From: http://en.wikipedia.org/wiki/Probability_mass_function
In probability theory, a probability mass function (abbreviated pmf) gives the probability that a discrete random variable is exactly equal to some value.
Cumulative Distribution Function (CDF)
Probability mass functions are related discrete countable outcomes of an experiment and the probability each outcomes has at the discrete values.
Generalized properties of pmf (from Stark and Wood)
Properties of the pmf include 1. xforxf X ,0
2. 1
u
X uf
3.
x
u
XX ufxF
4.
2
1
21Prx
xuX ufxXx
Generalized Properties of CDF (from Stark and Wood)
Cumulative Distribution Function (CDF):The probability of the event that the observed random variable X is less than or equal to the allowed value x.
xXxFX Pr
The defined function can be discrete or continuous along the x-axis. Constraints on the cumulative distribution function are:
xforxFX ,10
0XF and 1XF (property #1 and #3 in the textbook)
XF is non-decreasing as x increases (property #2 in the textbook)
1221Pr xFxFxXx XX
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 22 of 32 ECE 3800
4.3 Expected Values, Moments, Central Moments and Variance
For random variables, the expected value operation produces a “probabilistic average” of the particular probability based function of interest.
𝐸 𝑔 𝑋 𝑔 𝑥 ∙ 𝑝 𝑘
https://en.wikipedia.org/wiki/Expected_value
where g(X) is a function of the random variable X and p(k) is the pmf. (Chapter 3 ROI calculations are expected values!)
Mean or 1st Moment
For example the 1st moment or mean value of the random variable is defined by
𝜇 ≡ 𝐸 𝑋 𝑋 𝑥 ∙ 𝑝 𝑘
The mean square value or second moment is
𝐸 𝑋 𝑋 𝑥 ∙ 𝑝 𝑘
Other “moments” ( the nth moment) are defined as
𝐸 𝑋 𝑋 𝑥 ∙ 𝑝 𝑘
Central Moments
𝐸 𝑋 𝜇 𝑋 𝜇 ∙ 𝑝 𝑘
The 2nd central moment or variance
𝜎 ≡ 𝐸 𝑋 𝜇 𝑋 𝜇 ∙ 𝑝 𝑘
The standard deviation is
𝜎 𝜎
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 23 of 32 ECE 3800
Importance of mean and standard deviation
Often when we talk about values we say
𝜇 𝜎
That is to say that we expect the result x to be
𝜇 𝜎 𝑥 𝜇 𝜎
A useful variance – moments proof (Theorem 4.1)
𝜎 𝐸 𝑋 𝜇 𝑋 𝜇 ∙ 𝑝 𝑘
𝜎 𝐸 𝑋 𝜇 𝑋 2 ∙ 𝜇 ∙ 𝑋 𝜇 ∙ 𝑝 𝑘
𝜎 𝐸 𝑋 𝜇 𝑋 ∙ 𝑝 𝑘 2 ∙ 𝜇 ∙ 𝑋 ∙ 𝑝 𝑘 𝜇 ∙ 𝑝 𝑘
𝜎 𝐸 𝑋 𝜇 𝐸 𝑋 2 ∙ 𝜇 𝜇
𝜎 𝐸 𝑋 𝜇 𝐸 𝑋 𝜇
Note that the second moment is related to signal power/energy!
𝐸 𝑋 𝜇 𝜎
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 24 of 32 ECE 3800
More basics about moments – the expected value operator
A constant, non-random variable
𝐸 𝑐 𝑐 ∙ 𝑝 𝑘 𝑐 ∙ 𝑝 𝑘 𝑐
A constant multiplication
𝐸 𝑐 ∙ 𝑔 𝑋 𝑐 ∙ 𝑔 𝑋 ∙ 𝑝 𝑘 𝑐 ∙ 𝑔 𝑋 ∙ 𝑝 𝑘 𝑐 ∙ 𝐸 𝑔 𝑋
Therefore
𝐸 𝑎 ∙ 𝑋 𝑏 𝑎 ∙ 𝑋 𝑏 ∙ 𝑝 𝑘 𝑏 𝑎 ∙ 𝑋 ∙ 𝑝 𝑘 𝑎 ∙ 𝜇 𝑏
Note: 𝐸 𝑌 𝐸 𝑎 ∙ 𝑋 𝑏 𝑎 ∙ 𝜇 𝑏 𝜇
Summations (superposition)
𝐸 𝑔 𝑋 𝑔 𝑋 𝑔 𝑋 𝑔 𝑋 ∙ 𝑝 𝑘 𝑔 𝑋 ∙ 𝑝 𝑘 𝑔 𝑋 ∙ 𝑝 𝑘
𝐸 𝑔 𝑋 𝑔 𝑋 𝐸 𝑔 𝑋 𝐸 𝑔 𝑋
Multiplication does not work!
Integration and differentiation
𝑑𝑑𝑣
𝐸 𝑔 𝑋, 𝑣𝑑
𝑑𝑣𝑔 𝑋, 𝑣 ∙ 𝑝 𝑘
𝑑𝑑𝑣
𝑔 𝑋, 𝑣 ∙ 𝑝 𝑘 𝐸𝑑
𝑑𝑣𝑔 𝑋, 𝑣
𝐸 𝑔 𝑋, 𝑣 ∙ 𝑑𝑣 𝑔 𝑋, 𝑣 ∙ 𝑝 𝑘 ∙ 𝑑𝑣 𝑔 𝑋, 𝑣 ∙ 𝑑𝑣 ∙ 𝑝 𝑘
𝐸 𝑔 𝑋, 𝑣 ∙ 𝑑𝑣 𝐸 𝑔 𝑋, 𝑣 ∙ 𝑑𝑣
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 25 of 32 ECE 3800
Moment-Generating Functions
The moment generation function (MGF) is the two sided Laplace transform of the probability mass function (pmf) or the probability density function (pdf). If the MGF exists, there is a forward and inverse relationship between the MGF and the pmf/pdf. The MGF is defined based on the expected value as
𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋
𝑀 𝑠 𝑒𝑥𝑝 𝑠 ∙ 𝑋 ∙ 𝑝 𝑘
Why do we do this?
1. It enables a convenient computation of the higher order moments
2. It can be used to estimate fx(x) from experimental measurements of the moments
3. It can be used to solve problems involving the computation of the sums of R.V.
4. It is an important analytical instrument that can be used to demonstrate results and establish additional bounds (the Chernoff Bound and the Central Limit Theorem).
Determining the moments
Perform the Taylor series expansion of the exponential
𝑒𝑥𝑝 𝑥 1𝑥1!
𝑥2!
⋯𝑥𝑛!
⋯
𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋 𝐸 1𝑠 ∙ 𝑋
1!𝑠 ∙ 𝑋
2!⋯
𝑠 ∙ 𝑋𝑛!
⋯
or
𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋 1𝑠 ∙ 𝑚
1!𝑠 ∙ 𝑚
2!⋯
𝑠 ∙ 𝑚𝑛!
⋯
The mi are the ith moments of the density function!
So how would we solve for the moments? By taking derivatives and setting s=0!
Taking the 1st derivative … 𝜕
𝜕𝑠𝑀 𝑠
𝜕𝜕𝑠
𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋 0𝑚1!
2 ∙ 𝑠 ∙ 𝑚2!
⋯𝑛 ∙ 𝑠 ∙ 𝑚
𝑛!⋯
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 26 of 32 ECE 3800
Setting s=0
𝜕𝜕𝑠
𝑀 𝑠 0𝑚1!
0 ⋯ 0 ⋯𝑚1!
Taking the nth derivative …
𝜕𝜕𝑠
𝑀 𝑠 0 ⋯ 0𝑛! ∙ 𝑚
𝑛!𝑛! ∙ 𝑠 ∙ 𝑚
𝑛 1!⋯
Setting s=0
𝜕𝜕𝑠
𝑀 𝑠 0 ⋯ 0𝑛! ∙ 𝑚
𝑛!0 ⋯
𝑛! ∙ 𝑚𝑛!
Therefore, all moments can be determined if the moment generation function exists!
If given a simple pmf, can you provide the exponential terms that form the MGF?!
𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋
𝑀 𝑠 𝑒𝑥𝑝 𝑠 ∙ 𝑋 ∙ 𝑝 𝑘
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 27 of 32 ECE 3800
Chebyshev Inequality Textbook description
Defining a bound based on the variance of a random variable.
𝑃𝑟 |𝑋 𝜇 | 𝜖𝑉𝑎𝑟 𝑋
𝜖
For the Chebyshev Inequality the region of interested is defined based on the equation
𝐴 |𝑋 𝜇 | 𝜖
Note that the Chebyshev Inequality is a bound that applies in all cases. There is no judgment or determination if it is a good or even useful bound!
Note that for any R.V. where the variance tends to zero, you would have
𝑃𝑟 |𝑋 𝜇 | 𝜖 → 0
and the “zero variance random variable must equal the mean value!
Extension, relating epsilon to sigma …
𝜖 𝑘 ∙ 𝜎 with 𝑉𝑎𝑟 𝑋 𝜎
𝑃𝑟 |𝑋 𝜇 | 𝑘 ∙ 𝜎𝜎
𝑘 ∙ 𝜎
𝑃𝑟 |𝑋 𝜇 | 𝑘 ∙ 𝜎1
𝑘
Considering the negative of the probability function
11
𝑘1 𝑃𝑟 |𝑋 𝜇 | 𝑘 ∙ 𝜎
or
11
𝑘𝑃𝑟 |𝑋 𝜇 | 𝑘 ∙ 𝜎
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 28 of 32 ECE 3800
Discrete Uniform pmf
The same probability for m different values from 1 to m.
𝑃𝑟 𝑋 𝑘 𝑝 𝑘
0, 𝑘 11𝑚
, 1 𝑘 𝑚
0, 𝑚 𝑘
Computing the mean or first moment
𝐸 𝑋 𝜇 𝑘 ∙1𝑚
𝐸 𝑋 𝜇1𝑚
∙ 𝑘1𝑚
∙𝑚 ∙ 𝑚 1
2𝑚 1
2
Computing the second moment
𝐸 𝑋 𝑘 ∙1𝑚
𝐸 𝑋1𝑚
∙ 𝑘1𝑚
∙2 ∙ 𝑚 1 ∙ 𝑚 1 ∙ 𝑚
3 ∙ 22 ∙ 𝑚 1 ∙ 𝑚 1
6
Computing the variance or second central moment
𝐸 𝑋 𝜇 𝑉𝑎𝑟 𝑋 𝜎 𝐸 𝑋 𝜇
𝑉𝑎𝑟 𝑋 𝜎𝑚 1
12𝑚 1 ∙ 𝑚 1
12
Computing the Moment Generating Function
𝑀 𝑢 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 𝑘 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙1𝑚
𝑀 𝑢1𝑚
∙ 𝑒𝑥𝑝 𝑢 ∙1 𝑒𝑥𝑝 𝑢 ∙ 𝑚
1 𝑒𝑥𝑝 𝑢1𝑚
∙𝑒𝑥𝑝 𝑢 ∙ 𝑚 1 𝑒𝑥𝑝 𝑢
𝑒𝑥𝑝 𝑢 1
Note that the computation of the moments is not straight forward and requires L’Hospital’s rule instead of direct derivation! (Note that at u=0 you always get 0/0.)
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 29 of 32 ECE 3800
Binomial pmf
𝑃𝑟 𝑋 𝑘 𝑝 𝑘𝑛𝑘 ∙ 𝑝 ∙ 𝑞
Determine the expected value
𝐸 𝑋 𝜇 𝑘 ∙𝑛𝑘 ∙ 𝑝 ∙ 𝑞
𝐸 𝑋 𝜇 𝑛 ∙ 𝑝
Determine the 2nd moment
𝐸 𝑋 𝑘 ∙𝑛𝑘 ∙ 𝑝 ∙ 𝑞
𝐸 𝑋 𝑛 ∙ 𝑛 1 ∙ 𝑝 𝑛 ∙ 𝑝
Determine the variance
𝐸 𝑋 𝜇 𝑉𝑎𝑟 𝑋 𝜎 𝐸 𝑋 𝜇
𝑉𝑎𝑟 𝑋 𝜎 𝑛 ∙ 𝑝 ∙ 𝑝 1 𝑛 ∙ 𝑝 ∙ 𝑞
Determine the Moment Generating Function
𝑀 𝑢 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 𝑘 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙𝑛𝑘 ∙ 𝑝 ∙ 𝑞
nX qtptM exp
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 30 of 32 ECE 3800
Geometric Distribution pmf
𝑃𝑟 𝑋 𝑘 𝑝 𝑘0, 𝑘 0
𝑝 ∙ 1 𝑝 , 𝑘 1,2, ⋯
The 1st Moment
𝐸 𝑋 𝜇 𝑘 ∙ 𝑝 ∙ 1 𝑝
𝐸 𝑋 𝜇 𝑝 ∙1
1 1 𝑝𝑝 ∙
1𝑝
1𝑝
The 2nd Moment
𝐸 𝑋 𝑘 ∙ 𝑝 ∙ 1 𝑝
𝐸 𝑋2 2 ∙ 𝑝
𝑝1𝑝
2 𝑝𝑝
The Variance
𝐸 𝑋 𝜇 𝑉𝑎𝑟 𝑋 𝜎 𝐸 𝑋 𝜇
𝑉𝑎𝑟 𝑋 𝜎2 𝑝
𝑝1
𝑝1 𝑝
𝑝
The Moment Generating Function (MGF)
𝑀 𝑢 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 𝑘 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 ∙ 1 𝑝
𝑀 𝑢𝑝 ∙ 𝑒𝑥𝑝 𝑢
1 𝑒𝑥𝑝 𝑢 ∙ 1 𝑝𝑝
𝑒𝑥𝑝 𝑢 1 𝑝
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 31 of 32 ECE 3800
Textbook Poisson Distribution (4.5.2)
𝑃𝑟 𝑋 𝑘 𝑝 𝑘0, 𝑘 0
𝜆𝑘!
∙ 𝑒𝑥𝑝 𝜆 , 𝑘 0,1,2, ⋯
The 1st Moment
𝐸 𝑋 𝜇 𝑘 ∙𝜆𝑘!
∙ 𝑒𝑥𝑝 𝜆
𝐸 𝑋 𝜇 𝜆
The 2nd Moment
𝐸 𝑋 𝑘 ∙𝜆𝑘!
∙ 𝑒𝑥𝑝 𝜆
𝐸 𝑋 𝜆 𝜆
The Variance
𝐸 𝑋 𝜇 𝑉𝑎𝑟 𝑋 𝜎 𝐸 𝑋 𝜇
𝑉𝑎𝑟 𝑋 𝜎 𝜆 𝜆 𝜆 𝜆
The Moment Generating Function (MGF)
𝑀 𝑢 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙ 𝑝 𝑘 𝑒𝑥𝑝 𝑢 ∙ 𝑘 ∙𝜆𝑘!
∙ 𝑒𝑥𝑝 𝜆
𝑀 𝑢 𝑒𝑥𝑝 𝜆 ∙ 𝑒𝑥𝑝 𝜆 ∙ 𝑒𝑥𝑝 𝑢 𝑒𝑥𝑝 𝜆 ∙ 𝑒𝑥𝑝 𝑢 1
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Spring 2020 32 of 32 ECE 3800
Risk Taking Decision Making (A function of the random variable)
𝑌 𝑔 𝑋1, 𝑃𝑟 𝑋 0 1 𝑝𝑤, 𝑃𝑟 𝑋 1 𝑝
Examples:
Win-lose propositions (X=1 win, X=0 lose) Gambling bets with odds Buying a stock as an investment
The 1st Moment
𝐸 𝑌 𝜇 𝑔 𝑋 ∙ 𝑃𝑟 𝑋
𝐸 𝑌 𝜇 1 ∙ 1 𝑝 𝑤 ∙ 𝑝
𝐸 𝑌 𝜇 𝑤 ∙ 𝑝 𝑝 1
The 2nd Moment
𝐸 𝑌 𝑔 𝑋 ∙ 𝑃𝑟 𝑋
𝐸 𝑌 𝑤 ∙ 𝑝 1 𝑝
The Variance
𝐸 𝑌 𝜇 𝑉𝑎𝑟 𝑌 𝜎 𝐸 𝑌 𝜇
𝑉𝑎𝑟 𝑌 𝜎 𝑝 ∙ 1 𝑝 ∙ 𝑤 1
To have a chance at winning ….
𝐸 𝑌 𝜇 0
𝑤 or 𝑝
Notice that the variance goes up by the square of the amount wagered!