Module #16: Probability Theory
-
Upload
janna-ryan -
Category
Documents
-
view
38 -
download
0
description
Transcript of Module #16: Probability Theory
![Page 1: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/1.jpg)
Module #16:Probability Theory
Rosen 5th ed., ch. 5
Let’s move on to probability, ch. 5.
![Page 2: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/2.jpg)
Terminology• A (stochastic) experiment is a procedure that
yields one of a given set of possible outcomes• The sample space S of the experiment is the
set of possible outcomes.• An event is a subset of sample space.• A random variable is a function that assigns a
real value to each outcome of an experimentNormally, a probability is related to an experiment or a trial.
Let’s take flipping a coin for example, what are the possible outcomes?
Heads or tails (front or back side) of the coin will be shown upwards.After a sufficient number of tossing, we can “statistically” concludethat the probability of head is 0.5.In rolling a dice, there are 6 outcomes. Suppose we want to calculate the prob. of the event of odd numbers of a dice. What is that probability?
![Page 3: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/3.jpg)
Probability: Laplacian Definition
• First, assume that all outcomes in the sample space are equally likely– This term still needs to be defined.
• Then, the probability of event E in sample space S is given by Pr[E] = |E|/|S|.
Even though there are many definitions of probability, I would like touse the one from Laplace. The expression “equally likely” may be a little bit vague from the perspective of pure mathematics. But in engineering viewpoint, I think that is ok.
![Page 4: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/4.jpg)
Probability of Complementary Events
• Let E be an event in a sample space S.
• Then, E represents the complementary event.
• Pr[E] = 1 − Pr[E]
• Pr[S] = 1
![Page 5: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/5.jpg)
Probability of Unions of Events
• Let E1,E2 S
• Then: Pr[E1 E2] = Pr[E1] + Pr[E2] − Pr[E1E2]
– By the inclusion-exclusion principle.
![Page 6: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/6.jpg)
Mutually Exclusive Events
• Two events E1, E2 are called mutually exclusive if they are disjoint: E1E2 =
• Note that two mutually exclusive events cannot both occur in the same instance of a given experiment.
• For mutually exclusive events,Pr[E1 E2] = Pr[E1] + Pr[E2].
![Page 7: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/7.jpg)
Exhaustive Sets of Events
• A set E = {E1, E2, …} of events in the sample space S is exhaustive if
• An exhaustive set of events that are all mutually exclusive with each other has the property that
SEi
1]Pr[ iE
![Page 8: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/8.jpg)
Independent Events
• Two events E,F are independent if Pr[EF] = Pr[E]·Pr[F].
• Relates to product rule for number of ways of doing two independent tasks
• Example: Flip a coin, and roll a die.Pr[ quarter is heads die is 1 ] =
Pr[quarter is heads] × Pr[die is 1]
Now the question is: how we can figure out whether two events are independent or not
![Page 9: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/9.jpg)
Conditional Probability
• Let E,F be events such that Pr[F]>0.• Then, the conditional probability of E given F,
written Pr[E|F], is defined as Pr[EF]/Pr[F].• This is the probability that E would turn out to
be true, given just the information that F is true.
• If E and F are independent, Pr[E|F] = Pr[E].
Here is the most important part in the probability, the cond. prob.
By the cond. prob., we can figure out whether there is a correlation or dependency between two probabilities.
![Page 10: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/10.jpg)
Bayes’ Theorem
• Allows one to compute the probability that a hypothesis H is correct, given data D:
]Pr[
]Pr[]|Pr[]|Pr[
D
HHDDH
jjj
iii HHD
HHDDH
]Pr[]|Pr[
]Pr[]|Pr[]|Pr[
Set of Hj is exhaustive
![Page 11: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/11.jpg)
Bayes’ theorem: example• Suppose 1% of population has AIDS• Prob. that the positive result is right: 95%• Prob. that the negative result is right: 90%• What is the probability that someone who has
the positive result is actually an AIDS patient?
• H: event that a person has AIDS• D: event of positive result• P[D] = P[D|H]P[H]+P[D|H]P[H ]
= 0.95*0.01+0.1*0.99=0.1085• P[H|D] = 0.95*0.01/0.1085=0.0876
![Page 12: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/12.jpg)
Expectation Values
• For a random variable X(s) having a numeric domain, its expectation value or expected value or weighted average value or arithmetic mean value E[X] is defined as
Ss
sXsp )()(
![Page 13: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/13.jpg)
Linearity of Expectation
• Let X1, X2 be any two random variables derived from the same sample space. Then:
• E[X1+X2] = E[X1] + E[X2]
• E[aX1 + b] = aE[X1] + b
![Page 14: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/14.jpg)
Variance
• The variance Var[X] = σ2(X) of a random variable X is the expected value of the square of the difference between the value of X and its expectation value E[X]:
• The standard deviation or root-mean-square (RMS) difference of X, σ(X) :≡ (Var[X])1/2.
Ss
spXsXX )(])[)((:][ 2EVar
![Page 15: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/15.jpg)
Visualizing Sample Space
• 1. Listing– S = {Head, Tail}
• 2. Venn Diagram
• 3. Contingency Table
• 4. Decision Tree Diagram
![Page 16: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/16.jpg)
SS
TailTail
HHHH
TTTT
THTHHTHT
Sample SpaceSample SpaceS = {HH, HT, TH, TT}S = {HH, HT, TH, TT}
Venn Diagram
OutcomeOutcome
Experiment: Toss 2 Coins. Note Faces.Experiment: Toss 2 Coins. Note Faces.
Event Event
![Page 17: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/17.jpg)
22ndnd CoinCoin11stst CoinCoin HeadHead TailTail TotalTotal
HeadHead HHHH HTHT HH, HTHH, HT
TailTail THTH TTTT TH, TTTH, TT
TotalTotal HH,HH, THTH HT,HT, TTTT SS
Contingency Table
Experiment: Toss 2 Coins. Note Faces.Experiment: Toss 2 Coins. Note Faces.
S = {HH, HT, TH, TT}S = {HH, HT, TH, TT} Sample SpaceSample Space
OutcomeOutcomeSimpleSimpleEvent Event (Head on(Head on1st Coin)1st Coin)
![Page 18: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/18.jpg)
EventEventEventEvent BB11 BB22 TotalTotal
AA11 P(AP(A1 1 BB11)) P(AP(A1 1 BB22)) P(AP(A11))
AA22 P(AP(A2 2 BB11)) P(AP(A2 2 BB22)) P(AP(A22))
TotalTotal P(BP(B11)) P(BP(B22)) 11
Event Probability Using Contingency Table
Joint ProbabilityJoint Probability Marginal (Simple) ProbabilityMarginal (Simple) Probability
![Page 19: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/19.jpg)
Marginal probability
• Let S be partitioned into m x n disjoint sets Ei and Fj where the general subset
is denoted Ei Fj . Then the marginal
probability of Ei is
j
n
ji FE
1
![Page 20: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/20.jpg)
Tree Diagram
Outcome Outcome
S = {HH, HT, TH, TT}S = {HH, HT, TH, TT} Sample SpaceSample Space
Experiment: Toss 2 Coins. Note Faces.Experiment: Toss 2 Coins. Note Faces.
TT
HH
TT
HH
TT
HHHH
HTHT
THTH
TTTT
HH
![Page 21: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/21.jpg)
Discrete Random Variable
– Possible values (outcomes) are discrete• E.g., natural number (0, 1, 2, 3 etc.)
– Obtained by Counting– Usually Finite Number of Values
• But could be infinite (must be “countable”)
![Page 22: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/22.jpg)
Discrete Probability Distribution
1.List of All possible [x, p(x)] pairs– x = Value of Random Variable (Outcome)– p(x) = Probability Associated with Value
2.Mutually Exclusive (No Overlap)
3.Collectively Exhaustive (Nothing Left Out)
4. 0 p(x) 1
5. p(x) = 1
![Page 23: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/23.jpg)
Visualizing Discrete Probability Distributions
• { (0, .25), (1, .50), (2, .25) }
• { (0, .25), (1, .50), (2, .25) }
ListingListingTableTable
GraphGraph EquationEquation
# # TailsTails f(xf(x))CountCount
p(xp(x))
00 11 .25.2511 22 .50.5022 11 .25.25
pp xxnn
xx nn xxpp ppxx nn xx(( ))
!!
!! (( )) !!(( ))
11
.00.00
.25.25
.50.50
00 11 22xx
p(x)p(x)
![Page 24: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/24.jpg)
Cumulative Distribution Function (CDF)
ax
x xpaXaF Pr
![Page 25: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/25.jpg)
Binomial Distribution
1. Sequence of n Identical Trials
2. Each Trial Has 2 Outcomes– ‘Success’ (Desired/specified Outcome) or ‘Failure’
3. Constant Trial Probability
4. Trials Are Independent
5. # of successes in n trials is a binomial random variable
![Page 26: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/26.jpg)
Binomial Probability Distribution Function
xnxxnx ppxnx
nqp
x
nxp
)1(
)!(!
!)( xnxxnx pp
xnx
nqp
x
nxp
)1(
)!(!
!)(
pp((xx) = Probability of ) = Probability of x x ‘Successes’‘Successes’
nn == SampleSample Size Size
pp == Probability of ‘Success’Probability of ‘Success’
xx == Number of ‘Successes’ in Number of ‘Successes’ in SampleSample ( (xx = 0, 1, 2, ..., = 0, 1, 2, ..., n n))
![Page 27: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/27.jpg)
Binomial Distribution Characteristics
.0
.2
.4
.6
0 1 2 3 4 5
X
P(X)
.0
.2
.4
.6
0 1 2 3 4 5
X
P(X)
.0
.2
.4
.6
0 1 2 3 4 5
X
P(X)
.0
.2
.4
.6
0 1 2 3 4 5
X
P(X)
n = 5 p = 0.1
n = 5 p = 0.5
E x np
np p
( )
( )1
E x np
np p
( )
( )1
MeanMean
Standard DeviationStandard Deviation
![Page 28: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/28.jpg)
Useful Observation 1
• For any X and Y
YEXE
yypxxp
yxpyyxpx
yxypyxxp
yxpyxYXE
yx
y xx y
x yx y
x y
)()(
)()(
)(
![Page 29: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/29.jpg)
One Binary Outcome
• Random variable X, one binary outcome
• Code success as 1, failure as 0
• P(success)=p, P(failure)=(1-p)=q
• E(X) = p
pppp
pqp
XEXE
XEXEXVar
1
0*1*2
222
22
2
![Page 30: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/30.jpg)
• Independent, identically distributed
• X1, …, Xn; E(Xi)=p; Binomial X =
Mean of a Binomial
np
XnE
XE
XE
i
i
1
iX
By useful By useful observation 1observation 1
![Page 31: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/31.jpg)
Useful Observation 2
• For independent X and Y
YEXEXEYExxpYE
YExxpyypxxp
ypxxyp
yxpxyXYE
x
xx y
x y
x y
)(
![Page 32: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/32.jpg)
Useful Observation 3
• For independent X and Y
222
22
2 YEXEXYYXE
YXEYXEYXVar
YVarXVarYEYEXEXE
YEXEXYE
YEYEXEXE
YEXEYEXE
XYEYEXE
2222
2222
22
22
22
2
2
cancelled by obs. 2cancelled by obs. 2
![Page 33: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/33.jpg)
Variance of Binomial
• Independent, identically distributed
• X1, …, Xn; E(Xi)=p; Binomial X =
pnp
XnVar
XVarXVar
i
ii
1
iX
![Page 34: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/34.jpg)
Useful Observation 4
• For any X
XVark
XEkXEk
XkEXEk
kXEXkEkXVar
2
2222
222
222
![Page 35: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/35.jpg)
Continuous random variable
![Page 36: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/36.jpg)
Continuous Prob. Density Function
1.Mathematical Formula
2.Shows All Values, x, and Frequencies, f(x)– f(x) Is Not Probability
3.Properties
((Area Under Curve)Area Under Curve)ValueValue
((Value, Frequency)Value, Frequency)
f(x)f(x)
aa bbxxff xx dxdx
ff xx
(( ))
(( ))
All All xx
aa x x bb
11
0,0,
![Page 37: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/37.jpg)
Continuous Random Variable Probability
Probability Is Area Probability Is Area Under Curve!Under Curve!
PP cc xx dd ff xx dxdxcc
dd(( )) (( ))
f(x)f(x)
Xc d
![Page 38: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/38.jpg)
Uniform Distribution1. Equally Likely Outcomes
2. Probability Density
3. Mean & Standard Deviation Mean Mean
MedianMedian
f xd c
( ) 1
f xd c
( ) 1
c d d c
2 12
c d d c2 12
1d c
1d c
x
f(x)
dc
![Page 39: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/39.jpg)
Uniform Distribution Example
• You’re production manager of a soft drink bottling company. You believe that when a machine is set to dispense 12 oz., it really dispenses 11.5 to 12.5 oz. inclusive.
• Suppose the amount dispensed has a uniform distribution.
• What is the probability that less than 11.8 oz. is dispensed?
![Page 40: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/40.jpg)
Uniform Distribution Solution
P(11.5 x 11.8) = (Base)(Height)
= (11.8 - 11.5)(1) = 0.30
11.511.5 12.512.5
ff((xx))
xx11.811.8
1 112 5 11511
10
d c
. .
.
1 112 5 11511
10
d c
. .
.
1.01.0
![Page 41: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/41.jpg)
Normal Distribution
1. Describes Many Random Processes or Continuous Phenomena
2. Can Be Used to Approximate Discrete Probability Distributions
– Example: Binomial
3. Basis for Classical Statistical Inference
4. A.k.a. Gaussian distribution
![Page 42: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/42.jpg)
Normal Distribution
1. ‘Bell-Shaped’ & Symmetrical
2.Mean, Median, Mode Are Equal
4. Random Variable Has Infinite Range Mean: Mean: 평균 평균
Median: Median: 중간값 중간값 Mode: Mode: 최빈값최빈값
X
f(X)
X
f(X)
* light-tailed distribution
![Page 43: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/43.jpg)
Probability Density Function
2
2
1
e2
1)(
x
xf
2
2
1
e2
1)(
x
xf
f(x) = Frequency of Random Variable x = Population Standard Deviation = 3.14159; e = 2.71828x = Value of Random Variable (-< x < ) = Population Mean
![Page 44: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/44.jpg)
Effect of Varying Parameters ( & )
X
f(X)
CA
B
![Page 45: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/45.jpg)
Normal Distribution Probability
?)()( dxxfdxcPd
c
c dx
f(x)
c dx
f(x)
Probability is Probability is area under area under curve!curve!
![Page 46: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/46.jpg)
X
f(X)
X
f(X)
Infinite Number of Tables
Normal distributions differ by Normal distributions differ by mean & standard deviation.mean & standard deviation.
Each distribution would Each distribution would require its own table.require its own table.
That’s an That’s an infinite infinite number!number!
![Page 47: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/47.jpg)
Standardize theNormal Distribution
X
X
One table! One table!
Normal DistributionNormal Distribution
= 0
= 1
Z = 0
= 1
Z
ZX
ZX
Standardized
Normal DistributionStandardized
Normal Distribution
![Page 48: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/48.jpg)
Intuitions on Standardizing
• Subtracting from each value X just moves the curve around, so values are centered on 0 instead of on
• Once the curve is centered, dividing each value by >1 moves all values toward 0, pressing the curve
![Page 49: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/49.jpg)
Standardizing Example
X= 5
= 10
6.2 X= 5
= 10
6.2
Normal DistributionNormal Distribution
ZX
6 2 5
1012
..Z
X
6 2 510
12.
.
![Page 50: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/50.jpg)
Standardizing Example
X= 5
= 10
6.2 X= 5
= 10
6.2
Normal DistributionNormal Distribution
ZX
6 2 5
1012
..Z
X
6 2 510
12.
.
Z= 0
= 1
.12 Z= 0
= 1
.12
Standardized Normal Distribution
Standardized Normal Distribution
![Page 51: Module #16: Probability Theory](https://reader035.fdocuments.net/reader035/viewer/2022062517/568137f2550346895d9fb04f/html5/thumbnails/51.jpg)
Z= 0
= 1
.12 Z= 0
= 1
.12
Z .00 .01
0.0 .0000 .0040 .0080
.0398 .0438
0.2 .0793 .0832 .0871
0.3 .1179 .1217 .1255
Z .00 .01
0.0 .0000 .0040 .0080
.0398 .0438
0.2 .0793 .0832 .0871
0.3 .1179 .1217 .1255
Obtaining the Probability
.0478.0478.0478.0478
.02.02
0.10.1 .0478
Standardized Normal Standardized Normal Probability Table (Portion)Probability Table (Portion)Standardized Normal Standardized Normal Probability Table (Portion)Probability Table (Portion)
ProbabilitiesProbabilitiesProbabilitiesProbabilitiesShaded area Shaded area exaggeratedexaggeratedShaded area Shaded area exaggeratedexaggerated