Chapter 2 Concepts of Prob. Theory
description
Transcript of Chapter 2 Concepts of Prob. Theory
tch-prob 1
Chapter 2 Concepts of Prob. Theory • Random Experiment: an experiment in which outcome
varies in an unpredictable fashion when the experiment is repeated under the same condition
Specified by: 1. An experimental procedure 2. One or more measurements or observations
tch-prob 2
EXAMPLE 2.1 :
Experiment E1 : Select a ball form an urn containing balls numbered 1 to 50.
Note the number of the ball.
Experiment E2 : Select a ball form an urn containing balls numbered 1 to 4.
Suppose that balls 1 and 2 are black and that balls 3 and 4 are white. Note
number and color of the ball you select.
Experiment E3 : Toss a coin three times and note the sequence of heads/
tails.
Experiment E4 : Toss a coin three times and note the number of heads.
Experiment E6 : A block of information is transmitted repeatedly over a
noisy channel until an error-free block arrives at the receiver.
Experiment E7 : Pick a number at random between zero and one.
Experiment E12 : Pick two numbers at random between zero and one.
Experiment E13 : Pick a number X at random between zero and one, then
pick a number Y at random between zero and X.
tch-prob 3
Discussions
• Compare E3 and E4: same procedure, different observations
• A random experiment with multiple measurements or observations: E2, E3, E12, E13
• sequential experiment consists of multiple subexperiments: E3, E4, E6, E12, E13
• dependent subexperiments: E13
tch-prob 4
Sample Space• Outcome, sample point:
– one and only one per experiment– mutually exclusive, cannot occur simultaneously
• Sample space: set of all possible outcomes
• Example 2.2: sample spaces
• number of outcomes: – Finite– countably infinite S6
– unaccountably infinite S7
1 2 3 6 7 12 13, , , , ,,S S S S S S S
ContinuousSample space
Discrete sample space
tch-prob 5
1
2
3
4
6
7
12
13
1,2,......,50
1, , 2, , 3, , 4,
, , , , , , ,
0,1,2,3
1,2,3,
: 0 1 0,1
, : 0 1 0 1
, : 0 1
S
S b b w w
S HHH HHT HTH THH TTH THT HTT TTT
S
S
S x x
S x y x and y
S x y y x
EXAMPLE 2.2 :
The sample spaces corresponding to the experiments in Example
2.1 are given below using set notation :
tch-prob 6
Event: a subset of S, a collection of outcomes that satisfy certain conditions
Certain event: S
null event:
elementary event: event of a single outcome
Example 2.3:
A2: The ball is white and even-numbered
A4: The number of heads equals the number of tails
A7: The number selected is nonnegative
, ,72 4
E E E
tch-prob 7
Set Operations
1. Union
2. Intersection
mutually exclusive
3. Complement
imply
equal A=B
- Commutative properties of set operations
- Associative properties of set operations
- Distributive properties of set operations
- De Morgan’s Rules
Venn diagrams: Fig. 2.2
A B
A B
A B
cA S A
A B
A B B A A B B A
( ) ( )A B C A B C ( ) ( )A B C A B C
( ) ( ) ( )A B C A B A C
( )c c cA B A B ( )c c cA B A B
A B
tch-prob 8
Example 2.5: Configuration of a three-component system
a. series all three
b. parallel at least one of three
c. two-out-of-three
.....1 21
.....1 21
1
nA A A Ankk
nA A A Ankk
Akk
1
Akk
1 2 3A A AI I
1 2 3A A A
1 2 3 1 2 3( ) ( ) ...cA A A A A A
: event that component is functioningkA k
C1 C2 C3
C1
C2
C3
tch-prob 9
2.2 The Axioms of Probability
A probability law for the experiment E is a rule that assigns to each event a number P(A), called the probability of A, that satisfies the following axioms:
Axiom I: nonnegative
Axiom II: P(S)=1 total=1
Axiom III: If , then
Axiom III’: If for all , then
0 ( )P A
A B P A B P A P B
A Ai j i j11
P A P Ak kkk
A set of consistency rules that any valid probability assignment must satisfy.
tch-prob 10
Corollary 1.
pf:
Corollary 2. pf: from Cor.1, Corollary 3. pf: Let A=S, in Cor.1.
Corollary 4. are pairwise mutually exclusive, then for
1cP A P A
cA A
1 c cP S P A A P A P A
1P A
1 1cP A P A
0P
cA
, ,...., ,1 2
A A An
11
n nP A P A
k kkk
2n
tch-prob 11
Corollary 5.
pf:
Corollary 6.
Corollary 7. If , then pf.
P A B P A P B P A B
P A
P B
c cP A B P A B P B A P A BcP A B P A BcP B A P A B
A B
1.... 1 ....
111
n n nP A P A P A A P A Anj jk kj j kk
cP B P A P A B P A
A B P A P B
tch-prob 12
Axioms + corollaries provide rules for computing the probability of
certain events in terms of other events. However, we still need initial probability assignment
1. Discrete Sample Spaces
find the prob.of elementary events;
all distinct elementary events are mutually exclusive,
Example 2.6., 2.7
2. Continuous Sample Spaces
assign prob. to intervals of the real line or rectangular regions in the plane
Example 2.11
tch-prob 13
Discrete Sample Spaces
First, suppose that the sample space is finite, and
is given by 1 2, nS a a a
' ' '1 2, , , mP B P a a a
' ' '1 2 (2.9)mP a P a P a
If S is countably infinite, then Axiom ’ implies that the probabiliⅢty of an event such as is given by '
2'1,bbD
' '1 2 (2.10)P D P b P b
(by corollary ?)
tch-prob 14
If the sample space has n elements, , a probability
assignment of particular interest is the case of equally likely
outcomes. The probability of the elementary events is
The probability of any event that consists of k outcomes, say
, is
naaS ,1
(2.11) n
aPaPaP n
121
''1, kaaB
2.12 n
kaPaPBP k ''
1
Thus, if outcomes are equally likely, then the probability of
an event is equal to the number of outcomes in the event
divided by the total number of outcomes in the sample space.
(remember the classical definition?)
tch-prob 15
EXAMPLE 2.6
An urn contains 10 identical balls numbered 0,1,…, 9. A random
experiment involves selecting a ball from the urn and noting the
number of the ball. Find the probability of the following events :
A = “number of ball selected is odd,”
B = “number of ball selected is a multiple of 3,”
C = “number of ball selected is less than 5,”
and of and .BA CBA
tch-prob 16
The sample space is , so the sets of outcomes corre
sponding to the above events are
, , and .
If we assume that the outcomes are equally likely, then
.
.
From Corollary 5.
.
9,,1,0 S
9,7,5,3,1A 9,6,3B 4,3,2,1,0C
10
597531 PPPPPAP
10
3963 PPPBP
10
543210 PPPPPCP
10
6
10
2
10
3
10
5 BAPBPAPBAP
tch-prob 17
where we have used the fact that , so
. From Corollary 6,
9,3 BA 102 BAP
BAPCPBPAPCBAP
P A C P B C P A B C
10
1
10
1
10
2
10
2
10
5
10
3
10
5
10
9
tch-prob 18
EXAMPLE 2.7
Suppose that a coin is tossed three times. If we observe the sequenc
e of heads and tails, then there are eight possible outcomes
. If we assume t
hat the outcomes of S3 are equiprobable, then the probability of each
of the eight elementary events is 1/8. This probability assignment imp
lies that the probability of obtaining two heads in three tosses is, by C
orollary 3,
TTTHTTTHTTTHTHHHTHHHTHHHS ,,,,,,,
THHHTHHHTPtossesinheadsP ,,"32"
8
3 THHPHTHPHHTP
4
12"32" PtossesinheadsP
*If we count the number of heads in three tosses, then 0,1,2,3S
tch-prob 19
Continuous Sample Spaces
EXAMPLE 2.11
Consider Experiment E12, where we picked two number x and y at
random between zero and one. The sample space is then the unit
square shown in Fig. 2.8(a). If we suppose that all pairs of numbers
in the unit square are equally likely to be selected, then it is
reasonable to use a probability assignment in which the probability of
any region R inside the unit square is equal to the area of R. Find the
probability of the following events: , , and
.
Figures 2.8(b) through 2.8(c) show the regions
corresponding to the events A, B, and C. Clearly each of these
regions has area ½. Thus
5.0 xA 5.0 yB
yxC
2
1AP
2
1BP
2
1CP
tch-prob 20
FIGURE 2.8 a Two-dimensional sample space and three events.
(a) Sample space
x
y
0 1
S
y
x
(b) Event
0 1x
y
2
1x
2
1x
(c) Event
0 1 x
y
2
1y
2
1y (d) Event
0 1 x
y
yx
yx
tch-prob 21
2.3 Computing probabilities using counting methods
In many experiments with finite sample space, the outcomes can be
assumed to be equiprobable.
Prob. Counting the number of outcomes in an event
1. Sampling with Replacement and with Ordering
choose k objects from a set A that has n distinct objects, with replacement and note the order. The experiment produces an ordered k-tuple
where and
The number of distinct ordered k-tuples =
Ex.2.12 5 balls, select 2
Pr[2 balls with same number] =
x Ai 1,2,...,i k
kn
1,2,3,4,5 25 25525
1, 2,..., kx x x
tch-prob 22
2. Sampling without Replacement and with ordering
choose k objects from a set A that has n distinct objects, without replacement.
number of distinct ordered k-tuples = n (n-1)…(n-k+1)
Ex.2.13 5 balls, select 2
Pr [1st number >2nd number ]= 10/20
k n
5 4 20
tch-prob 23
Permutations of n Distinct Objects
Consider the case when k=n
number of permutations of n objects = n!
For large n, stirling’s formula is useful
Ex. 2.16 12 balls randomly placed into 12 cells, where more than 1 ball is
allowed to occupy a cell.
Pr [ all cells are occupied ]= ~
12! 2
nnn n e
1212!
1255.37 10
tch-prob 24
3. Sampling without Replacement and without ordering
~ ” combination of size k” =
each combination has k! possible orders
“binomial coefficient”, read “n choose k”
Ex. 2.18 The number of distinctive permutation of k white balls and n-k black balls
n
kC
! 1 .... 1
1 .... 1 !! ! !
n
k
n
k
C k n n n k
n n n k nnCk k n k k
nk
!
! !n
k n k
tch-prob 25
Ex. 2.19 A batch of 50 items contains 10 defective items. Select 10 items at random, Pr [ 5 out of 10 defective ]=?
Exercise prob. 47: Multinomial
Ex.2.20 Toss a die 12 times. Pr [ each number appears twice ]= ?
{1,1,2,2,3,3, …….6,6} permutation 12
12!2!2!2!2!2!2!
6
10 40
5 550
10
tch-prob 26
4. Sampling with Replacement and without ordering
Suppose k=5, n=4
object 1 2 3 4
XX X XX
The result can be summarized as XX // X / XX
= k: x
n-1: /
=
Ex. Place k balls into n cells.
1 1
1
n k n k
k n
88!3!5! 3
1 !! 1 !
k nk n
Pick k balls from an urn ofn balls. (k may be greater than n)
tch-prob 27
2.4 Conditional Probability
We are interested in knowing whether A and B are related, i.e. ,
if B occurs, does it alter the likelihood of A occurs.
Define the conditional probability as
for P[B]>0
- renormalize
|P A B
P A BP B
I
B
A
[ ]P A BI
tch-prob 28
Example 2.21. A ball is selected from an urn containing 2 black balls (1,2)
and 2 white balls (3,4).
A: Black ball selected ½
B: even-numbered ball ½
C: number > 2 ½
0.25 0.50.5
P A BP A B P A
P B
I
0 00.5
P A CP A C P A
P C
I
Not related
Related
tch-prob 29
is useful in finding prob. in sequential experiments
Ex.2.22 An urn contains 2 black, 3 white balls.
Two balls are selected without replacement.
Sequence of colors is noted.
Sol. 1. b b two subexperiments
b w
w b
w w
Tree diagram
P A B P B P A B
I
P A P B A
1 2 1 2 1] ] / ][ [ [B BP B P B P BI
B1 W1
2/5 3/5
W2 B2B2 W2
1/4 3/4 2/4 2/4
3/103/10 3/101/10
tch-prob 30
Let be mutually exclusive, whose union = S ,
partition of S.
“ Theorem on total probability”
Ex. 2.24 In prev. example (2.22) find P[W2]=Pr [second ball is white]
1 2
1 2
)(
...n
n
B BA A S A B
A B A B A B
U U UI I
I U I U U I
1 2...
nP A P A B P A B P A B
I I I
1 1 2 2...
n nP A P B P A B P B P A B P A B P B
2 2 1 1 2 1 13 2 1 3 3
5 5 54 2P W P W B P B P W W P W
1 2, , , nB B B
A
S
B1 B2 Bn
tch-prob 31
Suppose event A occurs, what is the prob. of event Bj?
Bayes’ Rule
Before experiment: P[ Bj ]= a priori probabilityAfter experiment: A occurs P[ Bj/A ]= a posteriori probability
1
j j jj
k k
P A B P A B P BP B A nP A P A B P B
k
I
tch-prob 32
Ex.2.26. Binary symmetric channel
Suppose
0 1
12
A AP P
1 1 0 0 1 1 11 1 112 2 2
P B P B A P A P B A A e e
/ 21 0 00 1 1/ 2
1
1 1
,
1
eP B A P AP A B e
P B
P A B e
0 0
Bj
11
Ai 1-e
e
e
1-e
1-p
p
tch-prob 33
2.5 Independence of Events
If knowledge of the occurrence of an event B does not alter the probability
of some other event A, then A is independent of B.
Two events A and B are independent if
P A BP A B P A
P B
I
P A B P A P B
I
tch-prob 34
Ex. 2.28. A ball is selected from an urn of 4 balls
{1,b},{2,b},{3,w},{4,w}
A: black ball is selected
B: even-numbered ball
C: number >2
P[A]=P[B]=P[C]=0.5
If two events have non zero prob., and are mutually exclusive, then they cannot be independent.
14
P A B P A P B
I
0P A C
I
P A B P A
P A C P A
0 0A B A P BP P I
tch-prob 35
Three events A, B, and C are independent if
Ex. 2.30.
Common application of the independence concept is in making the assumption that the event of separate experiments are independent.
Ex.2.31. A fair coin is tossed three times. P[HHH]=1/8, ….
P A B P A P B
P A C P A P C
P B C P B P C
P A B C P A P B P C
I
I
I
I I
141414
0
B P F
D P F
P B D P B P D
P B F P
P D F P
P B D F P B P D P F
I
I
I
I I
BD
F
F
tch-prob 36
2.6. Sequential Experiments
• Sequence of independent experiments
If sub experiments are independent, and event concerns outcomes of
the kth subexperiment, then are independent.
• Bernoulli trial:
Perform an experiment once, note whether an event A occurs.
success, if A occurs prob= p
failure, otherwise. prob= 1-p
Pn(k)= prob. of k successes in n independent repetitions of a Bernoulli trial = ?
1 2 1 2...
n nA AP A P A P A P A
I I I
kA
1 2, , , nA A A
tch-prob 37
Ex.2.34. toss a coin 3 times. P[ H ]= p
Binomial probability law
k=0,1,…,n
30 1P k p
21 3 1P k p p
22 3 1P k p p 33P k p
1 n kkp pn
k pn k
tch-prob 38
Binomial theorem
Let a=b=1 ,
Let
Ex.2.37. A binary channel with BER . 3 bits are transmitted.
majority decoding is used. What is prob. of error?
0
n nn k n ka b a bkk
20
n nnkk
, 1 ,a p b p 1 10 0
n kkn nn
p p kp nkk k
310
3 32 3 62 .001 .999 .001 3 102 3
P k
tch-prob 39
Multinomial Probability Law
Bj’s: partition of S; mutually exclusive
n independent repetitions of experiment
number of times Bj occurs.
Ex.2.38. dart 3 areas
throw dart 9 times, P[3 on each area]=?
B1 BMB2 S........
j jP B p
1 21
Mp p p
1 21 2 1 21 2
![( , ,..., )] ...! !... !
k k kMM MM
nP k k k p p pk k k
jk
0.2,0.3,0.5p
9! 3 3 33,3,3 .2 .3 .53!3!3!
P
tch-prob 40
Geometric Probability Law
Repeat independent Bernoulli trials until the occurrence of the first success.
P[ more than k trials are required before a success occurs ]
11 ... 1 1 , 1,2,...
1
mp m p p p p p m
m times
111 1
1 111
mp m p p ppmm
1[ ] 1 11
m kp m k p p pm k
P[The initial k trials are failures]
tch-prob 41
Sequence of Dependent Experiments
Ex.2.41. A sequential experiment involves repeatedly drawing a ball from one of two urns, noting the number on the ball, and replacing the ball in its urn.
Urn 0 : 1 ball #1, 2 balls #0
Urn 1 : 5 balls #1, 1 ball #0
Initially, flip a coin. Thereafter, the urn used in the subexperiment corresponds to the number selected in the previous subexperiment.
1 1 1
h
0
1 1
0
0 0
1 1
....
....
0 0 0
t
0S1
S2
S3
S
....
Trellis diagram
Urn 2
Urn 1
tch-prob 42
conditional prob.
Markov property
Ex.2.42 calculate
with
....
0 1 1 1 2 1 0 0
0 1 1 0 0
0 1 2 2 0 1 0 1
2 0 1 1 0 0
2 1 1 0 0,... ...n n n n n
P S S P S S P S
P S S S P S S S P S S
P S S S P S S P S
P S S P S S P S
P S S S P S S P S S P S S P S
0011 1 1 1 0 0 0 0
1 2 11 0 0 0 0 [1]
3 3 2151 1 0 166
P P P P P
P P P P
P P
tch-prob 43
Problem 95 on Page 83
95 Consider a well-shuffled deck of cards consisting of 52 distinct cards, of
which four are aces and four are kings.
a. Pr[Obtaining an ace in the first draw]=
b. Draw a card from the deck and look at it.
Pr[Obtaining an ace in the second draw]=
If does not know the previous outcome
4 152 13
3
51
4
51
A
A
31 12 413 51 13 51
1
13
tch-prob 44
c. Suppose we draw 7 cards from the deck.
Pr[7 cards include 3 aces]=
Pr[7 cards include 2 kings]=
Pr[7 cards includes 3 aces and 2 kings]=
d. Suppose the entire deck of cards is distributed equally among four
players. Pr[each player gets an ace]=
4 48
3 452
7
4 48
2 552
7
4 48 3 36 2 24 134!
1 12 1 12 1 12 1352 39 26 13
13 13 13 13
4!
4 4 44
3 2 252
7
tch-prob 45
Birthday problem.
p =No two people in a group of n people will have a common birthday.
Lotto Problem.
Six numbers and one special number are picked from a pool of 42 numbers. …
1. What is the probability of winning the grand prize?
2. What is the probability of winning 2nd prize?
3. What is the probability that number 37 is drawn next time?
1 2 11 1 1
365 365 365
123
20.01 56
np
p n
p n