Outline of the course

Chapter 2 : Probability

Chapter 3 and 4 : Univariate random vari-

ables and their probability distribution

Chapter 5 : Multivariate random variables

and their probability distribution

Chapter 6 : Functions of random variables

Chapter 7 : Sampling distributions and the

Central Limit Theorem

1

[Chapter 2. Probability]

2.0 Introduction

2.1 Interpretation of probability

2.2 Probability and Inference

2.3 Set notation

2.4 Probabilistic model

2.5 Probability of an event: sample-point method

2.6 Tools for Counting sample points

2.7 Conditional probability

2.8 Two laws of probability

2.9 Probability of an event: event-composition

method

2.10 Bayes’ rule

2

2.0 Introduction

• Definition of Statistics

- common elements in many definitions of

statistics :

A) selection of a subset of a large collection

of data

B) inference of the characteristics of the

complete set

- Key element A) : a subset of a large col-

lection of data

· Population: the large collection of data

that is the target of our interest

· Sample: subset selected from a popu-

lation

E.G.) Suppose that two candidates are

running for a public office in our com-

munity and that we wish to determine

whether our candidate, Jones, is favored

to win. 20 egligible voters are randomly

selected from the courthouse roster of

3

voters and contacted for their opinion.

Population : all eligible voters

Sample : 20 voters sampled

- Key element B) : inference of the charac-

teristics of the complete set

· objective of statistics: make an infer-

ence about a population based on infor-

mation contained in a sample from that

population and to provide an associated

measure of goodness for the inference.

E.G.) one wants to

i) estimate the true fraction of all eligi-

ble voters who favor Jones

ii) know how the estimate obtained is

close to the true fraction

iii) conclude about the Jones’s prospects

for winning the election.

2.1 - 2.2 Probability and Inference

• An inference example: suppose all 20 vot-

ers collected favor Jones. What do you

conclude about Jones acquiring more than

50% of the votes?

Answer: If less than 50% of the voters fa-

vor Jones, the chance of drawing 20 out of

20 favoring Jones is slim.

Need to calculate the probability of all 20

favoring Jones, assuming that the percent

of population favoring Jones is p and p < .5.

• Another inference example: A gambler who

wishes to make an inference concerning the

balance of a die. He tossed the die 10 times

in order to test the hypothesis that the die

is balanced.· Population of our interest : set of num-

bers that would be generated if the die

were rolled over and over again. For

perfectly balanced die, what is the pop-

ulation like?

- All ten tosses result in 3s. What is

4

your conclusion? Why?

Need to calculate the probability of 10

3s, assuming a balanced die.

- Four 2s along with two 1s, two 4s, one

5 and one 6. Is this result so improba-

ble that we should reject the hypothesis

that the die is balanced and conclude

that the die is loaded in favor of 2s?

Need to calculate the probability of 4 2s,

2 1s, 2 4s, one 5 and one 6, assuming

a balance die

- Four 2s. Is this result so improba-

ble that we should reject the hypothesis

that the die is balanced and conclude

that the die is loaded in favor of 2s?

Need to calculate the probability of 4 2s

in 10 tosses, assuming a balance die

• Probabilists assume that they know the structureof the population of interest and use the theory ofprobability to compute the probability of obtaininga particular sample

• Statisticians use probability to make inferences aboutthe population based on the observed sample.

2.3 Set notation (using Venn diagrams)

• A, B, C, . . . : sets of points.

• A = {a1, a2, . . . , an} : the elements in the

set A are a1, a2, . . . , an.

• A ⊂ B : A is the subset of B

• φ : null set. a subset of every set.

• S : the universal set(the set of all elements

under consideration)

• A∪B : Union of A and B={x;x ∈ A or x ∈

B}

• A∩B(or AB) : Intersection of A and B={x;x ∈

A and x ∈ B}

• A under A ⊂ S : Complement of A =

{x;x is in S, but not in A}, (Q) A∪A =

• A∩B = φ : A and B are disjoint or mutually

exclusive. (Q) A ∩ A =

5

• A ∪ B = S : A and B are exhaustive

• A and B are mutually exclusive and exhaus-

tive if .

[Generalization to k sets, Ai, i = 1, . . . , k].

· A1, . . . , Ak are mutually exclusive : for i 6= j

· A1, . . . , Ak are exhaustive :

· A1, . . . , Ak are mutually exclusive and exhaustive :

• Distributive laws :

A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C),

A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C)

• DeMorgan’s laws :

(A ∪ B) = A ∩ B, (A ∩ B) = A ∪ B

(E.G.) S = {1,2,3,4,5,6}. Let A = {1,3}, B = {1,4}and C = {2,3,5}.

(1) A ∪ B = {1}, (2) B ∩ C = φ,

(3) C = {1,4,6}, (4) Are A, B and C mutually exclusive

or exhaustive?

6

2.4 Probabilistic modelProbability : a measure of one’s belief in the

occurrence of a future random event that can

not be predicted with certainty.

(Def 2.1) An experiment is the process by which

an observation is made.

(Def 2.2) The events are the outcomes of an

experiment.

• Simple event : an event that can not be

decomposed and happens only in one way.

• Compound event : an event which can be

decomposed into other events and can hap-

pen more than one distinct way

• Sets are collections of points. So, we asso-

ciate a distinct point, called a sample point,

with each and every simple event associ-

ated with an experiment.

• Each simple event corresponds to one and

only one sample point. The letter E with a

subscript represents a simple event or the

corresponding sample point.7

(Def 2.3) A sample space S associated with an

experiment is the set consisting of all possible

sample points(or the collection of all possible

distinct outcomes).

(Example: the die tossing experiment) Sup-

pose we observe the following outcomes/events

in an experiment of a single toss of a balanced

die:

A : Observe an even number.

E1 : Observe a 1. E2 : Observe a 2.

E3 : Observe a 3. E4 : Observe a 4.

E5 : Observe a 5. E6 : Observe a 6.

Q1) What is Ei, i = 1, . . . ,6?

Q2) What is A?

Q3) What is S?

8

(Def 2.4) An discrete sample space is one that

contains either a finite or a countable number

of distinct sample points.

· observe one and only one simple event

in an experiment conducted a single time.

So, all distinct simple events are mutually

exclusive events.

(e.g.) In the die tossing experiment,

· compound events = unions of the sets of

single sample point corresponding to the

suitable simple events.

(e.g.) In the die tossing experiment, A =

E2 ∪ E4 ∪ E6

(Def 2.5) An event in a discrete sample space

S is a collection of sample points

· any event in a discrete sample space S is

a subset of S

(e.g.) In the die tossing experiment, A S

9

(Def 2.6) [Definition of probability] Suppose S

is a sample space associated with an experi-

ment. To every event A in S(A is a subset

of S) we assign a number, P(A), called the

probability of A, so that the following axioms

hold:

Axiom 1 : 0 ≤ P(A) ≤ 1.

Axiom 2 : P(S) = 1.

Axiom 3 : If A1, A2, A3, . . . form a sequence

of pairwise mutually exclusive events in S

(that is, Ai ∩ Aj = φ if i 6= j), then

P(A1 ∪ A2 ∪ A3 ∪ · · · ) =∞∑

i=1

P(Ai).

(Def 2.6) states only the conditions that an as-

signment of probabilities must satisfy; it does

not tell us how to assign specific probabilities

to events.

10

How can one assign probabilities to some events,

say A for discrete sample space?

• assign probabilities to each simple event,

Ei. Here, the assigned probability is the

value presenting the proportion of times Ei

is expected to occur when the experiment

is repeated under identical conditions. The

assignment can be based on assumptions,

hyptheses, and relative frequency of past

occurrence.

If a balanced die is used, P(Ei) = for

i = 1, . . . ,6.

• If one can list all simple events in A, then

calculate the probability of A, P(A) :

P(A) =∑

i such that Ei∈A

P(Ei)

(Revisted the die tossing experiment in p.11)

For A = {E2, E4, E6}, P(A) is .

11

(Example 2.1) A manufacturer has five seem-

ingly identical computer terminals available for

shipping. Unknown to her, two of the five are

defective. A particular order calls for two of

the terminals and is filled by randomly select-

ing two of the five that are available.

(Q1) List the sample space for this experiment.

(Q2) Let A denote the event that the order is

filled with two nondefective terminals. List the

sample points in A.

(Q3) Construct a Venn diagram for the exper-

iment that illustrates event A.

(Q4) Assign probabilities to the simple events

in such a way that the information about the

experiment is used and the axioms in (Def 2.6)

are met.

(Q5) Find the probability of event A.

12

2.5 Calculation of the probability of anevent

Sample-point method : find the probability of

an event defined on a discrete sample space

1. Define the experiment and clearly deter-

mine how to describe one simple event.

2. List the simple events associated with the

experiment and test each to make certain

that it cannot be decomposed. This de-

fines the sample space S.

3. Assign reasonable probabilities to the sam-

ple points in S, making certain that P(Ei) ≥

0 and∑

i P(Ei) = 1.

4. Define the event of interest, A, as a spe-

cific collection of sample points.(A sample

point is in A if A occurs when the sample

point occurs. Test all sample points in S

to identify those in A.)

5. Find P(A) by summing the probabilities of

the sample points in A.

13

(Example: equiprobable simple event/sample

point) A balanced coin is tossed three times.

Calculate the probability that exactly two of

the three tosses result in heads.

(Example: simple event/sample point with un-

equal probability) The odds are two to one

that, when A and B play tennis, A wins. Sup-

pose that A and B play two matches. What is

the probability that A wins at least one match?

14

2.6 Tools for Counting sample points

(Example) If S contains N equiprobable sam-

ple points and an event A contains exactly na

sample points, P(A) = na/N .

Counting the elements in A and S becomes

vitally important!

But, when the number of simple events in S is

too large, manual enumeration of every sample

point in S is tedious or even impossible.

Need combinatorics(counting rules)!!

• Multiplication principle of counting : mn

rule and its generalization

• Permutation

• Combination and its extension

15

(Example) Roll a pair of dice. How many sam-

ple points in S, the sample space?

(Theorem 2.1)(mn rule and its generalization)

i) With m elements a1, a2, . . . , am and n ele-

ments b1, b2, . . . , bn it is possible to form mn =

m × n pairs containing one element from each

group.

ii) If R groups are such that each group may

contains nr elements where r = 1, . . . , R, it is

possible to form n1·n2 · · ·nR possible outcomes

from R groups.

(Example)What is the probability that you draw

2 aces when you draw two cards from the deck?

(example) How many subsets of a set of 10

elements?

16

(Example) LotteryFrom the numbers 1,2,. . .,44, a person may pick anysix for her ticket. The winning number is then decidedby randomly selecting six numbers from the forty-four.How many possible lottery tickets?

1) Ordered, without replacement

:

2)Ordered, with replacement

:

3) Unordered, without replacement

:

• Two important factors in counting rules

· Ordering : ordered/unordered

· Sampling : with/without replacement

so, the number of possible arrangements of size r fromn objects are

With WithoutReplacement Replacement

Ordered nr Permutation(P nr )

Unordered Combination(Cnr )

17

(Definition 2.7)(permutation)

i) A permutation of n objects is a reorderingof them

• a series of n mini-experiments : choose 1st ob-ject then choose 2nd object then · · ·

• there are n×(n−1)×(n−2)×· · ·2×1 permutations(without replacement)

• n! = n × (n − 1) × (n − 2) × · · ·2 × 1

• 0! = 1

(example) How many ways can the letters in MIXUPbe mixed up?

ii) The number of permutations of n distinctobjects taken r at a time

• the number of ordered lists of r chosen from n ob-jects (without replacement)

• only care about the first r in a permutation of nobjects

• there are n × (n − 1) × (n − 2) × · · · × (n − r + 1)permutations

• P nr = n!

(n−r)!= n× (n− 1)× (n− 2)× · · · × (n− r +1)

(example) From 10 persons, choose a president, avice president, a secretary and a treasurer for a club.How many possible results?

18

(Definition 2.8)(combination)

The number of combinations of n objects takenr at a time

• the number of unordered sets of r chosen from nobjects (without replacement)

• there are r ! different orderings in P nr

• Cnr =

(

nr

)

=P n

r

r!= n!

r!(n−r)!= n!

(n−r)!r!= Cn

n−r

(example) How many possible choices for choosing5 students out of 20?

(example) Let A denote the event that exactly oneof the two best applicants appears in a selection oftwo out of five. Find the number of sample pointsin A and P (A).

(example) There are 10 balls, of which 3 are basketballs, and 7 are footballs. If we want to put themin a row, how many possible arrangement?

19

Example: Toss a pair of dice. What is the

probability of observing a sum of 7?

Example: Record the birthdays of 20 ran-

domly slected persons. What is the proba-

bility that each person has a different birth-

day? Ignore leap years.

Example:A company orders supplies from

M distributors and wishes to place n or-

ders (n ¡ M). Assume that the company

places the orders in a manner that allows

every distributor an equal chance of obtain-

ing any one order and there is no restriction

on the number of orders that can be palced

with any distributor.

Find the probability that all of the orders

go to different distributors.

Find the probability that a particular dis-

tributor - say, distributor I - gets exactly

k orders (k ≤ n)

20

A student prepares for an exam by study-

ing a list of 10 problems. She can solve

six of them. For the exam, the instructor

selects five problems for the ten. What is

the probability that the student can solve

all five on the exam?

A child mixes ten good and three dead bat-

teries. To find the dead batteries, his fa-

ther tests them one by one and without

replacement. What is the probability that

his father find all three dead batteries at

the fifth test?

The combination of n objects taken r at a

time is the number of ways in which one can

partition a set of n distinct objects into two

nonoverlapping groups(first r and the remain-

ing n − r).

Then how many number of ways to partition a

set of n distinct objects into k nonoverlapping

groups.

(Theorem 2.3) The number of ways of parti-

tioning n distinct objects into k distinct groups

containing n1, n2, . . . , nk objects, respectively,

where each object appears in exactly one and∑k

i=1 ni = n, is

( n

n1n2 · · ·nk

)

=n!

n1!n2! · · ·nk!.

21

(example) A labor disputes has arisen concern-

ing the distribution of 20 laborers to four differ-

ent construction jobs. The first job required 6

laborers; the second, third, and fourth utilized

4,5, and 5 laborers, respectively.

Determine the number of ways the 20 la-

borers can be divided into groups of the

appropriate sizes to fill all of the jobs.

What is the probability that an ethnic group

of 4 is all assigned to the first job?

(example) There are 10 balls, of which 3 basket

balls, 2 footballs, 3 are volleyballs and 2 soccer

balls put all balls in a row. How many possible

arrangement?

(example) How many distinguishable reorder-

ings of the letters MISSISSIPPI?

22

2.7 Conditional probability

(Example) The probability of a 1 in the toss

of one balanced die is 1/6.

One has new information that an odd number

has fallen. Then probability of a 1 is 1/3.

The probability of an event will sometimes de-

pend on whether we know that other events

have occurred.

• Unconditional probability of an event:

• Conditional probability of an event: the

probability of the event given the fact that

one or more events have already occurred.

(Example) The unconditional probability of a

1 in the toss of one balanced die is 1/6.

If we know that an even number has fallen, the

conditional probability of the occurrence of a

1 is .

23

(Def 2.9) The conditional probability of an

event A, given that an event B has occurred

(simply probability of A given B), is equal to

P(A|B) =P(A ∩ B)

P(B)

provided P(B) > 0. P(A|B) is read “probability

of A given B”.

• P(A ∩ B) = P(B)P(A|B)

• P(A ∩ B) = P(A)P(B|A)

(Example 2.14) Suppose that a balanced die is

tossed once.

1) Use (Def 2.9) to find the probability of a 1,

given that an odd number was obtained.

2) Use (Def 2.9) to find the probability of a 4,

given that an odd number was obtained.

24

(Example) In a class, there 44 Junior and Se-

nior students, 30 are men and 40 Junior of

whom 12 are women. If a student is randomly

selected from this class,

(a) What is the probability that the selected

student is a Junior male student?

(b) If the selected student is female, what is

the probability that she is a Senior?

(Example) Four cards are dealt from a well-

shuffled deck. Calculate the probability of 4

aces given that 2 aces have already been drawn

in the first two cards.

25

If knowing that the event B occurs has no in-

fluence on P(A), we would say that events A

and B are independent.

(Def 2.10) Two events A and B are said to be

independent, if any one of the following holds:

P(A|B) = P(A),

P(B|A) = P(B),

P(A ∩ B) = P(A)P(B).

Otherwise, the events are said to be depen-

dent.

(Warnings)

• Independence is not the same as mutual exclusivity

• Venn diagrams are no good for describing indepen-dence

(Example) Consider the following events in the

toss of a single die:

A : Observe an odd number.

B : Observe an even number.

C : Observe 1 or 2.

a) Are A and B independent events?

b) Are A and C independent events?

26

(Example) Two balls are taken from a box of

3 red and 7 black balls

A : the first ball is red and B : the second ball

is red.

Are A and B independent events if

a) the balls are drawn with replacement?

b)the balls are drawn without replacement?

(Example) A red fair dice and a white fair dice

are rolled:

A : 4 on the red die.

B : two faces equal.

C : Sum of dice is 10.

a) Are A and B independent events?

b) Are A and C independent events?

(Theorem)

If A and B are independent, then the following

pairs of events are also independent.

(a)A and B, (b)A and B, (c)A and B.

(Proof) (a)

27

Extension of (Def 2.10) Three events A, B

and C are said to be mutually independent, if

and only if the following two conditions hold:

(a) They are pairwise independent :

P(A ∩ B) = P(A)P(B),

P(A ∩ C) = P(A)P(C),

P(B ∩ C) = P(B)P(C),

(b) P(A ∩ B ∩ C) = P(A)P(B)P(C).

Otherwise, the events are said to be depen-

dent.

(Example) An urn contains 3 red, 2 white and

4 yellow balls. An ordered sample 3 is drawn

from the urn with replacement. Find the prob-

ability of the sequence RWY(i.e., R in 1st, W

in 2nd and Y in 3rd).

A : R in 1st.

B : W in 2nd.

C : Y in 3rd.

Are A, B and C mutually independent?

29

(Example) Consider tossing a fair coin three

times and consider the following three events:

A : Number of heads is even.

B : The first two flips are the same

C : The second two flips are heads.

Are A, B and C mutually independent?

(Example) An experiment consists of tossing

different balanced dice, white and black. The

sample space S of the outcomes consists of all

ordered pairs (i, j)(i = 1, . . . ,6, j = 1, . . . ,6):

S = (1,1), (1,2), . . . , (1,6), . . . , (6,1), (6,2), . . . , (6,6).

Define the following events:

E1 = First die is 1,2, or 3,

E2 = First die is 3,4, or 5,

E3 = Sum of the faces is 9.

Are E1, E2 and E3 mutually independent?

30

Generally, many experiments consists of a se-

quence of n trials that are mutually indepen-

dent. If the outcomes of the trials do not

have anything to do with one another, then

events, such that each is associated with a dif-

ferent trial, should be independent in probabil-

ity sense. That is if Ai is associated with the

ith trial, i = 1, . . . , n, then

P(A1 ∩ A2 ∩ · · · ∩ An) = P(A1) ∩ P(A2) · · ·P(An),

P(A1 ∩ A2 ∩ · · · ∩ An) = P(A1) ∩ P(A2) · · ·P(An).

(Example) A fair six-sided die is rolled 6 in-

dependent times. Define an event Ai as fol-

lows: Ai = side i is observed on the i-th roll ,

i = 1, . . . ,6. What is the probability that none

of Ai occurs?

31

2.8 Two laws of probability

(Theorem 2.5) The probability of the intersec-

tion of two events A and B is

P(A ∩ B) = P(A)P(B|A) = P(B)P(A|B)

If A and B are independent, then P(A ∩ B) =

P(A)P(B).

Its extension: for any k event, P (A1∩A2∩· · ·Ak) =

P (A1)P (A2|A1)P (A3|A1∩A2) · · ·P (Ak|A1∩A2∩· · ·∩Ak−1).

(Theorem 2.6) The probability of the union of

two events A and B is

P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

If A and B are mutually exclusive,

P(A ∩ B) = 0 and P(A ∪ B) = P(A) + P(B).

Its extension for three events: P(A ∪ B ∪ C) =

P(A) + P(B) + P(C) − P(A ∩ B) − P(B ∩ C)−

P(A ∩ C) + P(A ∩ B ∩ C)

32

(Theorem 2.7 If A is an event, then

P(A) = 1 − P(A)

Sometimes it is easier to calculate P(A) than

to calculate P(A). In such cases, it is easier to

find P(A) by the relationship P(A) = 1−P(A)

than to find P(A) directly.

(Exercise 2.94)

(Exercise 2.95)

(Exercise 2.104)

33

2.9 Calculating the Probability of anevent

The event-composition method : calculate the

probability of an event(defined on a discrete

sample space), A, expresses A, as a compo-

sition involving unions and/or intersections of

other events. The laws of probability are then

applied to find P(A).

A summary of the steps follows:

1. Define the experiment.

2. Visualize the nature of the sample points.

Identify a few to clarify your thinking.

3. Write an equation expressing the event of

interest, say, A, as a composition of two

or more events, using unions, intersections,

and/or complements. Make certain that

event A and the event implied by the com-

position represent the same set of sample

points.

4. Apply the additive and multiplicative laws

of probability to the compositions obtained

in step 3 to find P(A).

34

(Examples) It is known that a patient with a

disease will respond to treatment with proba-

bility equal to 0.9. If three patients with dis-

ease are treated and respond independently,

find the probability that at least one will re-

spond.

(Exercise 2.111) An advertising agency notices

that approximately 1 in 50 potential buyers of

a product sees a given magazine ad, and 1 in

5 sees a corresponding ad on television. One

in 100 see both. One in 3 actually purchases

the product after seeing the ad, 1 in 10 with-

out seeing it. What is the probability that a

randomly selected potential customer will pur-

chase the product?

35

2.10 Bayes’ Rule (Def 2.11) For some pos-

itive integer k, let the sets B1, B2, . . . , Bk be

such that

1. S = B1 ∪ B2 ∪ · · ·Bk.

2. Bi ∩ Bj = φ for i 6= j.

Note that Bi, i = 1, . . . , k are mutually exclusive

and exhaustive. Then the collection of sets

{B1, B2, . . . , Bk} is said to be a partition of S.

If A is any subset of S and {B1, B2, . . . , Bk}

is a partition of S, A can be decomposed as

follows:

A =

36

(Theorem 2.8 : Law of Total Probability)

Assume that {B1, B2, . . . , Bk} is a partition of

S such that P(Bi) > 0 for i = 1,2, . . . , k. Then

for any event A

P(A) =k

∑

i=1

P(A ∩ Bi) =k

∑

i=1

P(A|Bi)P(Bi)

(Theorem 2.9)[Bayes’ Rule]

Assume that {B1, B2, . . . , Bk} is a partition of

S such that P(Bi) > 0 for i = 1,2, . . . , k. Then

P(Bj|A) =P(Bj ∩ A)

P(A)=

P(A|Bj)P(Bj)∑k

i=1 P(A|Bi)P(Bi)

Note how Bayes’ rule

• switches the order of conditioning

• updates the probability of Bj for given A based onthe (prior) information P (Bi) and P (A|Bi)

37

(Exercise 2.124) A population of voters con-

tains 40% Republicans and 60% Democrats.

It is reported that 30% of the Republicans and

70% of the Democrats favor an election issue.

A person chosen at random from this popu-

lation is found to favor the issue in question.

Find the conditional probability that this per-

son is a Democrat.

(Example) Suppose you have been tested pos-

itive for a disease. What is the probability that

you actually have the disease? This probability

depends on the accuracy and sensitivity of the

test, and on the (prior) information on the dis-

ease. Let the false negative rate and the false

positive rate of this test be 5% and 5%, re-

spectively. Suppose the disease is rare in that

the probability that one has the disease is 1%.

38

(Example) A package, say P1, of 24 balls, con-

tains 8 green, 8 white, and 8 purple balls. A

Package, say P2, of 24 balls contains 6 are

green, 6 white and 12 purple balls. One of the

two packages is selected at random.

(a) If 3 balls from this package were selected,

all 3 are purple. Compute the conditional prob-

ability that package P2 was selected.

(b) If 3 balls from this package were selected,

they are 1 green, 1 white, and 1 purple ball.

Compute the conditional probability that pack-

age P2 was selected.

39