PyData NYC 2015 - Automatically Detecting Outliers with Datadog
Review Detecting Outliers. Review Detecting Outliers –Standard Deviation –Percentiles/Box Plots...
-
Upload
amanda-washington -
Category
Documents
-
view
242 -
download
0
Transcript of Review Detecting Outliers. Review Detecting Outliers –Standard Deviation –Percentiles/Box Plots...
Review
Detecting Outliers
Review
Detecting Outliers– Standard Deviation– Percentiles/Box Plots
– Suspected and Highly Suspected Outliers
Review
Detecting Outliers– Standard Deviation– Percentiles/Box Plots
– Suspected and Highly Suspected Outliers
Review
Detecting Outliers– Standard Deviation
• Chebyshev’s Rule• Emperical Rule• Which points are within k standard deviations?• Z-scores• Suspected and Highly Suspected Outliers
Review
Detecting Outliers– Percentiles/Box Plots
• Find Percentiles
• Find Qu, M, QL, IQR. – *** Use the method I showed you, not your calculator***
• Building a box plot– Calculate the Upper/Lower Inner and Outer Fences– *** Use the method I showed you, not your calculator***– Include a menu and show all your work
• Suspected and Highly Suspected Outliers
Examples
Big Picture
Detecting Outliers
Big Picture (Outliers)
Typically we know a lot of historical data about what we are trying to test. From that data we estimate what the population center (the mean) and population standard deviation are. We can:
1) make predictions (within a certain percentage chance) about future events.
2) collect new data and check to see if that would be an outlier in the old data.
Probability
Probability
An experiment is any process that allows researchers to obtain observations.
An event is any collection of results or outcomes of an experiment.
A simple event is an outcome or an event that cannot be broken down any further.
Example
Rolling a die is an experiment. It has 6 different possible outcomes
An example of an event is rolling a 5.
Rolling a 5 is a simple event. It cannot be broken down any further.
Example
Rolling a die is an experiment. It has 6 different possible outcomes.
Another example of an event is rolling an odd number.
This event can be broken down into three simple events: Rolling a 1, rolling a 3 and rolling a 5.
Sample Space
The sample space for an experiment consists of all simple events.
Example: When we roll on die the sample space is: 1, 2, 3, 4, 5, 6
Sample Space
Example: When we roll on die the sample space is: 1, 2, 3, 4, 5, 6
Example: When we roll two dice the sample space is:
Sample SpaceExample: When we roll two dice the sample space is all possible pairs of rolls
1,1 1,2 1,3 1,4 1,5 1,6
2,1 2,2 2,3 2,4 2,5 2,6
3,1 3,2 3,3 3,4 3,5 3,6
4,1 4,2 4,3 4,4 4,5 4,6
5,1 5,2 5,3 5,4 5,5 5,6
6,1 6,2 6,3 6,4 6,5 6,6
Sample Space
Sample Space
Event
Simple Events (all the red dots)
We often represent the sample space with a Venn Diagram.
Sample Space
Sample Space
Event
Simple Events
Usually the simple events are not included in our diagram
Sample Space
Here is a Venn Diagram depicting two events which overlap, or intersect.
Assigning Probabilities
Sample Space
Event
Simple Events (all the red dots)
Assigning Probabilities
Each Simple event has a probability associated with it.
trialsofnumber
successes possible ofnumber )( sP
Assigning Probabilities
Each Simple event has a probability associated with it.
This is really the relative frequency of the simple event.
occurcan points sample theall waysofnumber
occurcan s waysofnumber )( sP
Assigning Probabilities
Each Simple event has a probability associated with it.
This is really the relative frequency of the simple event.
To find the probability of an event, add up the probabilities of the simple events inside of it.
Example
A cage contains 7 black mice, 4 brown mice and 1 white mouse. A mouse is selected at random from the cage. What is the probability it is either a black mouse or a white mouse?
Frequency
Black 7
Brown 4
White 1
Total 12
Example
A cage contains 7 black mice, 4 brown mice and 1 white mouse. A mouse is selected at random from the cage. What is the probability it is either a black mouse or a white mouse?
Frequency Probability
Black 7 7/12
Brown 4 4/12
White 1 1/12
Total 12 1
Example
A cage contains 7 black mice, 4 brown mice and 1 white mouse. A mouse is selected at random from the cage. What is the probability it is either a black mouse or a white mouse?
P (Black or White) =
Example
A cage contains 7 black mice, 4 brown mice and 1 white mouse. A mouse is selected at random from the cage. What is the probability it is either a black mouse or a white mouse?
P (Black or White) = P(Black) +P(White)
Example
A cage contains 7 black mice, 4 brown mice and 1 white mouse. A mouse is selected at random from the cage. What is the probability it is either a black mouse or a white mouse?
P (Black or White) = P(Black) +P(White)
= 7/12 + 1/12 = 8/12
Sample SpaceExample: Roll two dice. What is the probability of rolling a 9?
1,1 1,2 1,3 1,4 1,5 1,6
2,1 2,2 2,3 2,4 2,5 2,6
3,1 3,2 3,3 3,4 3,5 3,6
4,1 4,2 4,3 4,4 4,5 4,6
5,1 5,2 5,3 5,4 5,5 5,6
6,1 6,2 6,3 6,4 6,5 6,6
Sample Space
Example: Roll two dice. What is the probability of rolling a 9?
)3,6()4,5()5,4()6,3()9( PPPPP
Sample Space
Example: Roll two dice. What is the probability of rolling a 9?
9
1
36
4
36
1
36
1
36
1
36
1
)3,6()4,5()5,4()6,3()9(
PPPPP
Properties of Probability
1)()()()( (2)
then},,,{
events simple of up made is space sample theIf
1)(0 (1)
satisfies sevent simple a ofy Probabilit The
321
321
n
n
sPsPsPsP
ssss
sP
Union
The union of events A and B is the event that A or B (or both) occur.
Union
The union of events A and B is the event that A or B (or both) occur.
AB
A or B
Intersection
The intersection of events A and B is the event that both A and B occur.
Intersection
The intersection of events A and B is the event that both A and B occur.
A B
A and B
Compliment
The compliment of an event A is the event that A does not occur.
Compliment
The compliment of an event A is the event that A does not occur.
AC
Compliment
The compliment of an event A is the event that A does not occur.
We use AC to denote the compliment of A.
P(AC)= 1 - P(A)
Compliment
The compliment of an event A is the event that A does not occur.
We use AC to denote the compliment of A.
P(A)= 1 - P(AC)
ExampleFor an experiment of randomly selecting one card from
a deck of 52 cards, letA=event the card selected is the King of Hearts
B=event the card selected is a King
C=event the card selected is a Heart
D=event the card selected is a face card.
Find:
a) P(DC) b) P(B and C)
c) P(B or C) d) P(C and D)
e) P(A or B) f) P(B)
ExampleFor an experiment of randomly selecting one card from
a deck of 52 cards, letA=event the card selected is the King of Hearts
B=event the card selected is a King
C=event the card selected is a Heart
D=event the card selected is a face card.
Find:
a) P(DC) =40/52 b) P(B and C)= 1/52
c) P(B or C)=16/52 d) P(C and D)=3/52
e) P(A or B)=4/52 f) P(B)=4/52
Unions and Intersections
Unions and Intersections are related by the following formulas
P(A and B)= P(A) + P(B) - P(A or B)
P(A or B)= P(A) + P(B) - P(A and B)
Mutually Exclusive
Two events are mutually exclusive if
P (A and B) = 0.
Mutually Exclusive
Two events are mutually exclusive if
P (A and B) = 0.
Suppose P (E) = .3, P (F) = .5, and E and F are mutually exclusive. Find:
P(E and F)= P(E or F)=
P(EC)= P(FC)=
P((E or F) C)= P((E and F) C)=
Mutually Exclusive
Two events are mutually exclusive if
P (A and B) = 0.
Suppose P (E) = .3, P (F) = .5, and E and F are mutually exclusive. Find:
P(E and F) = 0 P(E or F) = 0.8
P(EC) = 0.7 P(FC) = 0.5
P((E or F) C)=0.2 P((E and F) C)=1
Example
In buying a new computer (tower, monitor, keyboard and mouse) studies show that 4% have problems with their mouse and 2% have problems with their monitor and 0.2% have problems with both before the expirations of their manufactured warranty.a) Find the probability that a computer set purchased has one of the two problemsb) Neither c) Just a monitor problem
Example
In buying a new computer (tower, monitor, keyboard and mouse) studies show that 4% have problems with their mouse and 2% have problems with their monitor and 0.2% have problems with both before the expirations of their manufactured warranty.a) Find the probability that a computer set purchased has one of the two problems. (5.8%) b) Neitherc) Just a monitor problem
Example
In buying a new computer (tower, monitor, keyboard and mouse) studies show that 4% have problems with their mouse and 2% have problems with their monitor and 0.2% have problems with both before the expirations of their manufactured warranty.
a) Find the probability that a computer set purchased has one of the two problems. (5.8%)
b) Neither (94.2%)
c) Just a monitor problem (1.8%)
Example
In buying a new computer (tower, monitor, keyboard and mouse) studies show that 4% have problems with their mouse and 2% have problems with their monitor and 0.2% have problems with both before the expirations of their manufactured warranty.
a) Find the probability that a computer set purchased has one of the two problems. (5.8%)
b) Neither (94.2%)
c) Just a monitor problem (1.8%)
Example
Suppose P (E) = 0.4, P (F) = 0.3, and
P(E or F)=0.6. Find:
P(E and F) P(EC or FC)
P(EC and FC) P(EC and F)
Example
Suppose P (E) = 0.4, P (F) = 0.3, and
P(E or F)=0.6. Find:
P(E and F) =0.1 P(EC or FC)
P(EC and FC) P(EC and F)
Example
Suppose P (E) = 0.4, P (F) = 0.3, and
P(E or F)=0.6. Find:
P(E and F) =0.1 P(EC or FC)=0.9
P(EC and FC) P(EC and F)
Example
Suppose P (E) = 0.4, P (F) = 0.3, and
P(E or F)=0.6. Find:
P(E and F) =0.1 P(EC or FC)=0.9
P(EC and FC)=0.4 P(EC and F)
Example
Suppose P (E) = 0.4, P (F) = 0.3, and
P(E or F)=0.6. Find:
P(E and F) =0.1 P(EC or FC)=0.9
P(EC and FC)=0.4 P(EC and F)= 0.2
Example
Suppose P (E) = 0.4, P (F) = 0.3, and
P(E or F)=0.6. Find:
P(E and F) =0.1 P(EC or FC)=0.9
P(EC and FC)=0.4 P(EC and F)
Review
Probabilities– Definitions of experiment, event, simple event,
sample space, probabilities, intersection, union compliment
– Finding Probabilities– Drawing Venn Diagrams – If A and B are two events then
P(A or B) = P(A) + P(B) - P(A and B),
P(not A) = 1 - P(A). – Two events A and B are mutually exclusive if P(A and
B) = 0.
57
Homework
• Finish reading Chapter 3.1-3.7
• Assignment 1 due Thursday
• Quiz next Tuesday on Chapters 1 and 2
• Problems on next slide
58
Problems
• 3.15, 3.20,
• 3.44, 3.45, 3.54