Type I and type II errors

This article is about erroneous outcomes of statisticaltests. For related, but non-synonymous terms in binaryclassification and testing generally, see false positivesand false negatives.

In statistical hypothesis testing, a type I error is the in-correct rejection of a true null hypothesis (a “false posi-tive”), while a type II error is the failure to reject a falsenull hypothesis (a “false negative”). More simply stated, atype I error is detecting an effect that is not present, whilea type II error is failing to detect an effect that is present.The terms “type I error” and “type II error” are oftenused interchangeably with the general notion of false pos-itives and false negatives in binary classification, such asmedical testing, but narrowly speaking refer specificallyto statistical hypothesis testing in the Neyman–Pearsonframework, as discussed in this article.

1 Definition

In statistics, a null hypothesis is a statement that one seeksto nullify with evidence to the contrary. Most commonlyit is a statement that the phenomenon being studied pro-duces no effect or makes no difference. An example ofa null hypothesis is the statement “This diet has no effecton people’s weight.” Usually an experimenter frames anull hypothesis with the intent of rejecting it: that is, in-tending to run an experiment which produces data thatshows that the phenomenon under study does make adifference.[1] In some cases there is a specific alternativehypothesis that is opposed to the null hypothesis, in othercases the alternative hypothesis is not explicitly stated, oris simply “the null hypothesis is false” – in either eventthis is a binary judgment, but the interpretation differsand is a matter of significant dispute in statistics.A type I error (or error of the first kind) is the incor-rect rejection of a true null hypothesis. Usually a type Ierror leads one to conclude that a supposed effect or rela-tionship exists when in fact it doesn't. Examples of type Ierrors include a test that shows a patient to have a diseasewhen in fact the patient does not have the disease, a firealarm going off indicating a fire when in fact there is nofire, or an experiment indicating that a medical treatmentshould cure a disease when in fact it does not.A type II error (or error of the second kind) is thefailure to reject a false null hypothesis. Examples of typeII errors would be a blood test failing to detect the disease

it was designed to detect, in a patient who really has thedisease; a fire breaking out and the fire alarm does notring; or a clinical trial of a medical treatment failing toshow that the treatment works when really it does.[2]

In terms of false positives and false negatives, a positiveresult corresponds to rejecting the null hypothesis (or in-stead choosing the alternative hypothesis, if one exists),while a negative result corresponds to failing to rejectthe null hypothesis (or choosing the null hypothesis, ifphrased as a binary decision); roughly “positive = alter-native, negative = null”, or in some cases “positive = null,negative = alternative”, depending on the situation & re-quirements, though exact interpretation differs. In theseterms, a type I error is a false positive (incorrectly choos-ing alternative hypothesis instead of null hypothesis), anda type II error is a false negative (incorrectly choosing thenull hypothesis instead of the alternative hypothesis).When comparing two means, concluding the means weredifferent when in reality they were not different would bea Type I error; concluding the means were not differentwhen in reality they were different would be a Type IIerror. Various extensions have been suggested as "TypeIII errors", though none have wide use.All statistical hypothesis tests have a probability of mak-ing type I and type II errors. For example, all blood testsfor a disease will falsely detect the disease in some pro-portion of people who don't have it, and will fail to detectthe disease in some proportion of people who do have it.A test’s probability of making a type I error is denotedby α. A test’s probability of making a type II error is de-noted by β. These error rates are traded off against eachother: for any given sample set, the effort to reduce onetype of error generally results in increasing the other typeof error. For a given test, the only way to reduce both er-ror rates is to increase the sample size, and this may notbe feasible.These terms are also used in a more general way by socialscientists and others to refer to flaws in reasoning.[3] Thisarticle is specifically devoted to the statistical meaningsof those terms and the technical issues of the statisticalerrors that those terms describe.

2 Statistical test theory

In statistical test theory the notion of statistical error isan integral part of hypothesis testing. The test requires anunambiguous statement of a null hypothesis, which usu-

1

2 3 EXAMPLES

ally corresponds to a default “state of nature”, for exam-ple “this person is healthy”, “this accused is not guilty” or“this product is not broken”. An alternative hypothesis isthe negation of null hypothesis, for example, “this personis not healthy”, “this accused is guilty” or “this product isbroken”. The result of the test may be negative, relativeto null hypothesis (not healthy, guilty, broken) or positive(healthy, not guilty, not broken). If the result of the testcorresponds with reality, then a correct decision has beenmade. However, if the result of the test does not corre-spond with reality, then an error has occurred. Due tothe statistical nature of a test, the result is never, exceptin very rare cases, free of error. Two types of error aredistinguished: type I error and type II error.

2.1 Type I error

A type I error, also known as an error of the first kind,occurs when the null hypothesis (H0) is true, but is re-jected. It is asserting something that is absent, a falsehit. A type I error may be compared with a so-calledfalse positive (a result that indicates that a given conditionis present when it actually is not present) in tests where asingle condition is tested for.The type I error rate or significance level is the probabil-ity of rejecting the null hypothesis given that it is true.[4][5]It is denoted by the Greek letter α (alpha) and is alsocalled the alpha level. By convention, the significancelevel is set to 0.05 (5%), implying that it is acceptableto have a 5% probability of incorrectly rejecting the nullhypothesis.[4]

Type I errors are philosophically a focus of skepticismand Occam’s razor. A Type I error occurs when we be-lieve a falsehood.[6] In terms of folk tales, an investigatormay be “crying wolf” without a wolf in sight (raising afalse alarm) (H0: no wolf).

2.2 Type II error

A type II error, also known as an error of the secondkind, occurs when the null hypothesis is false, but erro-neously fails to be rejected. It is failing to assert what ispresent, a miss. A type II error may be compared witha so-called false negative (where an actual 'hit' was disre-garded by the test and seen as a 'miss’) in a test checkingfor a single condition with a definitive result of true orfalse. A Type II error is committed when we fail to be-lieve a truth.[6] In terms of folk tales, an investigator mayfail to see the wolf (“failing to raise an alarm”). Again,H0: no wolf.The rate of the type II error is denoted by the Greek letterβ (beta) and related to the power of a test (which equals1−β).What we actually call type I or type II error depends di-rectly on the null hypothesis. Negation of the null hypoth-

esis causes type I and type II errors to switch roles.The goal of the test is to determine if the null hypothesiscan be rejected. A statistical test can either reject or failto reject a null hypothesis, but never prove it true.

2.3 Table of error types

Tabularised relations between truth/falseness of the nullhypothesis and outcomes of the test:[1]

3 Examples

3.1 Example 1

Hypothesis: “Adding water to toothpaste protects againstcavities.”Null hypothesis: “Adding water to toothpaste has no effecton cavities.”This null hypothesis is tested against experimental datawith a view to nullifying it with evidence to the contrary.A type I occurs when detecting an effect (adding waterto toothpaste protects against cavities) that is not present.The null hypothesis is true (i.e., it is true that adding wa-ter to toothpaste has no effect on cavities), but this nullhypothesis is rejected based on bad experimental data.

3.2 Example 2

Hypothesis: “Adding fluoride to toothpaste protectsagainst cavities.”Null hypothesis: “Adding fluoride to toothpaste has no ef-fect on cavities.”This null hypothesis is tested against experimental datawith a view to nullifying it with evidence to the contrary.A type II error occurs when failing to detect an effect(adding fluoride to toothpaste protects against cavities)that is present. The null hypothesis is false (i.e., addingfluoride is actually effective against cavities), but the ex-perimental data is such that the null hypothesis cannot berejected.

3.3 Example 3

Hypothesis: “The evidence produced before the courtproves that this man is guilty.”Null hypothesis (H0): “This man is innocent.”A type I error occurs when convicting an innocent person(a miscarriage of justice). A type II error occurs whenletting a guilty person go free (an error of impunity).

3

A positive correct outcome occurs when convicting aguilty person. A negative correct outcome occurs whenletting an innocent person go free.

3.4 Example 4

Hypothesis: “The Medical A has a better treatment effectthan Medical B "Null hypothesis (H0): “Medical A has a better treatmenteffect than Medical B "This kind of hypothesis error may happen when the stafffailure on operation process (like applied wrong Medi-cal), medical outdated, measured tools error ...... etc.Type-1 will let experimenters thought Medical A havebatter effect than B, Type-2 is reverse situation.

3.5 Theory

From the Bayesian point of view, a type I error is one thatlooks at information that should not substantially changeone’s prior estimate of probability, but does. A type IIerror is one that looks at information which should changeone’s estimate, but does not. (Though the null hypothesisis not quite the same thing as one’s prior estimate, it is,rather, one’s pro forma prior estimate.)Hypothesis testing is the art of testing whether a varia-tion between two sample distributions can be explainedby chance or not. In many practical applications type I er-rors are more delicate than type II errors. In these cases,care is usually focused on minimizing the occurrence ofthis statistical error. Suppose the probability for a type Ierror is 1% , then there is a 1% chance that the observedvariation is not true. This is called the level of signifi-cance, denoted with the Greek letter α (alpha). While1% might be an acceptable level of significance for oneapplication, a different application can require a very dif-ferent level. For example, the standard goal of six sigmais to achieve precision to 4.5 standard deviations above orbelow the mean. This means that only 3.4 parts per mil-lion are allowed to be deficient in a normally distributedprocess

4 Etymology

In 1928, Jerzy Neyman (1894–1981) and Egon Pear-son (1895–1980), both eminent statisticians, discussedthe problems associated with "deciding whether or not aparticular sample may be judged as likely to have beenrandomly drawn from a certain population"[7]p. 1: and,as Florence Nightingale David remarked, "it is necessaryto remember the adjective 'random' [in the term 'randomsample'] should apply to the method of drawing the sampleand not to the sample itself".[8]

They identified "two sources of error", namely:

(a) the error of rejecting a hypothesis thatshould have been accepted, and(b) the error of accepting a hypothesis thatshould have been rejected.[7]p.31

In 1930, they elaborated on these two sources of error,remarking that:

...in testing hypotheses two consider-ations must be kept in view, (1) wemust be able to reduce the chance ofrejecting a true hypothesis to as lowa value as desired; (2) the test mustbe so devised that it will reject thehypothesis tested when it is likely tobe false.[9]

In 1933, they observed that these "problems are rarelypresented in such a form that we can discriminate withcertainty between the true and false hypothesis" (p. 187).They also noted that, in deciding whether to accept orreject a particular hypothesis amongst a "set of alternativehypotheses" (p. 201), H1, H2, . . ., it was easy to makean error:

...[and] these errors will be of two kinds:

(I) we reject H0 [i.e., the hypothesisto be tested] when it is true,(II) we accept H0 when some al-ternative hypothesis HA or H1 istrue.[10]p.187 (There are various no-tations for the alternative).

In all of the papers co-written by Neyman and Pearsonthe expression H0 always signifies “the hypothesis to betested”.In the same paper[10]p. 190 they call these two sources oferror, errors of type I and errors of type II respectively.

5 Related terms

5.1 Null hypothesis

Main article: Null hypothesis

It is standard practice for statisticians to conduct testsin order to determine whether or not a "speculativehypothesis" concerning the observed phenomena of theworld (or its inhabitants) can be supported. The results ofsuch testing determine whether a particular set of resultsagrees reasonably (or does not agree) with the speculatedhypothesis.On the basis that it is always assumed, by statistical con-vention, that the speculated hypothesis is wrong, and the

4 6 APPLICATION DOMAINS

so-called "null hypothesis" that the observed phenomenasimply occur by chance (and that, as a consequence, thespeculated agent has no effect) – the test will determinewhether this hypothesis is right or wrong. This is why thehypothesis under test is often called the null hypothesis(most likely, coined by Fisher (1935, p. 19)), because itis this hypothesis that is to be either nullified or not nul-lified by the test. When the null hypothesis is nullified, itis possible to conclude that data support the "alternativehypothesis" (which is the original speculated one).The consistent application by statisticians of Neyman andPearson’s convention of representing "the hypothesis to betested" (or "the hypothesis to be nullified") with the ex-pressionH0 has led to circumstances where many under-stand the term "the null hypothesis" as meaning "the nilhypothesis" – a statement that the results in question havearisen through chance. This is not necessarily the case –the key restriction, as per Fisher (1966), is that "the nullhypothesis must be exact, that is free from vagueness andambiguity, because it must supply the basis of the 'prob-lem of distribution,' of which the test of significance is thesolution."[11] As a consequence of this, in experimentalscience the null hypothesis is generally a statement thata particular treatment has no effect; in observational sci-ence, it is that there is no difference between the value ofa particular measured variable, and that of an experimen-tal prediction.

5.2 Statistical significance

The extent to which the test in question shows that the“speculated hypothesis” has (or has not) been nullifiedis called its significance level; and the higher the sig-nificance level, the less likely it is that the phenomenain question could have been produced by chance alone.British statistician Sir Ronald Aylmer Fisher (1890–1962) stressed that the “null hypothesis":

... is never proved or established, but ispossibly disproved, in the course of experi-mentation. Every experiment may be said toexist only in order to give the facts a chance ofdisproving the null hypothesis.— 1935, p.19

6 Application domains

Statistical tests always involve a trade-off between:

1. the acceptable level of false positives (in which anon-match is declared to be a match) and

2. the acceptable level of false negatives (in which anactual match is not detected).

A threshold value can be varied to make the test more re-strictive or more sensitive, with the more restrictive testsincreasing the risk of rejecting true positives, and themore sensitive tests increasing the risk of accepting falsepositives.

6.1 Inventory control

An automated inventory control system that rejects high-quality goods of a consignment commits a type I error,while a system that accepts low-quality goods commits atype II error.

6.2 Computers

The notions of false positives and false negatives have awide currency in the realm of computers and computerapplications, as follows.

6.2.1 Computer security

Main articles: computer security and computer insecurity

Security vulnerabilities are an important consideration inthe task of keeping computer data safe, while maintainingaccess to that data for appropriate users. Moulton (1983),stresses the importance of:

• avoiding the type I errors (or false negatives) thatclassify authorized users as imposters.

• avoiding the type II errors (or false positives) thatclassify imposters as authorized users.

6.2.2 Spam filtering

A false positive occurs when spam filtering or spamblocking techniques wrongly classify a legitimate emailmessage as spam and, as a result, interferes with its de-livery. While most anti-spam tactics can block or filtera high percentage of unwanted emails, doing so withoutcreating significant false-positive results is a much moredemanding task.A false negative occurs when a spam email is not detectedas spam, but is classified as non-spam. A low number offalse negatives is an indicator of the efficiency of spamfiltering.

6.2.3 Malware

The term “false positive” is also used when antivirus soft-ware wrongly classifies an innocuous file as a virus. Theincorrect detection may be due to heuristics or to an in-correct virus signature in a database. Similar problemscan occur with antitrojan or antispyware software.

6.5 Medical screening 5

6.2.4 Optical character recognition

Detection algorithms of all kinds often create false posi-tives. Optical character recognition (OCR) software maydetect an “a” where there are only some dots that appearto be an “a” to the algorithm being used.

6.3 Security screening

Main articles: explosive detection and metal detector

False positives are routinely found every day in airportsecurity screening, which are ultimately visual inspectionsystems. The installed security alarms are intended toprevent weapons being brought onto aircraft; yet they areoften set to such high sensitivity that they alarm manytimes a day for minor items, such as keys, belt buckles,loose change, mobile phones, and tacks in shoes.The ratio of false positives (identifying an innocent trav-eller as a terrorist) to true positives (detecting a would-beterrorist) is, therefore, very high; and because almost ev-ery alarm is a false positive, the positive predictive valueof these screening tests is very low.The relative cost of false results determines the likelihoodthat test creators allow these events to occur. As the costof a false negative in this scenario is extremely high (notdetecting a bomb being brought onto a plane could re-sult in hundreds of deaths) whilst the cost of a false posi-tive is relatively low (a reasonably simple further inspec-tion) the most appropriate test is one with a low statisticalspecificity but high statistical sensitivity (one that allowsa high rate of false positives in return for minimal falsenegatives).

6.4 Biometrics

Biometric matching, such as for fingerprint recognition,facial recognition or iris recognition, is susceptible to typeI and type II errors. The null hypothesis is that the inputdoes identify someone in the searched list of people, so:

• the probability of type I errors is called the “falsereject rate” (FRR) or false non-match rate (FNMR),

• while the probability of type II errors is calledthe “false accept rate” (FAR) or false match rate(FMR).[12]

If the system is designed to rarely match suspects thenthe probability of type II errors can be called the "falsealarm rate”. On the other hand, if the system is used forvalidation (and acceptance is the norm) then the FAR is ameasure of system security, while the FRRmeasures userinconvenience level.

6.5 Medical screening

In the practice of medicine, there is a significant differ-ence between the applications of screening and testing.

• Screening involves relatively cheap tests that aregiven to large populations, none of whom manifestany clinical indication of disease (e.g., Pap smears).

• Testing involves far more expensive, often invasive,procedures that are given only to those whomanifestsome clinical indication of disease, and are most of-ten applied to confirm a suspected diagnosis.

For example, most states in the USA require newbornsto be screened for phenylketonuria and hypothyroidism,among other congenital disorders. Although they dis-play a high rate of false positives, the screening testsare considered valuable because they greatly increase thelikelihood of detecting these disorders at a far earlierstage.[Note 1]

The simple blood tests used to screen possible blooddonors for HIV and hepatitis have a significant rate offalse positives; however, physicians use much more ex-pensive and far more precise tests to determine whether aperson is actually infected with either of these viruses.Perhaps the most widely discussed false positives in med-ical screening come from the breast cancer screening pro-cedure mammography. The US rate of false positivemammograms is up to 15%, the highest in world. Oneconsequence of the high false positive rate in the US isthat, in any 10-year period, half of the American womenscreened receive a false positive mammogram. False pos-itive mammograms are costly, with over $100 millionspent annually in the U.S. on follow-up testing and treat-ment. They also cause women unneeded anxiety. As aresult of the high false positive rate in the US, as manyas 90–95% of women who get a positive mammogramdo not have the condition. The lowest rate in the worldis in the Netherlands, 1%. The lowest rates are generallyin Northern Europe where mammography films are readtwice and a high threshold for additional testing is set (thehigh threshold decreases the power of the test).The ideal population screening test would be cheap, easyto administer, and produce zero false-negatives, if pos-sible. Such tests usually produce more false-positives,which can subsequently be sorted out by more sophisti-cated (and expensive) testing.

6.6 Medical testing

False negatives and false positives are significant issues inmedical testing. False negatives may provide a falsely re-assuring message to patients and physicians that disease isabsent, when it is actually present. This sometimes leadsto inappropriate or inadequate treatment of both the pa-tient and their disease. A common example is relying

6 9 REFERENCES

on cardiac stress tests to detect coronary atherosclerosis,even though cardiac stress tests are known to only detectlimitations of coronary artery blood flow due to advancedstenosis.False negatives produce serious and counter-intuitiveproblems, especially when the condition being searchedfor is common. If a test with a false negative rate of only10%, is used to test a population with a true occurrencerate of 70%, many of the negatives detected by the testwill be false.False positives can also produce serious and counter-intuitive problems when the condition being searched foris rare, as in screening. If a test has a false positive rate ofone in ten thousand, but only one in a million samples (orpeople) is a true positive, most of the positives detectedby that test will be false. The probability that an observedpositive result is a false positive may be calculated usingBayes’ theorem.

6.7 Paranormal investigation

The notion of a false positive is common in cases ofparanormal or ghost phenomena seen in images and such,when there is another plausible explanation. When ob-serving a photograph, recording, or some other evidencethat appears to have a paranormal origin – in this usage,a false positive is a disproven piece of media “evidence”(image, movie, audio recording, etc.) that actually has anatural explanation.[Note 2]

7 See also

• Binary classification

• Detection theory

• Egon Pearson

• False positive paradox

• Family-wise error rate

• Information retrieval performance measures

• Neyman–Pearson lemma

• Null hypothesis

• Probability of a hypothesis for Bayesian inference

• Precision and recall

• Prosecutor’s fallacy

• Prozone phenomenon

• Receiver operating characteristic

• Sensitivity and specificity

• Statisticians’ and engineers’ cross-reference of sta-tistical terms

• Testing hypotheses suggested by the data

• Type III error

8 Notes[1] In relation to this newborn screening, recent studies have

shown that there are more than 12 times more false posi-tives than correct screens (Gambrill, 2006. )

[2] Several sites provide examples of false positives, in-cluding The Atlantic Paranormal Society (TAPS) andMoorestown Ghost Research.

9 References[1] Sheskin, David (2004). Handbook of Parametric and

Nonparametric Statistical Procedures. CRC Press. p. 54.ISBN 1584884401.

[2] Peck, Roxy and Jay L. Devore (2011). Statistics: The Ex-ploration and Analysis of Data. Cengage Learning. pp.464–465. ISBN 0840058012.

[3] Cisco Secure IPS – Excluding False Positive Alarmshttp://www.cisco.com/en/US/products/hw/vpndevc/ps4077/products_tech_note09186a008009404e.shtml

[4] Lindenmayer, David; Burgman, Mark A. (2005). “Moni-toring, assessment and indicators”. Practical ConservationBiology (PAP/CDR ed.). Collingwood, Victoria, Aus-tralia: CSIRO Publishing. pp. 401–424. ISBN 0-643-09089-4.

[5] Schlotzhauer, Sandra (2007). Elementary Statistics UsingJMP (SAS Press) (1 ed.). Cary, NC: SAS Institute. pp.166–423. ISBN 1-599-94375-1.

[6] Shermer, Michael (2002). The Skeptic Encyclopedia ofPseudoscience 2 volume set. ABC-CLIO. p. 455. ISBN1-57607-653-9. Retrieved 10 January 2011.

[7] Neyman, J.; Pearson, E.S. (1967) [1928]. “On the Useand Interpretation of Certain Test Criteria for Purposesof Statistical Inference, Part I”. Joint Statistical Papers.Cambridge University Press. pp. 1–66.

[8] David, F.N. (1949). Probability Theory for StatisticalMethods. Cambridge University Press. p. 28.

[9] Pearson, E.S.; Neyman, J. (1967) [1930]. “On the Prob-lem of Two Samples”. Joint Statistical Papers. CambridgeUniversity Press. p. 100.

[10] Neyman, J.; Pearson, E.S. (1967) [1933]. “The testing ofstatistical hypotheses in relation to probabilities a priori”.Joint Statistical Papers. Cambridge University Press. pp.186–202.

[11] Fisher, R.A. (1966). The design of experiments. 8th edi-tion. Hafner:Edinburgh.

7

[12] Williams, G.O. (1996). “Iris Recognition Technology”(PDF). debut.cis.nctu.edu.tw. p. 56. Retrieved 2010-05-23. crossover error rate (that point where the probabilitiesof False Reject (Type I error) and False Accept (Type IIerror) are approximately equal) is .00076%

• Betz, M.A. & Gabriel, K.R., “Type IV Errors andAnalysis of Simple Effects”, Journal of EducationalStatistics, Vol.3, No.2, (Summer 1978), pp. 121–144.

• David, F.N., “A Power Function for Tests of Ran-domness in a Sequence of Alternatives”, Biometrika,Vol.34, Nos.3/4, (December 1947), pp. 335–339.

• Fisher, R.A., The Design of Experiments, Oliver &Boyd (Edinburgh), 1935.

• Gambrill, W., “False Positives on Newborns’ Dis-ease Tests Worry Parents”, Health Day, (5 June2006). 34471.html

• Kaiser, H.F., “Directional Statistical Decisions”,Psychological Review, Vol.67, No.3, (May 1960),pp. 160–167.

• Kimball, A.W., “Errors of the Third Kind in Statis-tical Consulting”, Journal of the American StatisticalAssociation, Vol.52, No.278, (June 1957), pp. 133–142.

• Lubin, A., “The Interpretation of Significant In-teraction”, Educational and Psychological Measure-ment, Vol.21, No.4, (Winter 1961), pp. 807–817.

• Marascuilo, L.A. & Levin, J.R., “Appropriate PostHoc Comparisons for Interaction and nested Hy-potheses in Analysis of Variance Designs: TheElimination of Type-IV Errors”, American Educa-tional Research Journal, Vol.7., No.3, (May 1970),pp. 397–421.

• Mitroff, I.I. & Featheringham, T.R., “On Sys-temic Problem Solving and the Error of the ThirdKind”, Behavioral Science, Vol.19, No.6, (Novem-ber 1974), pp. 383–393.

• Mosteller, F., “A k-Sample Slippage Test for anExtreme Population”, The Annals of MathematicalStatistics, Vol.19, No.1, (March 1948), pp. 58–65.

• Moulton, R.T., “Network Security”, Datamation,Vol.29, No.7, (July 1983), pp. 121–127.

• Raiffa, H., Decision Analysis: Introductory Lectureson Choices Under Uncertainty, Addison–Wesley,(Reading), 1968.

10 External links• Bias and Confounding – presentation by NigelPaneth, Graduate School of Public Health, Univer-sity of Pittsburgh

8 11 TEXT AND IMAGE SOURCES, CONTRIBUTORS, AND LICENSES

11 Text and image sources, contributors, and licenses

11.1 Text• Type I and type II errors Source: https://en.wikipedia.org/wiki/Type_I_and_type_II_errors?oldid=683980392 Contributors: MichaelHardy, Fred Bauder, Delirium, Den fjättrade ankan~enwiki, BenKovitz, Conti, Jnc, Mazin07, Chris 73, Nurg, Giftlite, Smjg, Pashute,Pgan002, Jdevine, Piotrus, James A. Donald, Pmanderson, Urhixidur, Poccil, Rich Farmbrough, Bender235, Kaisershatner, El-wikipedista~enwiki, El C, Nonpareility, Arcadian, BlueNovember, Bfg, Drf5n, Musiphil, Gary, Andrewpmk, Samohyl Jan, Cburnett,Suruena, Zin~enwiki, BDD, InBalance, Thryduulf, Mindmatrix, Shreevatsa, Contele de Grozavesti, Junes, Btyner, Shanedidona, Elvey,Rjwilmsi, Reinis, Billjefferys, Wragge, Jrtayloriv, TeaDrinker, Atif.hussain, Bgwhite, Roboto de Ajvol, Wavelength, Hairy Dude, RedSlash, Rsrikanth05, Sjb90, Dbfirs, Aaron Schulz, Vlad, Lcmortensen, DRosenbach, Arthur Rubin, Modify, Ketil3, Ethan Mitchell, BoJacoby, Snalwibma, SmackBot, Lantianer, NZUlysses, Jtneill, BiT, Xaosflux, The Gnome, The X, Taelus, Emufarmers, OrangeDog,Afasmit, Nbarth, Zven, Muboshgu, Danielkueh, Anthon.Eff, RolandR, G716, Jbergquist, Mwtoews, Spinality, Ohconfucius, Tim bates,Evenios, Nijdam, Edesio, Still A Student, Dicklyon, Lifeartist, Varuag doos, Cpeter, Meltingpot, Cadaeib, Ginkgo100, Dthvt, FreelanceIntellectual, Jkchoe, Frank Lofaro Jr., Quang thai, CmdrObot, Jackzhp, Floridi~enwiki, Pfhenshaw, Cydebot, Groovy12, Clayoquot,Sandlaus, Blaisorblade, YorkBW, Lindsay658, IP 84.5, Marek69, Sinclairway, Behco, Memphis-Ahn, XavierEverett, JAnDbot, Nthep,Ph.eyes, Dr mindbender, Myownfanclub, Acroterion, CrizCraig, Jarekt, SHCarter, Rami R, Baccyak4H, Destynova, David Eppstein, A2-computist, WLU, Raoulduke47, Epylar, CliffC, Dima373, Ndabney, Ranman45, R'n'B, Mbhiii, Lilac Soul, J.delanoy, Pharaoh of theWizards, Adavidb, Stolkin, Mikael Häggström, SteveChervitzTrutane, KCinDC, Nadiatalent, Fuenfundachtzig, DMCer, Ratfox, Anas.sal,Julienrl, UnitedStatesian, Jamelan, Temporaluser, Gabrielespnz, Yabti, Scarbrow, Thefellswooper, Reuqr, Bajaj.nikkey, Oda Mari, AnchorLink Bot, Melcombe, Krefts, Mikefero, Telakin, Doncampbell30, SpookyVale, ClueBot, Rumping, Andy1618, Darobian, Imperfectly-Informed, JP.Martin-Flatin, Rturlapaty, Auntof6, Stxera, Excirial, Tomeasy, SpikeToronto, Sun Creator, ZuluPapa5, Patrick30, Cjtysor,ThatProf, FinnMan, Qwfp, Zodon, Tayste, Addbot, Wickey-nl, Fgnievinski, OZJ, CanadianLinuxUser, Orlandoturner, AgadaUrbanit,Chris.Heward, Tide rolls, שי ,דוד Teles, Legobot, Yobot, Pedmayn, Wjastle, Raimundo Pastor, AnomieBOT, Kingpin13, Bluerasberry,Materialscientist, Jon187, Eumolpo, DynamoDegsy, MonojitSengupta, Peterdx, J04n, Omnipaedista, Kernel.package, Sqgl, JonDePlume,FrescoBot, Buzhan, Lion-hearted, Nagdeep, RedBot, Tsunhimtse, FloorSugar, Raidon Kane, Duoduoduo, Iowawindow, TheLongTone,Reenus, DARTH SIDIOUS 2, PPdd, Skelly 53, DASHBot, EmausBot, RA0808, Abcdefghijklmnop1234567, HenryXVII, Der Träumer(DE), Alpha Quadrant (alt), Loard, Muv4zqlvc3, Ocaasi, Hudson Stern, SmesharikiAreTheBest, Glorytothenation, Mikhail Ryazanov,ClueBot NG, Rainbyte, Mathstat, Peter James, Kkddkkdd, Liana, Accusativen hos Olsson, Snotbot, Frietjes, Alexhangartner, Widr,Airspace2, MerlIwBot, Helpful Pixie Bot, Jpgill86, Amoriarty21, Curb Chain, BG19bot, CeraBot, Amyunimus, Lnedelescu, Xcopyand-pasteskillz, EmilioDada, Khazar2, Dexbot, Sminthopsis84, Geomayne, Makecat-bot, 93, Tal.spackman, H.A.Phe, Azn2themax, Wik-iuser13, Ginsuloft, RSelove, Soumava1, Bilorv, Bpwilbur, Findslowly, Engheta, Errorfixer1, Isambard Kingdom, Qazplmqazplm, HemantRupani, Johnnylioltu, Texyalen and Anonymous: 322

11.2 Images• File:Commons-logo.svg Source: https://upload.wikimedia.org/wikipedia/en/4/4a/Commons-logo.svg License: ? Contributors: ? Originalartist: ?

• File:Fisher_iris_versicolor_sepalwidth.svg Source: https://upload.wikimedia.org/wikipedia/commons/4/40/Fisher_iris_versicolor_sepalwidth.svg License: CC BY-SA 3.0 Contributors: en:Image:Fisher iris versicolor sepalwidth.png Original artist: en:User:Qwfp (origi-nal); Pbroks13 (talk) (redraw)

• File:Folder_Hexagonal_Icon.svg Source: https://upload.wikimedia.org/wikipedia/en/4/48/Folder_Hexagonal_Icon.svg License: Cc-by-sa-3.0 Contributors: ? Original artist: ?

• File:People_icon.svg Source: https://upload.wikimedia.org/wikipedia/commons/3/37/People_icon.svg License: CC0 Contributors: Open-Clipart Original artist: OpenClipart

• File:Portal-puzzle.svg Source: https://upload.wikimedia.org/wikipedia/en/f/fd/Portal-puzzle.svg License: Public domain Contributors: ?Original artist: ?

11.3 Content license• Creative Commons Attribution-Share Alike 3.0