Concept of Test of Significance

(t-test, proportion test and Chi-square test)

By

Dr. Ramnath Takiar

Ex-Director Grade Scientist,National Cancer Registry Programme

(Indian Council of Medical Research)

October 23, 2014

Concept of Test of significance:

Tests of statistical significance are invariably applied now-a-days by research scientists. Good medical journals refuse to accept papers for publication if the authors have not used the philosophy of significance testing in evaluating their results.

It is not adequate to mechanically undertake significance tests , the scientists/experimental workers must fully understand the basic concepts underlying a significance test, the assumptions involved and the limitations, for making the proper interpretations.

Generally, the existence of statistical significance difference is regarded as a proof of the existence of an important difference between two sample results. Similarly, the non-significant differences are regarded as proof of no differences in two sample results. To properly appreciate the role of significance testing, it is important to understand first the concept of sampling fluctuation.

Mean 10.4 Lower limit = Mean-1.96 SEUpper limit = Mean+1.96 SE

1 2 31 10 13 9 10.7 1.20 8.31 13.022 10 13 12 11.7 0.88 9.94 13.403 10 13 8 10.3 1.45 7.49 13.184 10 9 12 10.3 0.88 8.60 12.065 10 9 8 9.0 0.58 7.87 10.136 10 12 8 10.0 1.15 7.74 12.267 13 9 12 11.3 1.20 8.98 13.698 13 9 8 10.0 1.53 7.01 12.999 13 12 8 11.0 1.53 8.01 13.9910 9 12 8 9.7 1.20 7.31 12.02

Please note that for confidence limit, I have used Normal values, while based on number of observations (6), t-values can be used. The use of Normal values was done for easy understanding.

SE Lower Limit

Upper limit

Mean (means) = 10.4; SE= SD(means) = 0.798

Sample Value

Mean

Concept of Sampling fluctuation:

T-test To test that the Population Mean is M

(Observed Mean – Population Mean )/ Standard error ≈ t (n-1) where (n-1) is the degrees of freedom (df)

= (10.7 – 10.4)/1.20 = (0.3/1.20) = 0.25 (t =4.30 at 2 d.f. from the table) => P>0.05

= (11.7 - 10.4) / 0.88 = (1.3/0.88) = 1.48

= (9.0 – 10.4) /0.58 = (1.4/0.58) = 2.41

Null Hypothesis: It is a definite statement about the population parameter. Such a hypothesis is of no difference, is called null hypothesis and usually denoted by Ho. According to Prof. R.A. Fisher, null hypothesis is the hypothesis which is tested for its possible rejection under the assumption that it true.

Alternative Hypothesis: Any hypothesis which is complimentary to the null hypothesis is called an alternative hypothesis usually denoted by H1.

Type I Error: Reject Ho when it is true.Type II Error: Accept Ho when it is not true.

P(Reject Ho when it true) = P(Reject Ho/Ho) = P(Accept Ho when it is not true) = P(Accept Ho/H1) =

Conventionally, the following values of errors are accepted: = 0.05; = 0.90

Examples of t-test: 1. For a random sample of 10 persons , fed on diet A, the increased in weight in Kgs for a certain period were: 10, 6,16,17, 13, 12, 8 14,15,9 (mean= 13.4; SE= 1.634) Test whether the gain in weight, on an average is 15.0 Kgs.

Ho = Sample mean is not different from population mean (15.0) or mean gain in weight = 15.0 kgs.

H1 = Sample mean is different from population mean ( 15.0) or Mean gain in weight 15.0 kgs

t = (Sample mean – Population mean)/ SE of mean = ( 13.4-15.0)/1.634 = - 0.979 (df = 9) = P=0.35 or P > 0.05 Hence, Ho is accepted => the sample mean = 15.0

2. The manufacturer of a certain make of electric bulb claims that his bulbs have a mean life of 25 months. A random sample of 6 such bulbs gave the following values: 24, 26,30, 20, 20, 18 (mean= 23.0; SE= 1.844)

can you regard the producers claim to be valid ? Ho = Sample mean is not different from population mean

(25.0) or mean life of bulb = 25.0 months

H1 = Sample mean is different from population mean ( 25.0) or Mean life of bulb 25.0 months

t = Sample mean – Population mean)/ SE of mean = ( 23.0-25.0)/1.428 = 1.085 (df = 5) = P=0.33 or P > 0.05Hence, Ho is accepted => the mean life of bulb = 25.0

months

T test- Two means:

t test = (mean1-mean2)/S* (1/n1 + 1/n2)^0.5

Where S is Pooled SD of Sample 1 and Sample2 and can be given by the following formula:

S = [(n1-1) (SD1)^2+ (n2-1)*(SD2)^2/(n1+n2-2)]^0.5

In above formula, the degrees of freedom = df = (n1+n2-2)

Examples of T test- Two means:

3. Below are given the gain in weights (Kgs) of preschool children fed for certain period of time on two diets A and BDiet A: 2.5, 3.2, 3.0, 3.4, 2.4, 1.4, 3.2, 2.4, 3.0, 2.5 Diet B: 4.4, 3.4, 7.2, 1.0, 4.7, 3.1, 4.0, 3.2, 3.5, 1.8, 2.1, 2.9 Test if two diets differ significantly as regards their effect on increase in weight. Diet A: MeanA = 2.7; Diet B Mean = 3.45 SD A = 0.59 SD B = 1.59 N1 = 10 N2 = 12 Pooled SD2 = {(N1-1)* SDA2 + (N2-1)* SDB2 } /(N1+N2-2)o = [(10-1)*0.592 + (12-1)* 1.592 ]/(10+12-2) = [3.12 + 27.87]/20 = 1.55(1/N1) = 1/10 = 0.1; (1/N2) =(1/12) = 0.083

(MEAN1-MEAN2)/{S2 * [(1/N1)+(1/N2)]}^0.5

(3.7 – 2.45)/[1.55*[(0.1)+(0.083)]]^0.5 = 0.75/ (0.284)^0.5 = 0.75/0.533 = 1.39 df = 20 = P = 0.18 => P > 0.05 ; Hence Ho is accepted. Diet A and Diet B are equally effective in producing the weight

gain among preschool children. ------------------------------------------------------------------------------------4. A reading test is given to two different sections of the same

class. The results of the test are Section A : Mean = 75 Section B: Mean=65 SD = 8 SD = 10 N1= 12 N2 = 15Is the difference between the means of the two section is

significant? Mean1-Mean2 = 10; S2 = 84.16 ; (1/N1+1/N2) = (0.083+0.067)t = 2.81 ; df = 25; P = 0.009 => The difference between the

mean scores are different.

Some more examples of t test applications:

5. Measurements of the fat content of two kinds of Ice cream, Brand A and Brand B yielded the following sample data:

Brand A : 13.5, 14.0, 13.6, 12.9, 13.0 Brand B : 12.9, 12.5, 11.5, 10.0, 10.0 Test whether the fat contents of ice cream of both the

brands are comparable.

6. Two independent groups of 10 children were tested to find how many digits they could repeat from memory after hearing them. The results are as follow:

Group A : 8 6 5 7 6 8 7 4 5 6 Group B: 10 6 7 8 6 9 7 6 7 7 Is the difference between the mean scores of two groups

are significant?

Paird t test : Examples

7. Eleven school boys were given a test in Statistics. They were given a month’s tuition and a second test was held at the end of it. Do the marks give evidence that the students have gained from the extra coaching?

Marks in I test : 23 20 19 21 18 20 18 17 23 16 19 Marks in II test: 24 19 22 18 20 22 20 20 23 20 18 8. A drug was administered to 10 patients and the increments

in their blood pressure were recorded to be 6 3 -2 4 -3 4 6 0 3 2 Is it reasonable to believe that drug has no effect on

change of blood pressure? _ t = diff(mean)/ SE of diff = d / SE(d) with (n-1) df. where d is the difference between the observations.

Sl. No. Test I Test 2 Diff. Change

in BP1 23 24 1 62 20 19 -1 33 19 22 3 -24 21 18 -3 45 18 20 2 -36 20 22 2 47 18 20 2 68 17 20 3 09 23 23 0 310 16 20 4 2

SUM 13 23MEAN 1.3 2.3

SD 2.11 3.09SE 0.70 1.03

STATISTICS SCORE

Variable Statistics

score Change in BP

MEAN 1.3 2.3

SE 0.704 1.031

t test 1.85 2.23

P vluae 0.098 0.053

tabulated value (9 df)

2.26

Conclusions:

1. Ho accepted => The mean marks obtained after the tuition are not

significantly different from that obtained before the tuition. Extra coaching among the students has not resulted in

improving their test scores. 2. Ho is accepted =>

The increments in blood pressure was not significantly different from zero.

The drug has no effect on change of blood pressure.

It should be noted that the t-test which we have discussed is often used

1. To test whether the sample mean is significantly different from a hypothetical mean or not?

2. To test whether two sample means are comparable or different?

3. When the number of observations for selected sample(s) is(are) small say below 30.

Assumptions used in application of t-test:

1. The population from which the sample(s) is (are) drawn, follows normal distribution.

2. In case of comparison between two sample means, it is assumed that both the samples are drawn from normal population and their variances are comparable.

It should be noted that when n is large (above 30) we use Normal test instead of t-test. The Normal test could be usedFor testing of single mean and for comparison of two sample means. The formulae remain same as that of t-test with the difference that Normal table is used instead of t-table to get the tabulated value.

1% 5% 10%

Two tailed 2.58 1.96 1.645

One tailed 2.33 1.645 1.28

Level of significanceCritical value

Critical values of Z statistic

Test for Proportion of Successes:

Instead of dealing with number of successes, very often we may be interested in proportion of success obtained in an experiment that is the number of successes are divided by the total number of trials made. Therefore,

p = probability of success in each trial;

q = (1-p) = probability of failure.

n = size of sample; then SE(p) = √pq/n

Z = (p –P) / √(PQ/n) where p = sample proportion &

P = Population proportion

= (p-P)/ √(pq/n)

= (sample p – Population p)/SE(p)

Example 9: In a simple random sample of 600 men taken from a big city 400 are found to be smokers. Test the hypothesis that 60% of the men are smokers.

Ho: The 60% of men are smokers. (Given P = 0.6) H1: The percentage of smokers is not equal to 60% (P 0.6)

Proportion of smokers = p = 400/600 = 0.666

SE(p) = √pq/n = (0.666*0.334/600)^0.5 = (0.000371)^0.5=

= 0.0193

Z = (p-P)/ SE(p) = (0.666-0.6)/SE = 0.066/0.0.0193 = 3.42 ;

The difference between observed and expected proportion of smoker is more than 1.96 SE (5% level of significance) .

Hence our hypothesis is rejected and we conclude that the the proportion of smokers in the city is greater than 60%.

Example 10: 500 subjects were surveyed for their dental hygiene and 30 of them were found to be with dental problems. Test the hypothesis that proportion of dental problems in population is not different from 5%.?

Ho: The dental problem in population is 5%. (Given P=0.05)

H1: The dental problem in population is different from 5%. .

P 0.05

p = 30/500 = 0.06 ; SE(p) = √pq/n

SE (p) = (0.06*0.94/500)^0.5 = 0.0106

Z = (p – P)/ √pq/n =(0.06-0.05)/0.0106

= 0.01/0.0106 = 0.94

Since Z calculated is less than 1.96, we accept Ho.

The dental problems in population is not different from 5%.

Test for difference between Proportions: If two samples are drawn from different populations, we may

be interested in finding out whether the difference between the proportion of successes is significant or not. In such a case we take the hypothesis that proportion of success in one sample (p1) and success in another sample (p2) is due to fluctuations of random sampling.

Z = (p1-p2)/√(pq(1/n1+1/n2)) where

p1 = proportion in sample 1.

p2 = proportion in sample 2.

n1 = sample size of sample 1

n2 = sample size of sample 2.

p = (n1*p1+n2*p2)/(n1+n2)

11. In a random sample of 100 men taken from a village A, 60 are found to be consuming alcohol. In another sample of 200 men taken from village B, 100 were found to be consuming alcohol. Do the two villages differ significantly in respect of their consuming alcohol?

Given p1=60/100 = 0.6; n1=100 p2= 100/200 = 0.5; n2=200 Ho = p1=p2 ; H1= p1p2

P = (n1*p1+n2*p2)/(n1+n2) = (100*0.6+200*0.5)/(100+200) = (60+100)/300 = 160/300 = 0.53 Z = (p1-p2)/√(pq(1/n1+1/n2)) = (0.6-0.5)/ √(0.53*0.47(1/100+1/200) = (0.1/ √0.249(0.01+0.005) =0.1/ √0.003737 = 0.1/0.6112 = 1.63 Since calculated Z is less than 1.96, we accept Ho. => The percentage of alcohol consumers are comparable bet. Two villages.

12. In a large city A 25% of a random sample of 900 school boys has defective eyesight. In another large city B, 20% of a random sample of 1600 had the same defect. Is this difference between the two proportion significant?

Given p1=0.25; n1=900 p2= = 0.20; n2=1600 Ho = p1=p2 ; H1= p1p2

P = (n1*p1+n2*p2)/(n1+n2) = (900*0.25+1600*0.2)/(900+1600) = (225+320)/2500 = 545/2500 = 0.218 Z = (p1-p2)/√(pq(1/n1+1/n2)) = (0.25-0.2)/ √(0.22*0.78*(1/900+1/1600) = (0.05/ √0.172(0.001+0.0006) =0.05/ √0.000298 = 0.05/0.01726 = 2.89 Since calculated Z is more than 1.96, we reject Ho. => The percentage of eye sight problem is different between the cities. City A has more eyesight problem.

Chi-square test: Very often in the field of research, we come across

qualitative types of data like presence or absence of a symptom, classification of a pregnancy as ‘high risk’ or ‘low risk’ , the degree of severity of a disease ( mild, moderate, severe). When we are interested in tabulating such type of data for more than one group and want meaningful comparisons then a method which is useful in such situations is Chi-square test.

The Chi-square test is designed to examine whether a series of observed numbers in various categories of the data are consistent with the numbers expected in those categories on some specific hypothesis.

In practice, there will be some differences between the observed (O) and expected (E) numbers in each category and our aim is to derive a single quantity to conclude whether the variation seen is genuine or due to sampling.

The Chi-square is defined as ∑ (O-E)2/E with (n-1) df.

13. In a hospital, 480 female and 520 male babies were born in a week. Do these figures confirm the hypothesis that males and females are born in equal number?

Ho: The male and female babies are born in equal proportions

H1: The male and female babies are not born in equal proportions.

Under Ho male = female = (480+520)/2 = 1000/2 = 500

X2 = Chi-square = ∑ (O-E)2/E with (n-1) df.

= (480-500)2/500 + (520-500)2/500

= (20)2/500 + (20)2/500

= 400 /500 + 400/500 = 0.8+0.8 =1.6

Since 1.6 is less than 3.84, we accept Ho.

=> Male and female babies are born in equal proportions.

Degress of freedom

1 2 3 4 5

Value 3.84 5.99 7.82 9.49 11.07

Tabulated values of Chi-square by degree of freedom

14. A pharmaceutical company claimed that a new product introduced by them can cure 80% of the patients with a particular disease in seven days. In an experiment conducted to test this claim, it was observed that among 80 patients with the disease, only 56 (70%) were cured within the stipulated time. Can we conclude that the company’s claim is exaggerated?

X2 = (56-64)2/64 + (24-16)2/16 = 64/64+ 64/16 =5; df=1 Since Calculated Chi square is more than 3.84, we

conclude that the claim is exaggerated.

Cured Not cured Total

Observed 56 24 80

Expected 64 16 80

15. Consider a controlled clinical trial in which 90 of 100 patients received treatment A got cured compared with 105 of 150 who received Treatment. Test the hypothesis that Treatment A is more effective than Treatment B?

Ho = Both the treatments are equally effective H1 = Treatment A is more effective than B e11 = R1*C1/T = 195*100/250 = 78 e12 = R1*C2/T = 55*100/250 = 22 e21 = R2*C1/T = 195*150/250 = 117 e22 = R2*C2/T = 55*150/250 = 33

Cured Not cured Total (Rj)

TREATMENT A 90 (a11) 10 (a12) 100 (R1)

TREATMENT B 105 (a21) 45 (a22) 150 (R2)

Total (Cj) 195 (C1) 55 (C2) 250

Response to treatment

Having known the Expected values, the Chi-square can be calculated as follows:

X2 = (90-78)2/78 + (10-22)2/22 + (105-117)2/117 + (45-33)2/33

= 1.85 + 6.55 + 1.23 + 4.36 = 13.99; df=1

In general the formula for calculation of df is

= (m-1)*(n-1) where m = no. of rows; n= no. of columns

Since calculated chi-square is more than tabulated value (3.84), we decide to reject Ho.

The cure rate is different between Treatment A and Treatment B.

Treatment A is better in respect of cure rate.

Chi-square test for association: 16. In a college, 1072 students were classified according to their intelligence and economic conditions. Test whether there is any association between intelligence and economic conditions.

Ho : There is no association between economic condition and Intelligence. H1: There is association bet economic cond. and Intelligence

Economic condition

Excellent Good Mediocore Dull Total

Good 48 199 181 82 510

Not good 81 185 190 106 562

Total 129 384 371 188 1072

Classification of students according to their economic condition and intelligence

Calculations of Expected values and Chi-square e11 = R1*C1/T = 129*510/1072 = 61.4 o11 = 48

e12 = R1*C2/T = 384*510/1072 = 182.7 o12 = 199

e13 = R1*C3/T = 371*510/1072 = 176.5 o13 = 182

e14 = R1*C4/T = 188*510/1072 = 89.4 o14 = 82

e21 = R1*C1/T = 129*562/1072 = 67.6 o21 = 81

e22 = R1*C2/T = 384*562/1072 = 201.3 o22 = 185

e23 = R1*C3/T = 371*562/1072 = 194.5 o23 = 190

e24 = R1*C4/T = 188*562/1072 = 98.6 o24 = 106

Using the formula X2 = Chi-square = ∑ (O-E)2/E with (n-1) df.

We get X2 = 9.735; df = (2-1)*(4-1)=1*3 =3 df. (p=0.0209)

Since calculated chi-square is greater than the tabulated value ( 7.82), Ho is rejected => there is association bet Intelligence and Economic condition

The following data are for a sample of 300 car owners who were classified with respect to age and the number of accidents they had during the past two years. Test whether there is any relationship between these two variables.

0 1-2 3 or more Total

<21 8 23 14 45

22-26 21 42 12 75

>= 27 71 90 19 180

Total 100 155 45 300

Accidents Age group

Conditions for the Validity of Chi-square test:The sample observations should be independent.

The constrains on cell frequency, if any, should be linear e.g., ∑ Oi = ∑ Ei.

The total frequency should be reasonably large say greater than 50.

No theoretical cell frequency should be less than 5.

If any theoretical cell frequency is less than 5, then for the application of Chi-square test, it is pooled with preceding or succeeding frequency so that pooled frequency is more than 5 and finally adjust for d.f. lost in pooling.

Chi-square test should be applied only to frequencies and not to percentages or prortions.

Chi-square test depends only on the set of observed and expected frequencies and on degree of freedom. It does not make any assumption s regarding the population from which the observations are drawn. Hence , termed as non-parametric test.