# Joshua Naranjo - wmich. Joshua Naranjo Questionnaire validity. Joshua Naranjo Questionnaire validity

date post

13-Jul-2020Category

## Documents

view

9download

0

Embed Size (px)

### Transcript of Joshua Naranjo - wmich. Joshua Naranjo Questionnaire validity. Joshua Naranjo Questionnaire validity

Questionnaire validity

Joshua Naranjo

Stat 5630 Western Michigan University

Joshua Naranjo Questionnaire validity

Questionnaire theory

Recall: Study context

research aims

design

measurement/questionnaire (satisfaction, ease of use, religiosity)

analysis

Statistical concepts in questionnaire theory

reliability and validity

intraclass correlation coefficient

Cronbach’s alpha

factor analysis

Joshua Naranjo Questionnaire validity

Questionnaire theory

Reliability and validity of measurements

Ex: ”Association between intelligence, communication skills, and effective leadership”

Validity – does an instrument measure what it claims to measure? Reliability – does it work consistently?

Joshua Naranjo Questionnaire validity

Questionnaire theory

Types of validity:

Construct validity – is it valid in theory?

Face validity – does it measure what it aims to measure?

Convergent validity – does the survey compare well with another measure of the same thing?

Discriminant validity – can it tell different groups apart?

(Ex: intelligence, communication skills, leadership)

Joshua Naranjo Questionnaire validity

Questionnaire theory

Types of Reliability:

Internal reliability – do respondents give consistent answers over similar questions?

Test-retest reliability – do respondents give consistent answers over time?

Inter-rater reliability – do raters give consistent answers for the same item?

Joshua Naranjo Questionnaire validity

Intraclass correlation coefficient

Ex: “How much agreement is there between physicians reading CT scans for disease progression?”

CT Scan Physician 1 Physician 2 1 7 5 2 6 9 : : : : : : n 2 2

Q: Pearson correlation?

A: Which student is X and which is Y ?

Joshua Naranjo Questionnaire validity

Intraclass correlation coefficient

Ex: “How much agreement is there between physicians reading CT scans for disease progression?”

CT Scan Physician 1 Physician 2 1 7 5 2 6 9 : : : : : : n 2 2

Q: Pearson correlation? A: Which student is X and which is Y ?

Joshua Naranjo Questionnaire validity

Intraclass correlation coefficient

ICC measures agreement between exchangeable variables (i.e. no inherent ordering). Used for test-retest reliability, and inter-rater reliability.

Let

x = 1

2n

∑ (xi1 + xi2)

s2 = 1

2n

{∑ (xi1 − x)2 +

∑ (xi2 − x)2

} Then

ICC = 1

ns2

∑ (xi1 − x)(xi2 − x)

Joshua Naranjo Questionnaire validity

Intraclass correlation coefficient

For groups of three,

x = 1

2n

∑ (xi1 + xi2 + xi3)

s2 = 1

2n

{∑ (xi1 − x)2 +

∑ (xi2 − x)2 +

∑ (xi3 − x)2

} Then

ICC = 1

3ns2

∑ {(xi1 − x)(xi2 − x) + (xi1 − x)(xi3 − x)

+(xi2 − x)(xi3 − x)}

Joshua Naranjo Questionnaire validity

Cronbach alpha

Internal Consistency: Hearing aid survey

Suppose we want to measure how well the hearing aid “listens in conversation”

Q: Do the questions “measure the same thing”?

Joshua Naranjo Questionnaire validity

Cronbach’s alpha:

Suppose that T = x1 + · · ·+ xk is the composite score for a factor or construct. Then

α = k

k − 1

( 1−

∑ var(xi )

var(T )

)

Joshua Naranjo Questionnaire validity

Cronbach alpha

α = 1010−1 ( 1− 19.02240.6933

) = 0.59172

Joshua Naranjo Questionnaire validity

It may be easier to understand as follows:

α = k

k − 1

( 1−

∑ var(xi )

var(T )

) =

k

k − 1

( var(T )−

∑ var(xi )

var(T )

)

= k

k − 1

(∑ i 6=j cov(xi , xj)

var(T )

) So α is an average covariance between the xi s.

Joshua Naranjo Questionnaire validity

Cronbach alpha

But this is still just a computing formula. In Cronbach Psychometrika, 1951, Cronbach proposes the statistic as a

“....mean of all split-half coefficients resulting from different splittings of a test... therefore an estimate of the correlation between two random samples of items...”

Joshua Naranjo Questionnaire validity

Examples

Published example:

Joshua Naranjo Questionnaire validity

Factor Analysis

Joshua Naranjo Questionnaire validity

Factor Analysis

Joshua Naranjo Questionnaire validity

Factor Analysis

Joshua Naranjo Questionnaire validity

Factor Analysis

Let Yi be the response to the ith question, i = 1, . . . , 15. The idea behind factor analysis is that the 15 responses actually depend on only two or three common underlying unobservable factors, i.e.

Yi = µi + ai f1 + bi f2 + �

History of factor analysis

Charles Spearman (1904) two-factor theory of intelligence: general and specific

Thurstone 7-factor: numerical, verbal comprehension, word fluency, perceptual speed, memory, inductive reasoning, spatial ability

Sternberg 3-factor: analytical (problem-solving), creative (new ideas), and practical (everyday logic)

Joshua Naranjo Questionnaire validity

Factor Analysis

Back to article, let

Y1 Y2 :

Y15

=

µ1 µ2 : µ15

+

l11 l12 l21 l22 : :

l15,1 l15,2

( f1f2 )

+

�1 �2 : �15

or

Y = µ+ LF + �

where µ and L are constants, F and � have mean 0

V (F) =

( 1 0 0 1

) , V (�) = Ψ =

ψ1 0 . . . 0 0 ψ2 . . . 0 ...

... ...

0 0 . . . ψ15

Joshua Naranjo Questionnaire validity

so that V (Y) = LL′ + Ψ

Estimation:

S = 15∑ i=1

λivivi ′ ∼=

2∑ i=1

λivivi ′ =

(√ λ1v1

√ λ2v2

)( √λ1v1′√ λ2v2

′

) = L̂L̂′

where λ1 > λ2 > · · · > λ15. The estimated coefficients of the underlying factors are derived from the eigenvalues and eigenvectors of S.

Joshua Naranjo Questionnaire validity

Factor Analysis: Rotation

Let A be any 2× 2 orthogonal matrix, i.e. AA′ = I. Then

Y = µ+ LF + � = µ+ LAA′F + � = µ+ L∗F∗ + �

so that there is an ambiguity to the factor analysis model. But this can be exploited, might as well choose A′F∗ = (f ∗1 f

∗ 2 ) ′ so that the

factors are more interpretable. In practice, this means we want to rotate so that each of the 15 questions will have high coefficient in only one factor.

Joshua Naranjo Questionnaire validity

Joshua Naranjo Questionnaire validity

Factor Analysis: Explained variance

Recall that S = ∑15

i=1 λivivi ′ ∼=

∑2 i=1 λivivi

′

trace(S) = s21 + s 2 2 + · · · s215

= λ1 + λ2 + · · ·+ λ15 The proportion of total variation explained by the ith factor is

λi λ1 + λ2 + · · ·+ λ15

Joshua Naranjo Questionnaire validity

Joshua Naranjo Questionnaire validity

Questionnaire theory

*View more*