Joshua Naranjo - wmich. Joshua Naranjo Questionnaire validity. Joshua Naranjo Questionnaire validity

download Joshua Naranjo - wmich. Joshua Naranjo Questionnaire validity. Joshua Naranjo Questionnaire validity

of 25

  • date post

    13-Jul-2020
  • Category

    Documents

  • view

    9
  • download

    0

Embed Size (px)

Transcript of Joshua Naranjo - wmich. Joshua Naranjo Questionnaire validity. Joshua Naranjo Questionnaire validity

  • Questionnaire validity

    Joshua Naranjo

    Stat 5630 Western Michigan University

    Joshua Naranjo Questionnaire validity

  • Questionnaire theory

    Recall: Study context

    research aims

    design

    measurement/questionnaire (satisfaction, ease of use, religiosity)

    analysis

    Statistical concepts in questionnaire theory

    reliability and validity

    intraclass correlation coefficient

    Cronbach’s alpha

    factor analysis

    Joshua Naranjo Questionnaire validity

  • Questionnaire theory

    Reliability and validity of measurements

    Ex: ”Association between intelligence, communication skills, and effective leadership”

    Validity – does an instrument measure what it claims to measure? Reliability – does it work consistently?

    Joshua Naranjo Questionnaire validity

  • Questionnaire theory

    Types of validity:

    Construct validity – is it valid in theory?

    Face validity – does it measure what it aims to measure?

    Convergent validity – does the survey compare well with another measure of the same thing?

    Discriminant validity – can it tell different groups apart?

    (Ex: intelligence, communication skills, leadership)

    Joshua Naranjo Questionnaire validity

  • Questionnaire theory

    Types of Reliability:

    Internal reliability – do respondents give consistent answers over similar questions?

    Test-retest reliability – do respondents give consistent answers over time?

    Inter-rater reliability – do raters give consistent answers for the same item?

    Joshua Naranjo Questionnaire validity

  • Intraclass correlation coefficient

    Ex: “How much agreement is there between physicians reading CT scans for disease progression?”

    CT Scan Physician 1 Physician 2 1 7 5 2 6 9 : : : : : : n 2 2

    Q: Pearson correlation?

    A: Which student is X and which is Y ?

    Joshua Naranjo Questionnaire validity

  • Intraclass correlation coefficient

    Ex: “How much agreement is there between physicians reading CT scans for disease progression?”

    CT Scan Physician 1 Physician 2 1 7 5 2 6 9 : : : : : : n 2 2

    Q: Pearson correlation? A: Which student is X and which is Y ?

    Joshua Naranjo Questionnaire validity

  • Intraclass correlation coefficient

    ICC measures agreement between exchangeable variables (i.e. no inherent ordering). Used for test-retest reliability, and inter-rater reliability.

    Let

    x = 1

    2n

    ∑ (xi1 + xi2)

    s2 = 1

    2n

    {∑ (xi1 − x)2 +

    ∑ (xi2 − x)2

    } Then

    ICC = 1

    ns2

    ∑ (xi1 − x)(xi2 − x)

    Joshua Naranjo Questionnaire validity

  • Intraclass correlation coefficient

    For groups of three,

    x = 1

    2n

    ∑ (xi1 + xi2 + xi3)

    s2 = 1

    2n

    {∑ (xi1 − x)2 +

    ∑ (xi2 − x)2 +

    ∑ (xi3 − x)2

    } Then

    ICC = 1

    3ns2

    ∑ {(xi1 − x)(xi2 − x) + (xi1 − x)(xi3 − x)

    +(xi2 − x)(xi3 − x)}

    Joshua Naranjo Questionnaire validity

  • Cronbach alpha

    Internal Consistency: Hearing aid survey

    Suppose we want to measure how well the hearing aid “listens in conversation”

    Q: Do the questions “measure the same thing”?

    Joshua Naranjo Questionnaire validity

  • Cronbach’s alpha:

    Suppose that T = x1 + · · ·+ xk is the composite score for a factor or construct. Then

    α = k

    k − 1

    ( 1−

    ∑ var(xi )

    var(T )

    )

    Joshua Naranjo Questionnaire validity

  • Cronbach alpha

    α = 1010−1 ( 1− 19.02240.6933

    ) = 0.59172

    Joshua Naranjo Questionnaire validity

  • It may be easier to understand as follows:

    α = k

    k − 1

    ( 1−

    ∑ var(xi )

    var(T )

    ) =

    k

    k − 1

    ( var(T )−

    ∑ var(xi )

    var(T )

    )

    = k

    k − 1

    (∑ i 6=j cov(xi , xj)

    var(T )

    ) So α is an average covariance between the xi s.

    Joshua Naranjo Questionnaire validity

  • Cronbach alpha

    But this is still just a computing formula. In Cronbach Psychometrika, 1951, Cronbach proposes the statistic as a

    “....mean of all split-half coefficients resulting from different splittings of a test... therefore an estimate of the correlation between two random samples of items...”

    Joshua Naranjo Questionnaire validity

  • Examples

    Published example:

    Joshua Naranjo Questionnaire validity

  • Factor Analysis

    Joshua Naranjo Questionnaire validity

  • Factor Analysis

    Joshua Naranjo Questionnaire validity

  • Factor Analysis

    Joshua Naranjo Questionnaire validity

  • Factor Analysis

    Let Yi be the response to the ith question, i = 1, . . . , 15. The idea behind factor analysis is that the 15 responses actually depend on only two or three common underlying unobservable factors, i.e.

    Yi = µi + ai f1 + bi f2 + �

    History of factor analysis

    Charles Spearman (1904) two-factor theory of intelligence: general and specific

    Thurstone 7-factor: numerical, verbal comprehension, word fluency, perceptual speed, memory, inductive reasoning, spatial ability

    Sternberg 3-factor: analytical (problem-solving), creative (new ideas), and practical (everyday logic)

    Joshua Naranjo Questionnaire validity

  • Factor Analysis

    Back to article, let

     Y1 Y2 :

    Y15

     = 

    µ1 µ2 : µ15

    + 

    l11 l12 l21 l22 : :

    l15,1 l15,2

    ( f1f2 )

    +

     �1 �2 : �15

     or

    Y = µ+ LF + �

    where µ and L are constants, F and � have mean 0

    V (F) =

    ( 1 0 0 1

    ) , V (�) = Ψ =

     ψ1 0 . . . 0 0 ψ2 . . . 0 ...

    ... ...

    0 0 . . . ψ15

     Joshua Naranjo Questionnaire validity

  • so that V (Y) = LL′ + Ψ

    Estimation:

    S = 15∑ i=1

    λivivi ′ ∼=

    2∑ i=1

    λivivi ′ =

    (√ λ1v1

    √ λ2v2

    )( √λ1v1′√ λ2v2

    ) = L̂L̂′

    where λ1 > λ2 > · · · > λ15. The estimated coefficients of the underlying factors are derived from the eigenvalues and eigenvectors of S.

    Joshua Naranjo Questionnaire validity

  • Factor Analysis: Rotation

    Let A be any 2× 2 orthogonal matrix, i.e. AA′ = I. Then

    Y = µ+ LF + � = µ+ LAA′F + � = µ+ L∗F∗ + �

    so that there is an ambiguity to the factor analysis model. But this can be exploited, might as well choose A′F∗ = (f ∗1 f

    ∗ 2 ) ′ so that the

    factors are more interpretable. In practice, this means we want to rotate so that each of the 15 questions will have high coefficient in only one factor.

    Joshua Naranjo Questionnaire validity

  • Joshua Naranjo Questionnaire validity

  • Factor Analysis: Explained variance

    Recall that S = ∑15

    i=1 λivivi ′ ∼=

    ∑2 i=1 λivivi

    trace(S) = s21 + s 2 2 + · · · s215

    = λ1 + λ2 + · · ·+ λ15 The proportion of total variation explained by the ith factor is

    λi λ1 + λ2 + · · ·+ λ15

    Joshua Naranjo Questionnaire validity

  • Joshua Naranjo Questionnaire validity

    Questionnaire theory