Measurement Class Presentation - DePaul...
Transcript of Measurement Class Presentation - DePaul...
![Page 1: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/1.jpg)
Farzad Eskandanian
![Page 2: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/2.jpg)
§Measurement is the process of observing and recording the observations.
§Two important issues:1. Understanding the fundamental ideas:§ Levels of measurement: nominal, ordinal, interval and ratio§ Reliability
2. Types of measures:§ Survey research: design of interviews and questionnaires.§ Scaling: methods developing a scale§ Qualitative research: non-numerical measurement approaches.
![Page 3: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/3.jpg)
§ Generalizing from your program or measures to the concept of your program or measures.
§ A labeling issue. § Examples:
§ A “Head Start” program. Is the label accurate?§ A measure that you term "self esteem" is that what you were really measuring?
§ The degree to which a test measures what it claims, to be measuring.
![Page 4: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/4.jpg)
§ We really want to talk about the validity of any operationalization.§ Operationalization?!§ Any time you translate a concept or construct into a functioning and
operating reality, § Be concerned about how well you did the translation.
§ Construct validity is the approximate truth of the conclusion that your operationalization accurately reflects its construct.
![Page 5: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/5.jpg)
§ Different ways you can demonstrate different aspects of construct validity:
§ Translation Validity: degree to which you accurately translated your construct into the operationalization.§ Face Validity:
§ Look at the operationalization and see whether "on its face" it seems like a good translation of the construct.
§ Content Validity:§ Check the operationalization against the relevant content domain for the construct.§ It is not always easy to decide on the criteria that constitute the content domain.
![Page 6: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/6.jpg)
§ Checks the performance of your operationalization against some criterion.§ make a prediction about how the operationalization will perform based on our theory
of the construct.§ Types:
§ Predictive validity:§ Assess the operationalization's ability to predict something it should theoretically be able to predict.
§ Concurrent validity:§ Assess the operationalization's ability to distinguish between groups that it should theoretically be
able to distinguish between.§ Convergent validity:
§ Examine the degree to which the operationalization is similar to other operationalizations that it theoretically should be similar to.
§ Discriminant validity§ Examine the degree to which the operationalization is not similar to other operationalizations that it
theoretically should be not be similar to.
![Page 7: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/7.jpg)
![Page 8: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/8.jpg)
• Idea of Construct Validity:• Definitionalist: Precise absolute definitions.• Rationalist: Meanings differs relatively, Not
absolutely. • In court
• Tell "the truth, the whole truth and nothing but the truth.”
• In our context:• Our measure should reflect "the construct, the
whole construct, and nothing but the construct.”
• Example: "self esteem, all of self esteem, and nothing but self esteem?"
![Page 9: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/9.jpg)
§ To establish construct validity the following conditions are required:§ Operationalize within a semantic net.
§ Control the operationalization of the construct, so it looks similar to what you theoretically mean.
§ Provide evidence that your data support your theoretical view of the relations among constructs.
![Page 10: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/10.jpg)
§ Show that:§ Correspondence or convergence between similar constructs, and§ Discriminate between dissimilar constructs.
§ Correlations between theoretically similar measures should be "high”.§ While correlations between theoretically dissimilar measures should be
"low”.§ Note convergent correlations should always be higher than the
discriminant ones.
![Page 11: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/11.jpg)
§ We theorize that all four items reflect the idea of self esteem.
§ Observations show the intercorrelations of four items.
§ Pattern of correlations states that the four items are converging on the same idea (construct).
![Page 12: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/12.jpg)
§ Show that measures that should not be related in reality are not related.
![Page 13: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/13.jpg)
![Page 14: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/14.jpg)
§ Inadequate Preoperational Explication of Constructs§ You didn't do a good enough job of defining (operationally) what you mean by the construct.§ Think more about the concepts.§ Use concept mappings and experts’ opinions.
§ Mono-Operation Bias§ If you only use a single version of a program in a single place at a single point in time, then
you are not capturing the whole picture.§ Solution: try to implement multiple versions of your program.
§ Mono-Method Bias§ refers to your measures or observations, not to your programs or causes.§ Solution: try to implement multiple measures of key constructs
![Page 15: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/15.jpg)
§ Interaction of Different Treatments§ Can you really label the program effect as a consequence of your program?
§ Interaction of Testing and Treatment§ Restricted Generalizability Across Constructs
§ Unintended consequences of the program.
§ Confounding Constructs and Levels of Constructs§ Your label is not a good description for what you implemented.
§ Social Threats:§ Hypothesis Guessing§ Evaluation Apprehension§ Experimenter Expectancies
![Page 16: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/16.jpg)
§ To provide evidence that your measure has construct validity.
§ This model links the conceptual/theoretical realm with the observable one, because this is the central concern of construct validity.
![Page 17: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/17.jpg)
§ A correlation matrix.§ The Reliability Diagonal
(monotrait-monomethod)§ The Validity Diagonals
(monotrait-heteromethod)§ The Heterotrait-Monomethod
Triangles§ Heterotrait-Heteromethod
Triangles§ The Monomethod Blocks§ The Heteromethod Blocks
![Page 18: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/18.jpg)
§ Linking two patterns:§ Theoretical and§ Observational
§ A test of significance is usually required:§ t-test§ ANOVA
![Page 19: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/19.jpg)
§ About quality of measurement. § Reliability is the "consistency" or "repeatability" of
your measures.§ True Score Theory
§ True ability + Random error
§ A measure that has no random error (is all true score) is perfectly reliable.
![Page 20: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/20.jpg)
§ Random error or Noise.§ Systematic error or Bias.
![Page 21: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/21.jpg)
![Page 22: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/22.jpg)
• Usually we don’t know about true score.
• Only observation X.• Using two observations we can
see the true score is shared.• But we can’t calculate the
variance of true score, so we estimate it.
• 𝑐𝑜𝑟𝑟 𝑋%, 𝑋' = *+,(./,.0)23 ./ ∗23(.0)
![Page 23: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/23.jpg)
§ Inter-Rater or Inter-Observer Reliability§ Used to assess the degree to which different raters/observers give consistent
estimates of the same phenomenon.
§ Test-Retest Reliability§ Used to assess the consistency of a measure from one time to another.
§ Parallel-Forms Reliability§ Used to assess the consistency of the results of two tests constructed in the same
way from the same content domain.
§ Internal Consistency Reliability§ Used to assess the consistency of results across items within a test.
![Page 24: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/24.jpg)
§ Think of the center of the target as the concept that you are trying to measure.
![Page 25: Measurement Class Presentation - DePaul Universityfacweb.cs.depaul.edu/.../classes/csc426/notes/CSC426_S2017_Meas… · §Measurement is the process of observing and recording the](https://reader033.fdocuments.net/reader033/viewer/2022052712/5af654517f8b9a9e598f3c93/html5/thumbnails/25.jpg)