Creating Internet Surveys and Collecting Data Using HTML Lecture 4, Psych 350 - R. Chris Fraley
Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris...
-
Upload
silvia-anthony -
Category
Documents
-
view
215 -
download
0
Transcript of Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris...
![Page 1: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/1.jpg)
Reliability, the Properties of Random Errors, and Composite Scores
Week 7, Psych 350 - R. Chris Fraleyhttp://www.yourpersonality.net/psych350/fall2015/
![Page 2: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/2.jpg)
Reliability
• Reliability: the extent to which measurements are free of random errors.
• Random error: nonsystematic mistakes in measurement– misreading a questionnaire item– observer looks away when coding behavior– response scale not quite fitting
![Page 3: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/3.jpg)
Reliability
• What are the implications of random measurement errors for the quality of our measurements?
![Page 4: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/4.jpg)
Psychometric Reliability
•O = T + E + SO = a measured score (e.g., performance on an exam)
T = true score (e.g., the value we want)
E = random error
S = systematic error
•O = T + E(we’ll ignore S for now, but we’ll return to it later)
![Page 5: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/5.jpg)
Reliability
• O = T + E• The error becomes a part of what we’re measuring• This is a problem if we’re using a single
measurement of a variable because part of our measurement is based on the true value that we want and part is based on error.
• Once we’ve taken a measurement, we have an equation with two unknowns. We can’t separate the relative contribution of T and E.10 = T + E
![Page 6: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/6.jpg)
Reliability: Do random errors accumulate?
• Question: If we aggregate or average multiple observations, will random errors accumulate?
![Page 7: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/7.jpg)
Reliability: Do random errors accumulate?
• Answer: No. If E is truly random, we are just as likely to overestimate T as we are to underestimate T.
• Height example
![Page 8: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/8.jpg)
5’2
5’3
5’4
5’5
5’6
5’7
5’8
5’9
5’10
5’11
6 6’1
6’2
6’3
6’4
6’5
6’6
6’7
6’8
6’9
62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81
![Page 9: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/9.jpg)
Reliability: Do random errors accumulate?
Note: The average of the seven O’s is equal to T
O = T + E Obs. 1 72 72 0 Obs. 2 71 72 -1 Obs. 3 72 72 0 Obs. 4 73 72 +1 Obs. 5 70 72 -2 Obs. 6 72 72 0 Obs. 7 74 72 +2
Average 72 72 0
![Page 10: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/10.jpg)
Composite scores
• These demonstrations suggest that one important way to help eliminate the influence of random errors is to aggregate or average multiple measurements of the same construct. Composite scores.
– use multiple questionnaire items in surveys of an attitude, behavior, or trait
– use more than one observer when coding behavior– use observer- and self-reports when possible
![Page 11: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/11.jpg)
• Example: Self-esteem survey items
• 1. I feel that I'm a person of worth, at least on an equal plane with others.Strongly Disagree 1 2 3 4 5 Strongly Agree
2. I feel that I have a number of good qualities.Strongly Disagree 1 2 3 4 5 Strongly Agree
4. I am able to do things as well as most other people. Strongly Disagree 1 2 3 4 5 Strongly Agree
![Page 12: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/12.jpg)
• Example: Self-esteem survey items
• 1. I feel that I'm a person of worth, at least on an equal plane with others.Strongly Disagree 1 2 3 4 5 Strongly Agree
2. I feel that I have a number of good qualities.Strongly Disagree 1 2 3 4 5 Strongly Agree
4. I am able to do things as well as most other people. Strongly Disagree 1 2 3 4 5 Strongly Agree
Composite self-esteem score = (4 + 5 + 3)/3 = 4
![Page 13: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/13.jpg)
Two things to note about aggregation
• Reverse Keyed Items• Some measurements are keyed in the
direction opposite of the construct of interest. High values represent low values on the trait of interest.
![Page 14: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/14.jpg)
• Example: Self-esteem survey items
• 1. I feel that I'm a person of worth, at least on an equal plane with others.Strongly Disagree 1 2 3 4 5 Strongly Agree
2. I feel that I have a number of good qualities.Strongly Disagree 1 2 3 4 5 Strongly Agree
3. All in all, I am inclined to feel that I am a failure.Strongly Disagree 1 2 3 4 5 Strongly Agree
4. I am able to do things as well as most other people. Strongly Disagree 1 2 3 4 5 Strongly Agree
5. I feel I do not have much to be proud of. Strongly Disagree 1 2 3 4 5 Strongly AgreeInappropriate composite self-esteem score =
(5 + 5+ 1 + 4 + 1)/5 = 3.2
![Page 15: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/15.jpg)
Reverse keying: Transform the measures such that high scores become low scores and vice versa.
• Example: Self-esteem survey items
• 1. I feel that I'm a person of worth, at least on an equal plane with others.Strongly Disagree 1 2 3 4 5 Strongly Agree
2. I feel that I have a number of good qualities.Strongly Disagree 1 2 3 4 5 Strongly Agree
3. All in all, I am inclined to feel that I am a failure.Strongly Disagree 1 2 3 4 5 Strongly Agree
4. I am able to do things as well as most other people. Strongly Disagree 1 2 3 4 5 Strongly Agree
5. I feel I do not have much to be proud of. Strongly Disagree 1 2 3 4 5 Strongly AgreeAppropriate composite self-esteem score =
(5 + 5+ 5 + 4 + 5)/5 = 4.8
![Page 16: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/16.jpg)
• You don’t want to do this by hand when you have data on multiple people.
• A simple algorithm for reverse keying X in SPSS or Excel:
New X = Max + Min - X
• Max represents the highest possible value (5 on the self-esteem scale). Min represents the lowest possible value (1 on the self-esteem scale).
![Page 17: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/17.jpg)
• Example: stress
Person Heart rate Complaints Average/composite
A 80 2 41
B 80 3 42
C 120 2 61
D 120 3 62
Cautions: Two potential problems with aggregation
![Page 18: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/18.jpg)
• Example: stress
Person Heart rate Complaints Average/composite
A 80 2 41
B 80 3 42
C 120 2 61
D 120 3 62
Cautions: Two potential problems with aggregation
The first problem is that the metric for the composite doesn’t make much sense.
Person A: 2 complaints + 80 beats per minute = 41 complaints/beats per
minute???
![Page 19: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/19.jpg)
Two things to note about aggregation
• Second, the variables may have different variances.
• If this is true, then some indicators will “count” more in the average than others.
![Page 20: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/20.jpg)
• Example: stress
Person Heart rate Complaints Average/composite
A 80 2 41
B 80 3 42
C 120 2 61
D 120 3 62
Beats per minuteNumber of complaints
The correlation between the composite and HR is .99. The correlation between the composite and Complaints is .05.
![Page 21: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/21.jpg)
Two things to note about aggregation
• One common solution to these problems is to standardize the variables before aggregating them.
• Constant mean and variance
![Page 22: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/22.jpg)
• Standardization helps solve the problem that variables with a large range/variance will influence the composite score more than variable with a small range.
Person Heart rate(z) Complaints(z) Average
A -.87 -.87 -.87
B -.87 .87 0
C .87 -.87 0
D .87 .87 .87
The correlation between the composite and HR is .71. The correlation between the composite and Complaints is .71.
![Page 23: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/23.jpg)
Reliability: Estimating reliability
• Question: How can we quantify the reliability of our measurements?
• Answer: Two common ways:(a) test-retest reliability
(b) internal consistency reliability
![Page 24: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/24.jpg)
Reliability: Estimating reliability
• Test-retest reliability: Reliability assessed by measuring something at least twice at different time points. Test-retest correlation.
• The logic is as follows: If the errors of measurement are truly random, then the same errors are unlikely to be made more than once. Thus, to the degree that two measurements of the same thing agree, it is unlikely that those measurements contain random error.
![Page 25: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/25.jpg)
Less error(off by 1 point)
More error(off by 2 points)
Time1
Time2
Time1
Time2
Person A 1 2 1 3Person B 7 6 7 5Person C 2 3 2 4Person D 6 5 6 4Person E 3 4 3 5Person F 5 4 5 3Person G 4 5 4 6
r = .92 r = .27
![Page 26: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/26.jpg)
Reliability: Estimating reliability
• Internal consistency: Reliability assessed by measuring something at least twice within the same broad slice of time.
Split-half: based on an arbitrary split (e.g, comparing odd and even, first half and second half). Split-half correlation.
Cronbach’s alpha (): based on the average of all possible split-half correlations.
![Page 27: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/27.jpg)
Relationship between reliability and number of measurements
Ave r = .10
Ave r = .25
Ave r = .50 The reliability of the composite (a) increases as the number of measurements (k) increases.
In fact, the reliability of the composite can get relatively high even if the items themselves do not correlate strongly.
![Page 28: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/28.jpg)
Ave r = .10
Ave r = .10
![Page 29: Reliability, the Properties of Random Errors, and Composite Scores Week 7, Psych 350 - R. Chris Fraley](https://reader035.fdocuments.net/reader035/viewer/2022062517/56649f335503460f94c50575/html5/thumbnails/29.jpg)
Reliability: Final notes
• An important implication: As you increase the number of measures, the amount of random error in the averaged measurement decreases.
• An important assumption: The entity being measured is not changing.
• An important note: Common indices of reliability range from 0 to 1—in the metric of correlation coefficients; higher numbers indicate better reliability (i.e., less random error).