Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive...
-
Upload
anastasia-marsh -
Category
Documents
-
view
217 -
download
1
Transcript of Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive...
![Page 1: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/1.jpg)
Informant Accuracy
Christopher McCarty
![Page 2: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/2.jpg)
The Problem
• Social network data often rely on cognitive reports such as ‘Name the three people you interact with most in this group’
• How closely do these cognitive reports represent behavior?
• Does accuracy change based on the structural property a researcher is trying to describe?
![Page 3: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/3.jpg)
Informant Accuracy Study 1Peter D. Killworth and H. R. Bernard (1976) Informant Accuracy in Social Network Data. Human
Organization 35 (3): 269-286
• 21 people from a regional group of deaf respondents who relied on teletype to communicate with other group members
• Number of lines to and from each member were recorded (behavior)
• Respondent ranked all 31 members in order of their perceived (cognitive)
• The reported most communicated with was only in rank 1-4 52% of the time
![Page 4: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/4.jpg)
![Page 5: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/5.jpg)
Only 24% of the time will a person who ranks j 5th will they actually talk to them 2nd-8th
![Page 6: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/6.jpg)
![Page 7: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/7.jpg)
![Page 8: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/8.jpg)
Informant Accuracy Study 2H. R. Bernard and Peter D. Killworth (1977) Human Communication Research 4
(1): 3-18
• Four data sets1. Deaf – New data set of teletype data from two groups of 30
deaf respondents in Washington DC. Pre-cognitive and post-cognitive data
2. Hams - 44 Ham radio operators with recorded lines transmissions, and 0-9 scale assessment of interaction with each of 54 users
3. Office – Cognitive data from 40 people in an office with observed (behavioral) data about 44. Observers walked through office every 15 minutes for five hours every workday
4. Tech – 37 peopel in a graduate technology program at WVU were observed (behavioral) and reported interaction (cognitive)
![Page 9: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/9.jpg)
![Page 10: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/10.jpg)
![Page 11: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/11.jpg)
![Page 12: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/12.jpg)
Conclusions Study 2• People do not know, with any accuracy, those with whom they communicate
• Post-cognitive data more accurate than pre-cognitive, but not significantly
• Except for first ranked, no difference between a typical week and last week
• No difference between data sets
• No demographics that explain inaccuracy (e.g. women are no more accurate than men)
• Keeping logs does not improve accuracy
• Adjusting by respondents’ confidence in their assessment makes no difference
![Page 13: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/13.jpg)
Analysis of BK data in UCINET
![Page 14: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/14.jpg)
Informant Accuracy Study 3Peter D. Killworth and H. Russell Bernard (1979) Informant Accuracy in Social Network Data III: A Comparison
of Triadic Structure in Behavioral and Cognitive Data, Social Networks 2: 10-46
![Page 15: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/15.jpg)
Number/proportion of triads reported correctly
![Page 16: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/16.jpg)
Assumption that behavior changes because cognition changes is dangerous
• Researcher returning at time 2 would likely find different triads and conclude something had changed
• Change could be from inaccurate report from time 1 to an inaccurate report at time 2
![Page 17: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/17.jpg)
Conclusions Study 31. A lot of significant structure in both the behavioral and cognitive
data (what triadic structures don’t exist?)
2. Structures from ranked behavioral data are similar to ranked cognitive data; and scaled behavioral data are similar to scaled cognitive data; but structures produced by ranking and those produced by scaling are different
3. A 76% level of scatter exists between behavioral and cognitive structures when compared triad by triad
4. Null hypothesis that cannot be rejected: Social structure never changes; only reports of it by informants changes due to informant inaccuracy
![Page 18: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/18.jpg)
Informant Inaccuracy Study 4H. R. Bernard, Peter D. Killworth and Lee Sailer (1979) Informant Accuracy in Social Network Data IV: A
Comparison of Clique-Level Structure in Behavioral and Cognitive Network Data, Social Networks 2: 191-218
• Compared the same four data sets using four clique-finding algorithms:– Factor Analysis– CONCOR– COMPLT
![Page 19: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/19.jpg)
Never more than 50% concordance (Hams on COMPLT) between recalled and observed cliques
![Page 20: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/20.jpg)
Conclusions Study 4
• At dyadic level 50% inaccuracy found (Study 2)
• At triadic level 76% inaccuracy found (Study 3)
• At the clique level 160% inaccuracy was found (Study 4)
• “We must therefore recommend unreservedly that any conclusions drawn from data gathered by the question “Who do you talk to?” are of no use in understanding social structure or communication.”
![Page 21: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/21.jpg)
Informant Accuracy Study 5H.R. Bernard, Peter D. Killworth and Lee Sailer (1982) Informant Accuracy in Social-Network Data V. An
Experimental Attempt to Predict Actual Communication from Recall Data, Social Science Research 11:30-66
• Examination of recall of communication over time
• Are respondents more accurate depending on the time period reported on?
• Does a respondent’s position within the network determine accuracy?
![Page 22: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/22.jpg)
Data
• Electronic Information Exchange System (EIES), the precursor to e-mail
• Quotes Page 35 and 36
• 57 paid users reported on their communication in each of 37 windows, including their “last-on”
• Respondents also rated their confidence in their rating
![Page 23: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/23.jpg)
-Of those recalled, 30% were not communicated with (T1P)-Two thirds of the people a respondent received messages from were forgotten (T2P)-79% of those recalled by a respondent are wrong in some way (T12AB)-More than 52% of the time respondents chose the wrong most frequent (TOP1)-More than 40% of the top 3 should not be in the top 3 (TOP3)
![Page 24: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/24.jpg)
Conclusion to Study 5
• Respondents are inaccurate at reporting no matter the lag in time
• Respondents were not accurate at communicating with nobody
• However, while individual respondents were not accurate, en masse reports match behavior about broad facts, such as the person spoken to most frequently
![Page 25: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/25.jpg)
Linton C. Freeman, A. Kimball Romney and Sue Freeman (1987) Cognitive Structure and Informant
Accuracy, American Anthropologist, 89: 310-325
• Data collected from colloquium series at UCI, 9 sessions
• 33 people interviewed about whether they had attended the last session, and to name the others who attended
• Observers recorded who actually attended
![Page 26: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/26.jpg)
Results• 141 recall errors out of 272 opportunities (51.8%) (very close to BKS
results)
• However reports were better at describing the long-term pattern than the detail about individual attendees
• Errors were biased in the direction of long-term patterns, similar to BKS findings that en masse the group gets some things right
• Some people were better than others at describing the pattern, presumably those who attended regularly and were part of the in-group
• Used Consensus Analysis to analyze results
![Page 27: Informant Accuracy Christopher McCarty. The Problem Social network data often rely on cognitive reports such as ‘Name the three people you interact with.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f535503460f94c77ebd/html5/thumbnails/27.jpg)
Informant Accuracy Overall Conclusions
• When asking respondents who they talk to in a bounded (whole) network, when they talk to them and how frequently they talk to them, one should be careful
• Are people more accurate at reporting their personal network interactions?