Informant Accuracy

Christopher McCarty

The Problem

• Social network data often rely on cognitive reports such as ‘Name the three people you interact with most in this group’

• How closely do these cognitive reports represent behavior?

• Does accuracy change based on the structural property a researcher is trying to describe?

Informant Accuracy Study 1Peter D. Killworth and H. R. Bernard (1976) Informant Accuracy in Social Network Data. Human

Organization 35 (3): 269-286

• 21 people from a regional group of deaf respondents who relied on teletype to communicate with other group members

• Number of lines to and from each member were recorded (behavior)

• Respondent ranked all 31 members in order of their perceived (cognitive)

• The reported most communicated with was only in rank 1-4 52% of the time

Only 24% of the time will a person who ranks j 5th will they actually talk to them 2nd-8th

Informant Accuracy Study 2H. R. Bernard and Peter D. Killworth (1977) Human Communication Research 4 (1):

3-18

• Four data sets1. Deaf – New data set of teletype data from two groups of 30 deaf

respondents in Washington DC. Pre-cognitive and post-cognitive data

2. Hams - 44 Ham radio operators with recorded lines transmissions, and 0-9 scale assessment of interaction with each of 54 users

3. Office – Cognitive data from 40 people in an office with observed (behavioral) data about 44. Observers walked through office every 15 minutes for five hours every workday

4. Tech – 37 peopel in a graduate technology program at WVU were observed (behavioral) and reported interaction (cognitive)

Conclusions Study 2• People do not know, with any accuracy, those with whom they communicate

• Post-cognitive data more accurate than pre-cognitive, but not significantly

• Except for first ranked, no difference between a typical week and last week

• No difference between data sets

• No demographics that explain inaccuracy (e.g. women are no more accurate than men)

• Keeping logs does not improve accuracy

• Adjusting by respondents’ confidence in their assessment makes no difference

Analysis of BK data in UCINET

Informant Accuracy Study 3Peter D. Killworth and H. Russell Bernard (1979) Informant Accuracy in Social Network Data III: A Comparison of

Triadic Structure in Behavioral and Cognitive Data, Social Networks 2: 10-46

Number/proportion of triads reported correctly

Assumption that behavior changes because cognition changes is dangerous

• Researcher returning at time 2 would likely find different triads and conclude something had changed

• Change could be from inaccurate report from time 1 to an inaccurate report at time 2

Conclusions Study 31. A lot of significant structure in both the behavioral and cognitive data

(what triadic structures don’t exist?)

2. Structures from ranked behavioral data are similar to ranked cognitive data; and scaled behavioral data are similar to scaled cognitive data; but structures produced by ranking and those produced by scaling are different

3. A 76% level of scatter exists between behavioral and cognitive structures when compared triad by triad

4. Null hypothesis that cannot be rejected: Social structure never changes; only reports of it by informants changes due to informant inaccuracy

Informant Inaccuracy Study 4H. R. Bernard, Peter D. Killworth and Lee Sailer (1979) Informant Accuracy in Social Network Data IV: A

Comparison of Clique-Level Structure in Behavioral and Cognitive Network Data, Social Networks 2: 191-218

• Compared the same four data sets using four clique-finding algorithms:– Factor Analysis– CONCOR– COMPLT

Never more than 50% concordance (Hams on COMPLT) between recalled and observed cliques

Conclusions Study 4

• At dyadic level 50% inaccuracy found (Study 2)

• At triadic level 76% inaccuracy found (Study 3)

• At the clique level 160% inaccuracy was found (Study 4)

• “We must therefore recommend unreservedly that any conclusions drawn from data gathered by the question “Who do you talk to?” are of no use in understanding social structure or communication.”

Informant Accuracy Study 5H.R. Bernard, Peter D. Killworth and Lee Sailer (1982) Informant Accuracy in Social-Network Data V. An

Experimental Attempt to Predict Actual Communication from Recall Data, Social Science Research 11:30-66

• Examination of recall of communication over time

• Are respondents more accurate depending on the time period reported on?

• Does a respondent’s position within the network determine accuracy?

Data

• Electronic Information Exchange System (EIES), the precursor to e-mail

• Quotes Page 35 and 36

• 57 paid users reported on their communication in each of 37 windows, including their “last-on”

• Respondents also rated their confidence in their rating

-Of those recalled, 30% were not communicated with (T1P)-Two thirds of the people a respondent received messages from were forgotten (T2P)-79% of those recalled by a respondent are wrong in some way (T12AB)-More than 52% of the time respondents chose the wrong most frequent (TOP1)-More than 40% of the top 3 should not be in the top 3 (TOP3)

Conclusion to Study 5

• Respondents are inaccurate at reporting no matter the lag in time

• Respondents were not accurate at communicating with nobody

• However, while individual respondents were not accurate, en masse reports match behavior about broad facts, such as the person spoken to most frequently

Linton C. Freeman, A. Kimball Romney and Sue Freeman (1987) Cognitive Structure and Informant

Accuracy, American Anthropologist, 89: 310-325

• Data collected from colloquium series at UCI, 9 sessions

• 33 people interviewed about whether they had attended the last session, and to name the others who attended

• Observers recorded who actually attended

Results• 141 recall errors out of 272 opportunities (51.8%) (very close to BKS

results)

• However reports were better at describing the long-term pattern than the detail about individual attendees

• Errors were biased in the direction of long-term patterns, similar to BKS findings that en masse the group gets some things right

• Some people were better than others at describing the pattern, presumably those who attended regularly and were part of the in-group

• Used Consensus Analysis to analyze results

Informant Accuracy Overall Conclusions

• When asking respondents who they talk to in a bounded (whole) network, when they talk to them and how frequently they talk to them, one should be careful

• Are people more accurate at reporting their personal network interactions?