What is the Imitation Game?

Robert Evans and Harry Collins

Centre for the Study of Knowledge, Expertise and Science (KES)Cardiff School of Social Sciences

http://www.cardiff.ac.uk/socsi/expertise

Overview

Theory and Background Different Kinds of Expertise Turing Test

Imitation Games in Practice Software and data Identify and Chance conditions Qualitative and Quantitative Results

Comparative Research Cross-cultural and longitudinal research

Understanding societies

http://bmrc.berkeley.edu/courseware/cs298/spring99/w9/slides/sld006.htm

Three Kinds of Expertise

Contributory expertise: enables those who have acquired it to contribute linguistically and practically to the community through the expertise is sustained. The most common usage of the word ‘expert’.

Interactional Expertise: expertise in the language of a specialism in the absence of expertise in its practice. Like contributory expertise, it requires the tacit-knowledge acquired by immersion in a form-of-life (i.e. socialisation). It enables individuals to talk as if they had contributory expertise even though they lack practical or craft skills.

No Expertise: not even the ability to talk intelligently about a given domain or topic

Collins and Evans (2002)

Turing Tests and Imitation Games

In the Turing Test, the judge must decide which is the

computer and which is the human.

In the Imitation Game, the judge must decide which participant shares their social group.

Modern Imitation Game

Female judge setting questions

Female answering naturally

Male pretending to be female

How often do you pluck your eyebrows?

R2‘not very often, when

they need doing’

R1‘once a week’

R2 is female ‘because I expected the man to believe women are more regulated in their beauty regime than they actually are

Judge compares answers

Judge’s question

Respondents Answers

Judge’s verdict

Judge’s confidence

Judge’s reason

Automatically recorded data

Software records Questions and answers

analysis of topics and assumptions used by judges (i.e. what is distinctive about own culture)

Guesses and confidences what discriminates and what does not (i.e.

what is shared with others and what is not) Judge’s reasons

cultural norms and variation (i.e. what counts as a ‘mistake’ and what as normal variation)

Researcher generated data

Researchers needed to record Problems with method and software

(e.g. how to use, do participants understand role etc)

Strategies used by judges and pretenders (e.g. what is hypothesis being tested by judge, how does pretender ‘know’ the answer)

Effort and determination (e.g. judge’s motives and fears)

Imitation Games with the Blind

Identify Condition Judge is blind Target expertise is

‘being blind’ Sighted participant

has to pretend to be blind

Prediction: Judge will succeed in identifying participants

Chance Condition Judge is sighted Target expertise is

‘being sighted’ Blind participant has

to pretend to be sighted

Prediction: Blind person will succeed in pretending

Sighted dialogs

Judge R1 R2

Could you tell me roughly how old you are and whether you have been registered blind since birth

I'm 50 and have been blind since I was 10

I'm 30 and I've been registered blind since I was

Do you have any residual sight and what mobility aids do you use?

No I have no residual sight. I use a white stick and have a guide dog

Twelve. I've got light and dark and color perception in one eye and I use a guide dog.

Judges’ reasons

Question One The second person is not black and white and you

do not usually lose your sight over-night, so the fact they mention being registered suggest that they are blind. If the first one was blind they would normally say how they became blind if it was sudden. (level 2)

Question Two I have both white stick and dog but would never use

both at same time. Therefore if I was responding I would say something like I use a guide dog predominantly but sometimes use a white stick -- but if you are blind you would call it a cane normally. Also, number 2 was much less black and white. It's always grades of blindness. (level 4)

Blind dialogs

Judge R1 R2

So let me start with sport. Are you interested in tennis and do you ever watch it on the television?

I watch Wimbledon a little bit on the television and occasionally the Australian Open in January

I like tennis but only watch big tournaments like Wimbledon

So tell me what you think about the Hawk-Eye line judging system

Not being a tennis professional it is not for me to say if it should or should not be used. It does not really alter viewing

It adds an other element to the game which could make it more interesting

Blind dialogs

Judge R1 R2

How accurately would you say a human can judge the flight of a tennis-ball? I mean, would you say they could tell the difference between touch the line and 1mm out 2mm out 1 cm out, 2 cm out, or what, and what would it depend on?

I think often a tennis player is not in a position to judge accurately as they are not usually parallel with the line. I think that if you set up a test for a line judge with two balls one which landed on the line and one which landed 1mm away from the line, I don't think they could tell the difference. If you think how small 1mm is then it would be so hard for them to judge.

it would depend on the speed the ball was travelling and the position of the judge relative to the line and obviously the closer the ball is the line the harder it would be to make a judgement. So you would have to judge each call on an individual bases as there are a lot of factors.

Judges’ reasons

R1 is the blind person ‘I think respondent 1 gives himself away when he

discusses the human judgments on the flight of a tennis ball.’

‘I cannot believe a sighted person saying that Hawk-eye does not alter the viewing.’

R2 is the blind person ‘The Hawk-Eye questions reveal some quite specific

information that I don’t think was published in audio media. Also, the story wasn’t that important that I’d expect it to be picked up by the audio news services provided to the blind.’

‘person 2 seems really unfamiliar with hawk-eye, given that they say they watch Wimbledon’

Interpreting Results

The Identification Ratio measures the excess of correct identifications

Identify Conditions

0.33

0.730.86

0.44

0.68

-0.20

0.00

0.20

0.40

0.60

0.80

1.00

colour blind pitch blind sexuality religion

Topics

Ide

nti

fic

ati

on

Ra

tio

ID Ratio Net Don't Know

Comparisons across topics

Mean IR for ID condition approx 0.6

Mean IR for Chance condition approx zero

Chance Conditions

0.05 0.000.13

-0.01-0.13

-0.20

0.00

0.20

0.40

0.60

0.80

1.00

colour blind pitch blind sexuality religion

Topics

Ide

nti

fic

ati

on

Ra

tio

ID Ratio Net Don't Know

IG and Comparative Research

Social Change and Social Difference How to standardise data collection? How to reflect local variations?

Imitation Game as ‘ethno-method’ Participants’ discourse is always contemporary Judge as proxy ethnographer Participants as proxy stranger Participant errors like breaching experiments

Imitation Game as secondary data Identification ratios by time, place and topic Qualitative data based on topics, reasons, errors

Imitation Games on Gender

Hypotheses Malestream culture: hegemonic

masculinity implies asymmetrical results

1960s and all that: gender roles and identities more flexible so knowledge widely shared

Age as confounding factor Younger people have less interaction The 1960s are history?

Research Design

Research Methods module project Recruit students and heterosexual,

cohabiting parents Phase One

Students: 29 games (15 F, 14 M) Parents: 25 games (12 F, 13 M)

Phase Two Students: 70 games (40 F, 30 M) Parents: 70 games (50 F, 20 M)

Results by gender  Female Male

Incorrect 11 12

Don't Know 15 10

Correct 29 22

Total 55 44

IR 0.33 0.23

  Female Male

Incorrect 21 13

Don't Know 19 9

Correct 22 11

Total 62 33

IR 0.02 -0.06

Phase One and Two, students

Approximates identify condition for both groups, female judges appear more successful than male judges but not statistically significant (p = 0.37)

Phase One and Two, students

Approximates chance condition for both groups, no statistically significant difference between genders (p = 0.26)

Results by age  Young Old

Incorrect 6 9

Don't Know 5 4

Correct 18 12

Total 29 25

IR 0.41 0.12

  Young Old

Incorrect 17 25

Don't Know 20 24

Correct 33 21

Total 70 70

IR 0.23 -0.06

Phase One only

Clear difference between groups; young judges more successful than older judges (p = 0.031)

Phase Two only

Clear difference between groups; young judges more successful than older judges (p = 0.007)

Conclusions for gender

Differences by gender No statistically significant differences in sample

as a whole Some differences in topics chosen to

‘represent’ and test cultures Clear differences by age

Older participants much better at pretending (i.e. have more interactional expertise)I

Impossible to distinguish between ‘1960s’ hypothesis and the ‘learning’ hypothesis without longitudinal study

Conclusions

Interactional expertise Useful concept for explaining social phenomena

Imitation Game Combines qualitative and quantitative

approaches Opportunities for comparative research within

cultures, across cultures, over time Want to know more

http://www.cardiff.ac.uk/socsi/expertise

References and reading

Collins, Harry and Robert Evans (2002) ‘The Third Wave of Science Studies: Studies of Expertise and Experience’, Social Studies of Sciences, 32 (2): 235-96.

Collins, Harry and Robert Evans (2007) Rethinking Expertise, Chicago, IL: The University of Chicago Press.

Collins, Harry, Robert Evans, Rodrigo Ribeiro and Martin Hall (2006), ‘Experiments with Interactional Expertise, Studies In History and Philosophy of Science, Volume 37, No. 4 (Dec 2006), pp. 656-674.

Evans, Robert and Harry Collins (forthcoming, 2010) ‘Interactional Expertise and the Imitation Game’ in Michael Gorman (ed) Trading Zones and Interactional Expertise: Creating New Kinds of Collaboration, Chicago, IL: MIT Press