Analysis of Group Estimation Study(data from Goldstone and Gold)

preliminary findings

Experiment 1

2

Dot estimation task

Experiment 1

Estimate the percentage of red dots in multiple rounds Solo condition Group condition: individuals observe estimates of other individuals

Procedure Round 1: individuals estimate independently of other individuals Round 2 and 3: individuals see same display again

in group condition, individuals also see estimates of other individuals from previous round

Each individual/group was tested on 36 displays with each 3 rounds. The displays consisted of 4 repetitions of 10,20,…,90% of red dots

giving 36 different displays

3

Round 1: what is the percentage of red circles?

4

Round 2: what is the percentage of red circles?

5

Individual 1: 75Individual 2: 85Individual 3: 72Individual 4: 78

Estimates from previous round

Number of individuals per group

6

1 2 3 4 5 6 7 8 9 10 110

1

2

3

4

5

6

7

8

Group

Num

ber

of

pe

ople

in g

rou

p

MAD per individual over rounds

7

0 10 20 30 40 50 60 702

4

6

8

10

12

14

16

18

Individuals (sorted by mean error)

Mea

n e

rror

ove

r q

uest

ion

s

round 1round 2round 3

MAD per question over rounds

8

0 50 100 150 200 250 300 3500

5

10

15

20

25

30

35

Questions (sorted by mean error)

Mea

n e

rror

ove

r q

uest

ion

s

round 1round 2round 3

why are somequestions so mucheasier than others?

Individual estimates across rounds for different displays

9

0 50 1000

50

100

150

10%

0 50 1000

20

40

60

80

100

20%

0 50 1000

20

40

60

30%

0 50 1000

20

40

60

80

10040%

0 50 1000

50

100

50%

0 50 1000

20

40

60

60%

0 50 1000

20

40

60

70%

0 50 1000

20

40

60

80

80%

0 50 1000

50

100

150

90%

(note: across groups too)

10

0 20 40 60 80 100

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Que

stio

ns

Group 1

0 20 40 60 80 100

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

Que

stio

ns

Group 1

all raw data from one group with six individuals: in each box we see the responses for three rounds (bottom row in each box is round 1). Dashed line represents ground truth

11

0 20 40 60 80 100

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

Que

stio

ns

Group 2

0 20 40 60 80 100

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

Que

stio

ns

Group 2

Higher accuracy on round 1 when individuals do not change their estimates in subsequent rounds

12

0 20 40 60 80 1000

10

20

30

40

50

60

70

80

90

Mean estimate change from round 1 to round 2

Mea

n e

rror

at

rou

nd 1

Correlation = 0.708

0 10 20 30 40 50 600

10

20

30

40

50

60

70

80

90

Std estimates across round

Mea

n e

rror

at

rou

nd 1

Correlation = 0.751

If people change their estimate, they generally increase their accuracy

13

0 20 40 60 80 100-20

0

20

40

60

80

100

Change in estimate from round 1 to round 2

MA

D(

rou

nd 1

) -

MA

D(

roun

d 2

)Correlation = 0.875

Analyzing Trivia Experiment

(Experiment 2)

14

Experiment 2

Estimate percentages for trivia questions in multiple rounds

Solo and group conditions 21 groups

Similar setup as in experiment 1 36 different trivia questions three estimation rounds per question

Major difference with experiment 1 No gain in perceptual information between rounds (no opportunity

to extract more information from same display)

15

Number of Individuals in 21 groups (Exp 2)

16

0 5 10 15 200

1

2

3

4

5

6

7

8

9

10

Group

Num

ber

of

pe

ople

in g

rou

p

Trivia Questions1. "What percent of people in the U.S. over the age of 25 have obtained a college degree?"

2. "What percent of U.S. women age 25 and older have completed high school?"

3. "What percent of the worldwide use of energy is by the United States?"

4. "What percent of the worldwide population lives in the United States?"

5. "What percent of U.S. energy consumption is with fossil fuels?"

6. "In 2000, the richest 1% of adults owned what percentage of the world's total assets?"

7. "What percentage of Americans are age 0-14?"

8. "What percentage of Americans are Protestant?"

9. "What percentage of Americans are black?"

10. "What percentage of Americans over the age of 15 can read and write?"

11. "What percentage of the American population is below the poverty line?"

12. "In 2005, what was the U.S. military expenditure as percent of the national income?"

13. "What percentage of people in the world are age 0-14?"

14. "What percentage of people in the world are Christian?"

15. "Across the world, what percentage of people's first language is Mandarin (a dialect of Chinese)?"

16. "Across the world, what percentage of people over the age of 15 can read and write?"

17. "Across the world, what percentage of the world labor force is in agriculture (not industry or services)?"

18. "What percentage of the world's surface is water (not land)?"

19. "What percentage of people in the world are aged 65 or over?"

20. "What percentage of people in the world are Muslim?"

21. "What percentage of college freshmen have never used a typewriter?"

22. "What percentage of the U.S. population will eat macaroni and cheese sometime in the next two weeks?"

23. "What percentage of Americans drink a soda pop nearly every day with breakfast?"

24. "In the United States, what percentage of households routinely speak a language other than English?"

25. "In the United States, what percentage of firms are owned by women?"

26. "What percentage of American men serve time in prison at some point in their life?"

27. "What percentage of Americans are regular cigarette smokers?"

28. "What percentage of Americans have no health insurance?"

29. "What percentage of children diagnosed with cancer make a full recovery?"

30. "What percentage of a woman's body consists of water?"

31. "What percentage of people grind their teeth while sleeping?"

32. "What percentage of the earth's water is fresh (not salt) water?"

33. "What percentage of people struck by lightning immediately die?"

34. "What percentage of Americans are left handed?"

35. "What percentage of the total number of bones in the human body are in the feet?"

36. "What percentage of Utah's population is Mormon?"

17

Distribution of answers to trivia questions (1)(regardless of what group an individual is part of)

18

0 20 40 60 80 1000

5

10

15

20

What percent of people in the U.S. over theage of 25 have obtained a college degree?

0 20 40 60 80 1000

5

10

15

20

25

What percent of U.S. women age 25 and olderhave completed high school?

0 20 40 60 80 1000

5

10

15

What percent of the worldwide use of energyis by the United States?

0 20 40 60 80 1000

5

10

15

20

25

30

What percent of the worldwide populationlives in the United States?

0 20 40 60 80 1000

5

10

15

20

25

What percent of U.S. energy consumption iswith fossil fuels?

0 20 40 60 80 1000

5

10

15

In 2000, the richest 1% of adults owned whatpercentage of the world's total assets?

0 20 40 60 80 1000

5

10

15

20

25

30

What percentage of Americans are age 0-14?

0 20 40 60 80 1000

5

10

15

20

25

What percentage of Americans are Protestant?

0 20 40 60 80 1000

5

10

15

20

25

30

What percentage of Americans are black?

0 20 40 60 80 1000

5

10

15

20

What percentage of Americans over the age of15 can read and write?

0 20 40 60 80 1000

5

10

15

20

25

30

What percentage of the American populationis below the poverty line?

0 20 40 60 80 1000

5

10

15

In 2005, what was the U.S. military expenditureas percent of the national income?

green = round 1, blue=round 2, red = round 3

Distribution of answers to trivia questions (2)

19

0 20 40 60 80 1000

10

20

30

What percentage of people in the world areage 0-14?

0 20 40 60 80 1000

5

10

15

20

What percentage of people in the world areChristian?

0 20 40 60 80 1000

10

20

30

40

Across the world, what percentage of people'sfirst language is Mandarin (a dialect of Chinese)?

0 20 40 60 80 1000

5

10

15

Across the world, what percentage of peopleover the age of 15 can read and write?

0 20 40 60 80 1000

5

10

15

20

Across the world, what percentage of theworld labor force is in agriculture (not industry or services)?

0 20 40 60 80 1000

10

20

30

40

50

What percentage of the world's surface iswater (not land)?

0 20 40 60 80 1000

5

10

15

20

What percentage of people in the world areaged 65 or over?

0 20 40 60 80 1000

5

10

15

20

What percentage of people in the world areMuslim?

0 20 40 60 80 1000

5

10

15

20

What percentage of college freshmen havenever used a typewriter?

0 20 40 60 80 1000

5

10

15

20

What percentage of the U.S. population willeat macaroni and cheese sometime in the next two weeks?

0 20 40 60 80 1000

5

10

15

20

What percentage of Americans drink a sodapop nearly every day with breakfast?

0 20 40 60 80 1000

5

10

15

20

In the United States, what percentage ofhouseholds routinely speak a language other than English?

Distribution of answers to trivia questions (3)

20

0 20 40 60 80 1000

5

10

15

20

25

In the United States, what percentage offirms are owned by women?

0 20 40 60 80 1000

5

10

15

20

25

What percentage of American men serve timein prison at some point in their life?

0 20 40 60 80 1000

5

10

15

20

25

What percentage of Americans are regularcigarette smokers?

0 20 40 60 80 1000

5

10

15

20

25

What percentage of Americans have nohealth insurance?

0 20 40 60 80 1000

5

10

What percentage of children diagnosed withcancer make a full recovery?

0 20 40 60 80 1000

10

20

30

40

What percentage of a woman's body consistsof water?

0 20 40 60 80 1000

5

10

15

20

25

What percentage of people grind their teethwhile sleeping?

0 20 40 60 80 1000

5

10

15

20

25

What percentage of the earth's water isfresh (not salt) water?

0 20 40 60 80 1000

2

4

6

8

10

12

What percentage of people struck bylightning immediately die?

0 20 40 60 80 1000

5

10

15

20

25

What percentage of Americans are lefthanded?

0 20 40 60 80 1000

5

10

15

20

25

What percentage of the total number ofbones in the human body are in the feet?

0 20 40 60 80 1000

2

4

6

8

10

12

What percentage of Utah's population isMormon?

Experiment 2 mean errors by individual

21

0 20 40 60 80 100 12010

15

20

25

30

35

40

Individuals (sorted by mean error)

Mea

n e

rror

ove

r q

uest

ion

s

round 1round 2round 3

Experiment 2 mean errors by question

22

0 5 10 15 20 25 30 355

10

15

20

25

30

35

40

45

Questions (sorted by mean error)

Mea

n e

rror

ove

r q

uest

ion

s

round 1round 2round 3

4 questions have very high mean error

Removed the following questions

Four questions had high error rates across individuals 12 16 29 33

(12) In 2005, what was the U.S. military expenditure as percent of the national income?

(16) Across the world, what percentage of people over the age of 15 can read and write?

(29) What percentage of children diagnosed with cancer make a full recovery?

(33) What percentage of people struck by lightning immediately die?

Removed these questions from dataset

23

Replotting mean error per individual

24

0 20 40 60 80 100 1200

5

10

15

20

25

30

35

Individuals (sorted by mean error)

Mea

n e

rror

ove

r q

uest

ion

s

round 1round 2round 3

error very low

25

0 20 40 60 80 100

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Que

stio

ns

Group 2

0 20 40 60 80 100

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

Que

stio

ns

Group 2

all raw data from one group with two individuals: in each box we see the responses for three rounds (bottom row in each box is round 1). Dashed line represents ground truth

26

0 20 40 60 80 100

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Que

stio

ns

Group 6

0 20 40 60 80 100

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

Que

stio

ns

Group 6

27

0 20 40 60 80 100

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Que

stio

ns

Group 4

0 20 40 60 80 100

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

Que

stio

ns

Group 4

Higher accuracy on round 1 when individuals do not change their estimates in subsequent rounds

28

0 20 40 60 80 1000

10

20

30

40

50

60

70

80

90

100

Mean estimate change from round 1 to round 2

Mea

n e

rror

at

rou

nd 1

Correlation = 0.369

0 10 20 30 40 50 600

10

20

30

40

50

60

70

80

90

100

Std estimates across roundM

ean

err

or a

t ro

und

1

Correlation = 0.389

(weak effect though)

If people change their estimate, they generally increase their accuracy on next round

29

0 20 40 60 80 100-80

-60

-40

-20

0

20

40

60

80

100

Change in estimate from round 1 to round 2

MA

D(

rou

nd 1

) -

MA

D(

roun

d 2

)Correlation = 0.547

Experiment 1 and 2

30

Wisdom of Crowds Analysis

How accurate is the group estimate compared to the individual estimates? Does the group estimate improve over rounds or do group processes start to correlate the errors, lowering the accuracy of group estimates?

31

MAD for experiment 1 (solo condition)

32

Exp 1 Solo 1 2 3Individual level

11.71 9.82 9.55Combined groups level

Mean 2.19 1.51 1.71Median 1.28 2.56 2.67Mode 5.00 4.72 5.56

Rounds

MAD for experiment 1 (group condition)

33

Exp 1 GroupIndividual level

8.68 6.96 6.50Group level

Mean 5.23 4.69 4.78Median 5.19 5.16 5.11Mode 6.08 5.66 5.58

Combined groups levelMean 1.78 2.75 3.11

Median 2.78 3.50 3.72Mode 3.89 3.61 3.33

MAD for experiment 2 (solo condition)

34

Exp 2 SoloIndividual level

16.97 17.34 17.32Combined groups level

Mean 14.48 14.38 14.70Median 13.01 12.05 12.45Mode 13.54 12.62 13.42

(note: very few individuals in this condition)

MAD for experiment 2 (group condition)

35

Exp 2 GroupIndividual level

17.83 16.03 15.66Group level

Mean 15.43 14.66 14.44Median 14.54 14.20 14.04Mode 15.31 14.67 14.60

Combined groups levelMean 14.28 13.23 12.98

Median 12.34 12.04 11.90Mode 10.12 11.13 11.20

Interpretation of Experiment 1

In the solo condition, accuracy increases because of increased information -- individuals can inspect the same display again in rounds 2 and 3

In the group condition, accuracy increases because of some combination of the increase in perceptual information available and some group dynamic

Possible reasons for difference in accuracy for round 1 in solo and group conditions: Between subject differences -- individuals in group conditions could be

inherently better “Somebody is watching” effect. People try harder when they are in

groups Improved calibration. People are learning, cross trials, by seeing

everybody else's guesses.  People never get feedback on what the RIGHT answer is, but they do see other people's responses, and they can use these to calibrate their own responses.

36

Interpretation of Experiment 2

Accuracy did not increase in solo condition not surprising because they have to answer the same trivia

question within short time period caveat: there were a few solo individuals in this experiment.

Individual accuracy does increase in the group condition this strongly suggests that a group process is responsible

37

Interpretation

Over rounds the group estimate does improve, but the effect is fairly small

Therefore, communication between individuals does not necessarily harm the “wisdom of crowd effect”

38

Models

39

Notes / Ideas

Current experiment is a variant of the Delphi method Look up published papers on the effectiveness of the Delphi

method. Some overview papers have claimed that communication between individuals can improve group accuracy

Can we do an experiment identical to Experiment 1 but the perceptual information is only shown in round 1? This would make the experiment comparable to the trivia questions

Can we do an experiment identical to Experiment 1 but where people do not know at round 1 whether they are in a solo or group condition? This should by design remove any differences found at round 1 and facilitate any differences found at round 2 and 3.

40

Next steps: Bayesian Aggregation Methods (not implemented yet)

Can we improve on these simple group estimation procedures such as the mean or median and get an even better group estimate? Can we build a model that removes any possible biases due to group processes because the model “knows” about such processes?

We need a model that analyzes individuals and groups across questions and rounds taking into account that each individual’s estimate is a blend of their own prior and the estimates of other individuals. This leads to recursive Bayesian models where at one level, we have a model for how people integrate their own information with other people’s estimates and at another level, how we take the output of individuals and make inferences on how people make inferences about other people in order to best infer the group estimate

41

Notes

Do people copy exactly the same values from other individuals or do they average/blend?

42