Effects of Simultaneous and Sequential Work Structures on Distributed Collaborative Interdependent Tasks

+ SIGCHI 2014

-Paul Andre, 2014

/이현정

x 2015 Autumn

Simultaneous

Distributed CollaborativeSequential

Effects of Simultaneous and Sequential Work Structures on Distributed Collaborative Interdependent Tasks

+ SIGCHI 2014

-Paul Andre /이현정

x 2015 Autumn

The Author

The Author

The Author

• This work was partially supported by Carnegie Mellon’s Center for the Future of Work, Bosch, Google, and NSF

The Paper

• Faculty and students from a range of CMU departments — the School of Computer Science, Carnegie Institute of Technology, the School of Design at the College of Fine Arts, Heinz College, and the Tepper School of Business - See more at: http://fow.heinz.cmu.edu/#sthash.PiaD6CiJ.dpuf

Distributed Online Groups + complex creative tasks

Coordinating multiple group members to work together effectively while minimizing process losses

What

The Challenge

Comparing the effectiveness of two coordination strategies(simultaneous vs. sequential work) on a complex creative task as the number of group members increased

Experiment

Results • sequential work structure > simultaneous work structure • territoriality partially accounts for these results (mediation analysis) • giving workers specific roles mitigate the detrimental effects of the

simultaneous work structure

• contribute to small group theory • practical understanding of how to use crowdsourcing

for creative and complex tasks

Implications

Abstractdesign product develop software write encyclopedia

Distributed Online Groups + complex creative tasks

Coordinating multiple group members to work together effectively while minimizing process losses

What

The Challenge

Comparing the effectiveness of two coordination strategies(simultaneous vs. sequential work) on a complex creative task as the number of group members increased

Experiment

Results • sequential work structure > simultaneous work structure • territoriality partially accounts for these results (mediation analysis) • giving workers specific roles mitigate the detrimental effects of the

simultaneous work structure

• contribute to small group theory • practical understanding of how to use crowdsourcing

for creative and complex tasks

Implications

Abstractdesign product develop software write encyclopedia

Process Loss

(social psychology)

Any aspect of group interactions that

inhibits good problem solving.

Raw material -> finisher products

Distributed Online Groups + complex creative tasks

Coordinating multiple group members to work together effectively while minimizing process losses

What

The Challenge

Comparing the effectiveness of two coordination strategies(simultaneous vs. sequential work) on a complex creative task as the number of group members increased

Experiment

Results • sequential work structure > simultaneous work structure • territoriality partially accounts for these results (mediation analysis) • giving workers specific roles mitigate the detrimental effects of the

simultaneous work structure

• contribute to small group theory • practical understanding of how to use crowdsourcing

for creative and complex tasks

Implications

Abstractdesign product develop software write encyclopedia

Introductionnumber of people way to organize their effort

Crucial Questions

pros and cons of sequential vs. simultaneous work structure on interdependent tasks as the number of workers increases

optimal output best use of each worker

StructuralcontingencyTheory

optimal structure or coordination method varies according to factors such as group size and task uncertainty

creative task

assembly line

work process not rigid unable to be defined in advance task is interdependent

Aim of Paper

+ +

++

+

Related WorkMore is good

• can offer new knowledge • cognitive stimulation • error detection

task type + coordination mechanism

Task Type

Both in one

• [larger > smaller] - tasks with objective answers - "Eureka" tasks(everyone will recognize a correct answer once it is offered), because larger group is more likely to contain individual with answer

• [larger< smaller] - problem solving tasks (Groups preform better than the best individuals...Lauglin, 2006)

• trip planning : generation of ideas + consider constraints • collaborative poetry translation: creative + negotiation with coworkers • iterative chair design: creative + practical consideration

Related WorkCoordinationStyles

• A group of workers can be utilized in different ways • and their interactions defined by their environment

pooled sequential teem

simply the aggregation of individually preformed tasks

one worker build on another's output

require interaction among members

creative taskassembly line

• synchronous work the best? • high coordination costs, adding workers may increase process losses > benefit

the extra worker brings • This paper argues that sequential work may mitigate such process loss

Related WorkCoordinationStyles

• A group of workers can be utilized in different ways • and their interactions defined by their environment

pooled sequential teem

simply the aggregation of individually preformed tasks

one worker build on another's output

require interaction among members

creative taskassembly line

• synchronous work the best? • high coordination costs, adding workers may increase process losses > benefit

the extra worker brings • This paper argues that sequential work may mitigate such process loss

Experiment 1group limerick(five-line poem) writing using etherpad

There was an old man with a beard Who said, 'It is just as I feared!' Two Owls and a Hen, Four Larks and a Wren, Have all built their nests in my beard! -Edward Lear

What

Why uncertain + interdependentproduction + problem-solving

How

1 worker 2 worker sequential

1 worker 2 worker simultaneous

1 worker 3 worker sequential

1 worker 3 worker simultaneous

2 x 2 experiment

add 1 worker in each iteration e,g. 1, 1+1, 1+1+1

Where Amazon Mechanical Turk (Online market place for work)

GROUP SIZE IN SIMULTANEOUS&SEQUENTIAL WORK

Procedure

Simultaneous

Sequential

1."Help Write a limerick(a short rhyme)" on MTurk 2.Waiting Room - crowdsourcing experiments 3.Page open in Background -> 2-3 people ready -> alerted

-> 10 minutes X -> 1 worker contribution 4.Taken to page with instructions and shown an etherpad

• no time limit • Payment - $0.70 per limerick, MTurk hourly rates

1."Improve /edit a limerick(short rhyme)" on MTurk 2.Waiting Room - crowdsourcing experiments 3.Page open in Background -> 2-3 people ready -> alerted

-> 10 minutes X -> 1 worker contribution 4.Taken to page with instructions and shown an etherpad

• no time limit • Payment - $0.50 per limerick,

Same collaborative interface, similar instructions, same task artifacts

MeasuresOutcome Measures

• overall quality • technical quality: is the limerick the right number of lines, correct rhyme structure

initially split assessment into four criteria to capture both objective and subjective aspects (technical, story coherence, creativity, and overall quality) initial testing found all to be correlated and for the remainder of the experiment to use the two clearest features

Raters • 3 raters : 2 undergraduate or masters in creative writing + 1 of the paper's authorAmabile, Hak&Bernts, Hennessey&Amabile

Rating • 1-7 likert scaleTechnical dimension: "Consider the number of lines, the rhyme scheme, and meter"Quality dimension: "A holistic rating, consider the story, the coherence, the creativity or interestingness, more generally: do you like it?

Calculation • Cronbach's alpha for the two dimensions (independent or combine?) combined to single rating (0.87)Agreement between raters (0.84)

MeasuresOutcome Measures

• overall quality • technical quality: is the limerick the right number of lines, correct rhyme structure

initially split assessment into four criteria to capture both objective and subjective aspects (technical, story coherence, creativity, and overall quality) initial testing found all to be correlated and for the remainder of the experiment to use the two clearest features

Raters • 3 raters : 2 undergraduate or masters in creative writing + 1 of the paper's authorAmabile, Hak&Bernts, Hennessey&Amabile

Rating • 1-7 likert scaleTechnical dimension: "Consider the number of lines, the rhyme scheme, and meter"Quality dimension: "A holistic rating, consider the story, the coherence, the creativity or interestingness, more generally: do you like it?

문항들의 내적일관성에 기초하여 추정되는 신뢰도 지수의 하나이다. 각 문항을 하나의 테스트로 간주하고 반응자들이 문항들에 대해 얼마나 일관성 있게 반응하는가를 계산하게 된다

Calculation • Cronbach's alpha for the two dimensions (independent or combine?) combined to single rating (0.87)Agreement between raters (0.84)

Cronbach's alpha 일관성을 나타내는 계수 Cronbach's alpha will generally increase as the intercorrelations among test items increase, and is thus known as an internal consistency estimate of reliability of test scores.

Results• 716 participants • 431 limericks • 1 person: 79

2 person group: 1853 person group: 167

Overall

Results• the improvement in work quality that resulted from working in groups of two or

three, rather than working individually was reliably greater when the work was organized sequentially

Analysisadditional workers improved production quality when sequential, but why?

• mediation analysis (territoriality and social loafing)

Territoriality

Mediation analysis

• people feeling uncomfortable editing another person's work (e.g. wikipedia) • non-physical but acknowledgment of the presence of co-worker • avoid editing or deleting others' work due to territoriality • could result in missed opportunities to improve, and fix error • When sequential no fear of 'stepping on someone else's toes', because

previous worker has departed

• measured the amount of deletion of someone else's work vs. deletion of my work

Social Loafing

• less effort because they believe other group member will 'pick up the slack' or they feel their ideas are dispensable

• When sequential, working alone -> less loafing

difference in quality?

• measured the amount of edits per person increase in group member reduce effort from worker?

Experiment 2Idea

Procedure

• Results from Experiment 1 indicate that additional sequential workers outperformed additional simultaneous workers

• Mediation analysis suggests that a territoriality phenomenon (whereby workers are reluctant to edit another’s work in the presence of that person contributed to this difference in quality)

IMPROVING SIMULTANEOUS WORK

• Control: same to Experiment 1 • Roles: writer and editor role

Results • Results suggest that assigning writer and editor roles in simultaneous work is able to mitigate process losses

Implicationhow might we get the most benefits from multiple workers working on a complex, interdependent task, where it is not clear what the answer is beforehand.

• prior work said simultaneous! but we say sequential!• recent crowdsourcing work examined iterative work -> we are different because

we compared the two methods • Simultaneous work was not able to capture the benefits of additional

workers, while iteration was able to do so, and removed communication and coordination overheads

• territoriality partially accounts for the difference in quality between simultaneous and iterative work

• This research appears at a time(2014) when the very notion of a team and group research paradigm is under discussion by organizational psychologists, as workers and collaboration are increasingly solely virtually mediated

Discussion• delete and edit을 보았는데... delete을 해야 edit을 하는거 아닌가 • distributed work를 보는데 있어서, 변수가 너무 많다. • output의 퀄러티는 어떻게 측정할 것인가? (creative work에 대한 evaluation 가능성 takeaway) • 영향을 주는 요인으로 무엇을 꼽을 것인가?

(domain에 따라? coworker들 사이의 관계에 따라?) • sequential=/ iterative work (software development)=lean UX?

Discussion• delete and edit을 보았는데... delete을 해야 edit을 하는거 아닌가 • distributed work를 보는데 있어서, 변수가 너무 많다. • output의 퀄러티는 어떻게 측정할 것인가? (creative work에 대한 evaluation 가능성 takeaway) • 영향을 주는 요인으로 무엇을 꼽을 것인가?

(domain에 따라? coworker들 사이의 관계에 따라?) • sequential=/ iterative work (software development)=lean UX?

Thank You.