CSCW Evaluation Techniques

Presented by:

Christopher Edwards

Overview of Presentation Evaluation Techniques Understanding Ethnography Using Ethnography in CSCW Understanding Ethnomethodology Ethnomethodology and CSCW Technomethodology Conclusion

Evaluation Overview Olson and Olson. What are we Evaluating? Evaluations Techniques… Internal/External Validity Conclusion

Characteristics of Groups Individuals Differ

in: Skills Ability Knowledge Personalities Motivations Agendas

Characteristics of Organizations System comprised of

people and technology

Social Technology Physical Technology Comprised of multiple

actors Dependency on

Communication Information

Processing Entities

Characteristics of Task Tasks involve

different types of material

Physical, Digital or Ethereal

Ease or Difficulty of Task

Differ on Core Activity Subtasks – Tightly

Coupled/Loosely coupled

Characteristics of Environment Physical

Environment Distance between

Group members CSCW technologies

designed to overcome Contextual Time

• When in the day the interaction occurs

• Effects on Distant Group member

Characteristics of Technology Increasingly Varied

Technologies to Support Conversation

• Auditory/Visual• Back

channels/gestures Technologies to

Support Shared Work• Objects that support

work• Fit of Tool to Material

Process

Technology Deployment How and Why Process Analysis

Why outcomes were affected•Progress of Task•Communication process•Examined through time scales

Outcomes The initial outcome of using

technology Quality of work Measure outcomes at every level

• Group Outcome• Organizational Productivity

Conceptual Framework for CSCW Studies

Group

Organization

Task

Environment

Technology

Process Outcomes

Tools used to evaluate CSCW technologies

The Survey Set of questions Fixed Alternatives Statistically

Analyzed Wording of

questions problematic

Interview Structure of

interview Formal and

structured Unstructured Analysis can be

complicated

Experimental Controlled Setting

Specific Task Conditions Assignment of Participants Useful for making inferences about

causality

Case Study Examines a single

or small number of cases

Exploratory research

Ethnography Method adopted

from Anthropology Describing Culture

Used originally to describe other cultures

Misunderstood method

Many other Methods Diaries Analytic Field Studies Quasi Experimental Longitudinal Studies Historical Studies

Internal and External Validity

HIGH

LOWHIGH

Level ofInternalvalidity

Level of External Validity

Laboratory Experiments

Surveys

Field Experiments

Ethnographies

Conclusion to the Overview Different factors influence use and

evaluation of CSCW software Framework of CSCW studies Evaluation Techniques Validity of Techniques

Short Break Reconvene in 3 minutes

*Upcoming – Understanding Ethnography Using Ethnography

Ethnography and CSCW

Ethnography Understanding Ethnography Sociology Adoption Using Ethnography CSCW Ethnography in Design

(Hughes)

Understanding Ethnography Ethnography is

loosely applied to qualitative research

Home is originally from Anthropology Aim to describe

cultural interpretation

Understanding Ethnography Understanding culture “from an

insiders point view” Three sources of data

Participant Observation Interviews Collection of representative artifacts

Sociology Adoption Originally used to study distant

cultures Chicago School of Sociology

Studies focused on exploration of groups in urban settings

Cultural comparisons in USA Family of Ethnographic Techniques

Class Participation Time In Pairs (Saul and

myself included) Everyone gets a

Handout For a total of 5

Minutes (2.5 Minutes each)

Each member of the pair (one at a time) asks the other questions from the sheet

Debrief of Class participation What answers were given…. Obviously not a long term

ethnography study Depending on your relationship to

this lab – differing perspectives Understanding Grouplab culture (to

some extent)

Ethnography and CSCW Prominence of Ethnography in CSCW

Insufficient attention to social context New problems for design of

collaborative character of work and activities

Ethnography and system design• Problem of scale• Pressure of time• Role of the ethnographer

Concurrent Ethnography Design is influenced by on-going

ethnographic study Sequenced process

Debriefing Meetings

System Prototype

EthnographicStudy

Systems Development

Concurrent Ethnography in action London Air Traffic Control Centre Four week Ethnography Each stage of fieldwork was

intended to target designers issues Small research team What ethnography provided

Quick and Dirty Ethnography Brief Ethnographic Studies Duration relative to the size of the task Selecting aspects of work setting of

importance to design Outline of projectMeetings

Debriefing Meetings

Scoping Document

Short Focus Studies

Quick and Dirty Ethnography in Action Ethnographic investigation of

software engineers Challenges of Large scale setting Working in Industrialized

Environments Acceptance into the setting (*Key to

Ethnographic research)

Evaluative Ethnography Ethnography used to verify

formulated design decisions

ShortEthnographicstudy

Initial outlineDesign or Specification

Debriefing Meetings

Amended DesignOr Specification

Evaluative Ethnography in action Fieldwork in Building Society Using research for IT developments Routine of work Finding what customers wanted Outlined limitations of model that

had been proposed

Re-examination of previous studies Previous studies are re-examined to

inform design Ethnography used for many decades Many studies related to work and

occupation Can be informative

Re-examination in action Inform preliminary design of Shared

Object Service Using previous Ethnographic studies

on: Social work, police work and invoice

processing in a multi-site fast food company

What common service should support

Summary of Ethnography Understanding Ethnography Ethnography and CSCW Uses of Ethnography

Concurrent Quick and Dirty Evaluative Re-examination

Big Break Time 5 Minutes Reconvene in 5 Minutes

Upcoming – Ethnomethodology Understanding Ethnomethodology Ethnomethodology in CSCW

Ethnomethodology Understanding Ethnomethodology Confusing Ethnography and

Ethnomethodology Ethnomethodology in system design Incorporation of Sociology and

Computer Science? ‘Technomethodology’

Understanding Ethnomethodology Ethnomethodology literally means

“People’s Methods” A Shift from ‘other’ Sociological

Methods Social Life is potentially Chaotic Social Actors Members methods for making sense

Understanding Ethnomethodology Garfinkel “Documentary Method” Example of Documentary Method Garfinkel “Indexicality” Disrupt Technique

Example in class

Understanding Ethnomethodology We can observe other members

methods of construction Development of Conversation

Analysis

Confusing Ethnography and Ethnomethodology Ethnography is a form of

investigative fieldwork Ethnography focuses on the

“Member’s Point of View” Ethnomethodology is a specific

analytical technique

Confusing Ethnography and Ethnomethodology Confusion arises because: Ethnomethodologist is likely to use

ethnographic techniques ‘Analytic mentality’-selection of

phenomena and topics for investigation

Ethnomethodology in HCI and CSCW Observations of work activities and

interactions help design process Understanding temporal

organization of activities and interactions and implications to design

Learning from Ethnomethodologists Division of Labour Field Observation conducted by

ethnomethodologists Act as proxy for end users Hand off requirements to computer

science people

Ethnomethodology for Critique and Design Ethnomethodology has provided:

Critique of the design Failure to support the work Technology doesn’t allow people to

engage in their work Outlines organization of work and

communication in the real world

Two Paradoxes Paradox of system design

Large scale activity Paradox of technomethodology

Transformational nature of technology Analysis of practice not invention

Technomethodology Develop a stance in which

ethnomethodology and computer science play equally significant roles

Foundational relationships

Technomethodology Trying to exploit generalizations

from ethnomethodology Abstractions from both disciplines Means by which such working

practices arise Dialogical interfaces

Conclusion Understanding Ethnomethodology Confusing Ethnography and

Ethnomethodology Ethnomethodology and system

design Technomethodology

Conclusion of Presentation Overview of Evaluation Techniques

used in CSCW Ethnography Ethnomethodology Technomethodology

My thoughts CSCW has been unfortunately

caught in a battle between sociological techniques

CSCW design should incorporate work context issues

It was this clear….