Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

27
Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser

Transcript of Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Page 1: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies in Cognition

Laboratory versus Field: the Evaluation Debate

Dr. Danaë Stanton Fraser

Page 2: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Contents

• Methodological – quantitative versus qualitative

• Practical constraints

• Changes in society

Page 3: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

QUANTITATIVE:

E.G. EXPERIMENTS

Page 4: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Experiments: Aim

• To answer a question or test an hypothesis that predicts a relationship between two or more events, known as variables.

• E.g. Will spatial knowledge be superior following exploration of a VR simulation of a building or from exploring a model of a building?

Page 5: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Variables

• Such hypotheses are tested by manipulating one or more of the variables.

• The variable that is manipulated is called the independent variable (the conditions to test this variable are setup independently before the experiment starts). In the study mentioned media type (VR vs model) is the independent measure

• The dependent variable would be accuracy of spatial information (ie time to reach point a from point B) because the time to carry out the task depends on the media explored.

Page 6: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Variables and Conditions

• In order to test an hypothesis the experimenter sets up the experimental conditions e.g

• Condition 1 Pretest Explore model Post test

• Condition 2 Pretest Explore VR simulation Post test

• Control condition – against which to compare the results?

Page 7: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Participants

• Within participantsTwo drawbacks – no. of participants needed, individual differences

Advantage – no order effects

• Between participantsCounterbalancing required

• Matched pairs on characteristics such as gender and expertiseDifficult to match across all variables

Page 8: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Data Collection and Analysis

• Data – performance measures are taken e.g response times, no. of errors

• Use graphs• The data should be averaged across conditions to

examine any differences• Statistical tests such as t-tests and ANOVAs can

reveal if the differences are significant• Software packages such as SPSS are often used• If there is no significant difference then the

hypothesis is refuted.

Page 9: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Summary of Experimental Design

• State a causal hypothesis• Manipulate independent variable• Assign participants randomly to groups• Use systematic procedures to test hypothesised causal

relationships• Use specific controls to ensure validity

Page 10: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

QUALITATIVE:

E.G. ETHNOGRAPHY

Page 11: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Ethnography by date

• Anthropology (c. 1920s)– e.g. E. E. Evans Pritchard

• Workplaces (c. 1950s)– Chicago School

– E. C. Hughes, Becker etc.

• Computer-Supported Cooperative Work (c. 1990s)– J. Hughes, C. Heath

Page 12: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Ethnography by issue

• Long history with its roots in anthropology

• Is ethnography a method?– eclectic group of techniques

– does not predefine a theoretical or conceptual status

– has been used in support of many methodological standpoints, e.g. distributed cognition etc…

Page 13: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

What does an ethnographer do?

• assume human activities are socially organised • commit to inquiring into patterns of interaction• go into the field• ‘learn the ropes’ (question, listen, watch, talk, etc.,

with practitioners)• take their time• write a report

Page 14: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Ethnography principles• Ethnography is naturalistic

– studies should be studies of real people and their activities, operating in their natural environment

– doesn’t deal with artificial worlds & controlled versions of work

• but what is ‘artificial’?– Mike Lynch: "Stop talking about science. Go to a laboratory - any

laboratory will do - hang around a while, listen to conversations, watch the technicians at work, ask them to explain what they are doing, read their notes, observe what they say when they examine data, and watch how they move equipment around..."

Page 15: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Ethnography principles (2)

• Ethnography understands the world from the point of view of those who inhabit it

– it is behavioural, i.e. interested in the detail of the behaviour to a greater or lesser extent

– it is not behaviourist, i.e. it does not consider the behaviour itself as the appropriate level of analysis

Page 16: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Ethnographic data (1)

• Ethnographic data can include:• general descriptions of behaviours, descriptions of

physical layouts, close descriptions of conversation, thoughts and feelings, work sequences, anecdotes, examples, common occurrences, hypotheses etc.

– analysis is skilful (but not complex)

– ethnographer usually provides 'examples'

Page 17: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Ethnographic data (2)

• Technologies mostly used for data collection – analysis is skilful!

• Questions, notebook, pen• Tape recording

– less intrusive than video but data is less detailed

– awareness of being recorded not a problem

– useful to record long explanations, especially highly technical or domain-specific kinds

Page 18: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Ethnographic data (3)

• Video

– Data can be analysed repeatedly off-site

– Time-consuming data collection and analysis

– Difficulty setting up and using equipment in some domains

– Missing visual actions that are peripheral to the video field-of-view

• Programs to support analysis

– e.g. Ethnograph, NUDIST

Page 19: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Carrying out an ethnographic study (1)

• individuals and organisations have ideas, expectations and fears about ethnography

• Gaining access and an ability to be simultaneously likeable and anonymous will help

• Gaining acceptance may include working shifts, sharing conditions, a non-intrusive demeanour, sharing dress codes, but not sharing opinions unless absolutely necessary

Page 20: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Practical Constraints

• An example: The Living Exhibition– Imagine you have been asked to study a museum exhibition

– You are working with both schools and designers who have certain goals for the project

• Curriculum and Pedagogical aims

• Technical innovation

– You need to study the exhibition in such a way as to achieve interesting and coherent results taking into account the other stakeholders’ aims

– What do you test and how do you test it?

Page 21: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

• Variables• A Control – Books?• Data collection – participants dispersed over space

and time simultaneously• Unstable software – performance and appearance

may change at any time

Page 22: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

The living exhibition

Page 23: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

• Clear ability to design experiment in this very situation is impaired by practical constraints

• Experiments could not cover the whole picture, only one or more selected constituent features

• Other methods of data collection and analysis are possible (e.g. ethnography), but do not specifically provide a thesis for understanding the data, unlike an experiment

• Any results are skilfully produced to inform stakeholders, and aren’t available according to a given procedure

Page 24: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Societal change (1)

• Evolving technological standards, e.g. large-scale systems development

• Evolving technological approaches, e.g. ubiquitous computing – moving away from the desktop

Page 25: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Societal change (2)

• Changing face on human-human interaction, e.g. internet, mobile phones (often only able to see one side of conversation)

• Interdisciplinary research – a push towards disciplines collaborating e.g. psychology and computer science – necessitating a mix of methods/approaches

Page 26: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

Conclusions

• Are traditional psychological methods appropriate?

• Are traditional psychological methods sufficiently adaptable?

• Should we be developing new methods in line with technological change?

Page 27: Controversies in Cognition Laboratory versus Field: the Evaluation Debate Dr. Danaë Stanton Fraser.

Controversies 2005

References

• Cheverst, K., Dix, A., Fitton, D., Friday, A., Rouncefield, M. Exploring the Utility of Remote Messaging and Situated Office Door Displays. Mobile HCI 2003: 336-341

• Crabtree, A. (2003) Designing Collaborative Systems: A Practical Guide to Ethnography, London: Springer-Verlag

• Heath, C.C., Hindmarsh, J. & P. Luff. (1999) Isolation and interaction: thefragmented world of the train driver on London Underground. Sociology 33,3, pp. 555-575

• Heath, C.C. and P. Luff (2000) Technology in Action Cambridge: CambridgeUniversity Press. pp 1-269.

• Hemmings T, Crabtree A, ( 2002 ) Ethnography for design?. The 1st International Workshop on "Interpretive" Approaches to Information Systems, pages 122--124. Association of Information Systems

• Hughes, J. A., King, V., Rodden, T., and Andersen, H. (1994) "Moving out from the control room: Ethnography in system design". In Proceedings of CSCW ‘94, Chapel Hill, North Carolina.