Controversies 2006

Controversies in Cognition

Evaluation of technology use ‘in the wild’

Dr. Danaë Stanton Fraser

Controversies 2006

Lectures • Space. Can we simulate it effectively?

• Spatial Cognition: learning what has been 'learnt' from cognitive maps.

• Laboratory versus Field: the Evaluation Debate

• Estimating time-to-collision: can we accurately estimate when we will reach an object or when a moving object will reach us?

• Evaluating ‘in the wild’

• Aiding Mobility: are electronic and environmental devices sufficient for aiding mobility in the mobility impaired?

Controversies 2006

Introduction

• Traditional technologies in subservient role to Psychology– Methods of recording and storing data

• e.g. desktop computers used to time reactions

– Domains of data gathering• e.g. digital video recording enables repeatable analysis of

behaviour

– Methods of conducting experiments• software provides reproducible experiences during experiments

– Access to people• Larger numbers of participants possible in on-line surveys or

questionnaires

Controversies 2006

Evaluating in the Wild

• New technologies also provide an interesting domain for study– New possibilities for collaboration and changing

methods of communication (e.g. telephone, email, SMS, videoconferencing, …)

– The Shared Desktop– Sensors and Context– Mobile Systems

Controversies 2006

Working with schools (1)

• Education– Working with schools versus testing in schools

• The practicalities: consent, ethics, relationship with user group, OFSTED, SATS

• Designing within the physical, social and organisational constraints of a real classroom

• Children as partners versus informants

• Design methods

Controversies 2006

Working with schools (2)

• How to gain access to schools

• Ethics approval – school policies

• Consent forms to parents– Through school– Include use of video recordings (research

papers now often go on the web)– Teacher’s approval

Controversies 2006

Designing with or for?

• Different ways of designing and working with user groups (see Druin, 2002)

– Design partners – Informants– “Subjects”/Participants

Controversies 2006

Participatory design

• Involve users as members of the design team from the start

• Methods of communication– brain (and body) storming– pen and paper interface walkthroughs– paper/cardboard mockups– early prototypes

• emerged from Scandinavia

Controversies 2006

Designing with Children

• Children and teachers involved in the design process.

• Work intensively with teachers and children in school to design and develop technologies to be integrated into the classroom?

• Interdisciplinary team? Those developing the technology should also go to the school!

•

Controversies 2006

Designing with Children

• Integrating with the National Curriculum?

• The physical nature of the classroom means that children are continually divided into small groups

• Iterative design sometimes slower development of technology but a more ‘integrated’ and ‘usable’ product.

Controversies 2006

Traditionally…

• Computer in the corner of the classroom• Computer lessons not linked to domain• Individual use of machine, or• Possibility for input < number of children per device• Children and teachers not actively involved in the

design of the technology being used

Controversies 2006

New Technologies: The Shared Desktop

• Working in pairs and groups can have advantageous effects on learning and development (Rogoff, 1990; Wood and O’Malley, 1996)

• Role of the computer unique in the way that it can structure collaborative activity (Littleton, 1999)

• Traditionally computer hardware and software designed with one user in mind

Controversies 2006

• KidStory project - Developing collaborative storytelling technologies for children aged 5-7

• Learning in class is a social activity - focus on group work

• Designing within the physical, social and organisational constraints of a real classroom

Controversies 2006

An evaluation study: Collaborative behaviour around the

computer

• An exploratory study to examine the effect of multiple mice on children’s dialogue and interaction

Controversies 2006

• Single Display Groupware - allows to co-located users to interact with a system simultaneously

• Inkpen et al (1995, 1997, 1999) found significant learning improvements - higher levels of activity and less off task behaviour

• Abnett et al (2001) gender effects.

Controversies 2006

Method

• Participants - 24 children from an infant school in Nottingham. Aged 6-7 years.

• Apparatus - KidPad a shared 2D drawing tool with a zooming interface (Druin et al, 1997).

• Task - Creative task carried out by pairs of children using one or two mice.

Controversies 2006

• Procedure

Recreate a poem in KidPad

Children encouraged to work together

20 minutes

Video capture of the computer screen and the children were mixed

An analysis of the process of collaboration

Controversies 2006

Examination of behaviour in depth

• Coding scheme developed to capture types of talk, physical interaction and their relationship with the on screen product

• Qualitative analysis

• The development of the car in the poem analysed for each pair (5minutes)

Controversies 2006

Characteristics and Behavioural styles observed

Interaction with 2 mice

Common themes

Verbalisation of action

Little reciprocity or elaboration of ideas

Active division of tasks - working in parallel

Still cases of dominant behaviour by one partner

Controversies 2006

Results

Use of two mice:

Greater degree of engagement in task

more total time spent on creation

symmetry of mouse use

Controversies 2006

Asymmetry with one mouse

0% 20% 40% 60% 80% 100%

Pair 3

Pair 5

Pair 7

Pair 9

Pair 11

Pair 13

Controversies 2006

Symmetry with 2 mice

0% 20% 40% 60% 80% 100%

Pair 2

Pair 4

Pair 6

Pair 8

Pair 10

Pair 12

Controversies 2006

Video

Controversies 2006

Interaction with one mouse

More of a mix of behavioursGood collaboration - long discussion of ideas,

reciprocity followed by input of joint ideas, conflict followed by compromise

Conflict not resolved, high degree of negativity about others work

Domination by one partner

Controversies 2006

New Technologies: Mobile and Wireless

Outside the school: the fieldtrip

• field trip ‘with a difference’

• Taking technology outdoors

• Making the invisible visible

• Carry out collaborative discovery and reflection

• To stimulate scientific enquiry

Controversies 2006

The Ambient Wood Project

• small groups of children using mobile technologies outdoors to support scientific enquiry about the biological processes taking place in a wood.

• One of the devices used, a probe tool, contained sensors enabling measurement of the light and moisture levels within the wood. A small screen was also provided which displayed the readings using appropriate visualisations.

Controversies 2006

Handhelds to make the invisible visible

• Measurements of light and moisture at different locations were displayed on a PDA in pictorial format.

• Mobile devices used to receive location-specific information.

Controversies 2006

Video

Controversies 2006

Key findings

Analysis of the patterns of interaction revealed: • The probe engendered exploration, the generation of ideas

(about where to probe in order to get different readings, or to see readings around particular plants).

• Children made links between their readings, for example, comparing readings taken by the same species of plant, but in different locations.

• Children made predictions about readings they might expect in particular locations, for example, one pair predicted a moist reading because there was lots of moss.

• Many also drew conclusions about the general physical state of the woodland, and how this related to the environment and the organisms found on the basis of their probe readings.

Controversies 2006

New Technologies: Mobile and Sensors

• Integrating the outdoors with back in the classroom• To explore how emerging networking technologies

can enhance science education• Hands-on approach to learning science in schools• Children learn about presence and impact of pollution• Encourage an understanding of the scientific process• Support collaborative activity between different

schools and with scientists

Controversies 2006

Activities Children as active scientists

• 2 schools involved– Primary in Nottingham– Secondary in Brighton

• Sessions to familiarise children with pollution• Children use mobile carbon monoxide sensors to

measure pollution levels in their local environment• Use software tools to analyse their data in the

classroom and to share with others

Controversies 2006

Mobile Sensors

• Children can collect pollution data

• Mobile sensor records and gives instant feedback

Controversies 2006

Controversies 2006

Analysing data

Video replay

Time series

Textual annotation

Interest point

Controversies 2006

Connectivity, schools and scientists

• Children in same school– Scientific Teams

• Children in different schools– Connectivity, Community

• Children and scientists – Learning from the Experts

Controversies 2006

Summary of results

Results of video analysis and teacher interviews suggest that this context-inclusive approach is significant for three reasons:

1. Firstly, it allows individuals to reflect on method as part of data collection.

2. Secondly it provides an aide-memoir to groups who have collected data together in interpreting results.

3. Thirdly, it allows new participants who have engaged in similar processes to understand new perspectives on their own and others’ data.

Controversies 2006

Controversies

• Should we be as controlled/experimental as possible/as the pre-determined factors allow?

• Is the only benefit of exploration and engagement in the field to provide ideas for experiments?

• Is lab-based work appropriate in studying field-based use?

Controversies 2006

References (1)Abnett, C., Stanton, D., Neale, H and O’Malley (2001) The effect of multiple input devices on collaboration and gender

issues. In the Proceedings of European Perspectives on Computer-Supported Collaborative Learning (EuroCSCL) 2001, March 22-24, Maastricht, the Netherlands. P.29-36.

 Druin, A., Stewart, J., Proft, D., Bederson, B., Hollan, J. (1997). KidPad: A design collaboration between children,

technologists, and educators. Proceedings of  CHI’97, Atlanta, GA. Druin, A. (2002). The Role of Children in the Design of New Technology. Behaviour and Information Technology, 21(1) 1-

25.  Inkpen, K., Booth, K.S., Klawe, M., and Upitis, R. (1995). Playing Together Beats Playing Apart, Especially for Girls.

Proceedings of Computer Supported Collaborative Learning (CSCL) '95. Lawrence Erlbaum Associates, 177-181. Inkpen, K. M., Booth, K. S., Klawe, M., & McGrenere, J. (1997). The Effect of Turn-Taking Protocols on Children's

Learning in Mouse-Driven Collaborative Environments. In Proceedings of Graphics Interface (GI 97) Canadian Information Processing Society, pp. 138-145.

 Inkpen, K.M., Ho-Ching, W., Kuederle, O., Scott, S.D. & Shoemaker, G.B.D. (1999) ‘This is fun! We’re all best friends and

we’re all playing’: Supporting children’s synchronous collaboration. In Proceedings of Computer Supported Collaborative Learning (CSCL99) (eds. C.M. Hoadley & J. Roschelle) pp. 252–259. Lawrence Erlbaum, Hillsdale, NJ.

  

Controversies 2006

References (2)Littleton, K. (1999). Productivity through interaction: An overview. In K. Littleton and P. Light (Eds.) Learning with Computers:

Analysing productive interaction. Routledge. London p.179-194.  Price, S., Rogers, Y., Stanton, D. and Smith, H. (2003). A new conceptual framework for CSCL: Supporting diverse forms of

reflection through multiple interactions. In Proceedings of Computer Support for Collaborative Learning. (CSCL) 2003, Kluwer, pp. 513-523.

  Rogers, Y, Price, S., Randell, C, Stanton Fraser, D., Weal M. and Fitzpatrick, G. (2005). Ubi-learning: Integrating Indoor and

Outdoor Learning Experiences. Communications of the ACM. January 2005/Vol. 48, No. 1 Rogoff, B., Apprenticeship in Thinking: Cognitive Development in Social Context. New York: Oxford University Press, 1990. Stanton, D., Neale, H. and Bayon, V. (2002) Interfaces to support children's co-present collaboration: multiple mice and tangible

technologies. Computer Support for Collaborative Learning. (CSCL) 2002. ACM Press. Boulder, Colorado, USA. January 7th-11th.p.342-352

Stanton, D. and Neale, H. (2003). Collaborative Behaviour around a computer: the effect of multiple mice on children’s talk and interaction. Journal of Computer Assisted Learning (JCAL), Blackwell, Vol. 19, no. 2, pp. 229-239.

 Stanton Fraser, D., Smith, H., Tallyn, E., Kirk, D., Benford, S., Rowland, D., Paxton, M., Price S and Fitzpatrick G. (2005). The

SENSE project: a context-inclusive approach to studying environmental science within and across schools. Computer Supported Collaborative Learning (CSCL 2005). Taiwan. May.

 Tallyn, E., Stanton, D., Benford, S., Rowland, D., Kirk, D., Paxton, M., et al. (2004). Introducing eScience to the classroom .

Proceedings of the UK e-Science All Hands Meeting, EPSRC, pp. 1027-1029.  Wood, D., & O'Malley, C., Collaborative learning between peers: An overview. Educational Psychology in Practice, 11(4), 4-9,

1996