Innovative Design & Manufacturing Research Centre University of Bath World-leading research in...

Innovative Design & Manufacturing Research Centre

University of Bath

“World-leading research in engineering design and manufacture.”

1

IdMRC Social Research MethodsAutumn Lecture-Workshop Series

BATH September 2009 2

Design Research Methods:

Observation and Coding

Janet McDonnell


Note

This collection of slides supports a lecture on the topic of observation and coding.

The slides in this set are not intended to serve independently of the lecture as a stand-alone introduction to, or overview of, the subject.

The lecture includes the use of video and other visual material excised from this sub-set of slides which is solely intended to support note taking at the lecture.

Janet McDonnell


Outline

[software tools] CASE STUDY: from the categorical to the conjectural Knowledge claims – from nature to narrative Traceability and reliability What is the data from observation Approaches to research using observation data CASE STUDY: single dataset, multiple approaches to analysis Pointers for quality


Software Tools - Let’s get them out of the way

Behavioural Research : recording behaviour, eye tracking, facial expression and other types of behaviour recognition, synchronising with transcripts and support for coding, etc.

www.mangold-international.com (Interact) www.tracksys.co.uk (Observer XT)

Qualitative Data Analysis – content analysis of texts – from transcripts, interview data, etc. linking in images and video clips, ‘qualitative data base management’, collaboration

www.qsrinternational.com (Nvivo) www.atlasti.com (ATLAS-ti)

Quantitative data analysis Stats packages, graphing and diagramming tools

Transcription aids


Anatomy of a small scale enquiry

An investigation into the exchange of ideas and information between an architect and building users in the early stages of a building (re)design project before the design brief or any drawings have been produced.

We look at the type of information users exchange.

We are interested in what influences the information exchanges - context of the meetings and- the conversational strategies of the architect.

Classifications were determined in advance of coding the data.


Data and method

Participant observation: interaction as it occurs in natural setting

Data consists of transcripts acquired from audio-taped meetings

Focus on manifest content Segment transcripts into units of analysis Apply predetermined codes* Inter coder issues Data analysis and data interpretation Construct an account of the topic

* Based on P. Medway (2000) Writing and Designing in Architectural Education in

A.Pare (ed) Transitions: writing in academic and workplace settings, Hampton Press, pp 89-129.


Conversation summaries

CONVERSATION 1 2 3 4

Length mins:secs 10:20 25:04 5:27 20:37

No. of turns 149 285 43 247

No. of information contributions

16 43 12 30


Information contributions


Building User 9 23 12 9

Architect 6 11 9

Deputy Head Teacher 1 8 12


Codes for information contributions

CODE DEFINITION

1 Functional or structural naming

Elements and properties of the building or space (floor, exposed brickwork, etc.) – its objective

character

2 Perceptual awareness Attributes of the building or space that can be perceived (visual, acoustics, etc.)

3 Phenomenological experience

Feelings and associations made when experiencing the space (e.g. solidity)

4 Symbolic meaning Evocation of ideas unrelated to the structural form (mystery, memory, etc.)


“shelving reachable from the floor”“easy to clean surfaces”

“light and quiet study areas”

“a place where it feels safe to think ”

“a link between our past and our future”


Information contributions


Building User 9 23 12 9

Architect 6 11 9

Deputy Head Teacher 1 8 12


It was noticeable that some users were competent at discussing contributions in a more informed, technical manner than others. There were differences between the users’ understanding of the structural properties of buildings and their range of abilities to articulate spatial preferences for the design of the workspace. Compare U3: … my wish is I suppose to take out this joint wall and incorporate this store and this area into a larger area …

withU1: … if I had my wish then I would basically um take this wall down …

Although conversation 3 was shorter than the others, U3’s ability to converse in a designerly language made it easier … to share a common understanding of the properties of the space.


Types of information contribution

CONVERSATION 1Ar BU

2Ar BU

3Ar BU

4Ar BU

Functional 6 8 9 22

9

9 14

Perceptual 7

Phenomenological 2 2 4

3

2 4

Symbolic 3


The contribution of information with a symbolic meaning was infrequent. Symbolic references only cropped up in conversation 4 … all contributed by the deputy head teacher rather than the primary user of the space. We note also that the symbolic meaning discussed concerned what it meant to be part of the school and only once was the symbolic meaning of the space being designed mentioned strictly. The three instances are

D: … this is regarded as a privilege … to give them a kind of base so that they can start acting can I say more acceptable, normal circumstances in their reaction and inter-reactions with other peopleand D: … that’s why we still insist on them being in school uniform so they’re still part of the school to stamp identity U1: er yeah the whole idea is that they’re to be re-integrated into schoolandD: … we use this facility as an escape to do course work in the privacy of here as opposed to the pressures in the school


Claims, evidence and … truth

what is provably true logico-deductive reasoning, exhaustive

search

what is probably true statistical reasoning

what is plausibly truethe evidence of arguments that are

sincere and will convince a reasonable audience


Knowledge claims from research

Provable and ProbablePositivist : natural sciences : experimental reproducibility;

falsifiable hypotheses

PlausibleInterpretivist : cultural and social sciences : plausibility –

traceability; recoverability; narratives providing understanding

Emancipatory‘Critical’ theorist : social sciences : emancipatory force


Transparency in enquiry

Declared or shared framework, methodology, methods: allow process by which results are obtained to be recovered by others

The process of enquiry may lead to contributions to knowledge in each aspect (framework, methodology, research area)

Sharing of the meaning-making addresses ‘subjectivity’ objections – a route to agreement

framework of ideas

methodology

area of research concern


Inter coder reliability

Context: decisions about what is required to adequately serve the claims and evidence

Kappa coefficient is a measure of agreement between coders on the classification of qualitative data

Measures agreement in using a defined coding scheme in a prescribed manner.

Kappa takes into account the potential for chance agreement.

Does not take account of degrees of disagreement. Is a good test of whether categories are sufficiently distinct

from each other and sufficiently clearly defined for more than one coder to code reliably using them. So can be used to test identification of units of analysis and application of codes.

[J Carletta (1996) Computational Linguistics 22(2) 249-254 Assessing agreement on classification tasks: the kappa statistic. ]


45 Adrian OK ++ so moving into a little bit more detail46 [paper shuffling, 7 seconds pause]47 Adrian erm starting with the entrance, we arrive at the car park here and we 48 move forward to the entrance and waiting area here we've upped the 49 importance of this area one of the items on the minutes last time was 50 to increase the size of the waiting area 51 Angela yes52 Adrian and we virtually doubled in size errm we've actually extended the 53 canopy over it as well to make more of an entrance to make more of a 54 greeting zone so there's lots of space here with seats for people to wait 55 for taxis or whatever at the end of their end of their 56 Angela /hmm-mm\57 0.03.00 Adrian /service\ places for people to to actually stand and mill in if they wish

58 to stand out here before they move into the waiting area then come 59 into the porte-cochere so it's an even grander entrance 60 Angela /OK yes\61 Adrian /than we had\ before 62 Adrian erm one of your next items on the list was to widen the access way so 63 that two vehicles perhaps two limousines or a hearse and limousine 64 could park side by side in this area so this has been widened so its 65 now possible as I say for two vehicles ++ to park here ++ before 66 moving through further on ++ one of the next points made was the size 67 of the sanctuary we've increased the diameter of this by correct me if 68 I'm wrong Toby three hundred millimeters internally69 Toby it’s three point four

Extract from a transcript


Transcription conventions + pause of one second ( ++ two seconds, and so on)

…/….\… crude indicators of simultaneous speech…/……\…

NO emphatic stress

GOOGLE names of people, organisations, and so on

…… material omitted

- incomplete or cut off utterance

( ) unclear utterance

(over there) unclear utterance, transcriber’s best guess within brackets

[points at drawings] transcriber’s comments

[laughs] paralinguistics

[For a more comprehensive scheme refer to Gail Jefferson’s notation described in J Atkinson and J Heritage (eds) Structures of Social Action (1984) CUP, pp ix-xvi]


Ways of approaching research using observation data

Starting with a ‘template’ e.g. a model or a theory, classifications determined in advance

that are imposed on the data

Starting with the data e.g. looking for emergent patterns, evolving classification to

cover the phenomena observed (coding scheme not inevitable consequence)

Flexible research agenda vs. rigid research agenda e.g. starting with a scheme but being open to the

unclassifiable (and to new classes); developing hypotheses to test based on indications of possible patterns, causations


Common Dataset Multi-angle video recordings, plans, drawings, sketches, flip-charts,

transcripts from meetings from two authentic design projects

Architectural Design Project Data Two meetings 8 months apart between architect and

clients/building users Brief to design a crematorium with chapel and related provision

including offices, vestry, waiting rooms, parking, landscaping Engineering / Product Design Project Data

Two meetings 3 days apart among engineers and other specialists from the same company – multi-disciplinary groups

Brief to develop novel product ideas for sort of digital pen to exploit a new technology

24 different studies on themes relevant to research into design thinking

Anatomy of a large collaboration: DTRS7


How the data was tackled

Styles of approaching the data Starting with a ‘template’ : e.g. a model or a theory Starting with the data Flexible agenda vs. rigid agenda

Themes Understanding Process Values in Designing Aspects of Design Cognition Design Process Models Language, Discourse, and Gesture Designing Contexts Objects, References, and Representations


Types of analysis : imposition of a model or theory

Focus of attention: uses of analogy in designingFinding: a previously unreported use of analogy,

namely function-finding in creative design[Ball and Christensen, 2009]

Focus of attention: object references during designingFinding: object references in meetings between co-

workers are directed effectively to draw attention efficiently to features

essential to the comparison [Stacey et al., 2009]

Focus of attention: language of appraisalsFinding: positive and negative appraisals have

different effects on the focus of attention and knowledge generation during design

[Dong et al., 2009]


Types of analysis : fine grained analysis of the data

Focus of attention: gestures and individual turns at talkingFinding: reveals fundamental distinctions between

view of designing as goal-oriented and as goal-directed [Glock, 2009]

Focus of attention: designing through talk-in-interactionFinding: design concept has a special status – less

negotiable – the preserve of the designated ‘design’ expert

[Luck, 2009]

Focus of attention: social order – the ‘rules’ of interactionFinding: adherence to interactional norms interferes

with the ‘rules’ of brainstorming[Matthews, 2009]


Types of analysis : testing the limitations of prior findings or models

Focus of attention: consideration of context during designingFinding: support for claims that experienced designers explore broad context before close context is considered in depth

[Atman et al., 2009]

Focus of attention: a design team’s development of shared notions about the task and the team’s operation (indicated by verbal communication); based on a prior model of the development of ‘sharedness’ in teamsFindings: some expected findings – predicted by the model; some unexpected findings (not predicted by prior model) – only some of these accounted for by particularities of the data

[Badke-Schaub et al., 2009]


Quality of work

For in-vivo studies err on the side of over collection of data And/or iterate over observation/data collection and analysis … and don’t make excuses Coding – invest in adequate granularity (beware of findings

which are artifacts of the coding or analysis) An underused question: what else could account for the

findings? Consider the arguments, claims and evidence explicitly at

epistemological level Choose research methods that will provide the supporting

evidence you need (e.g. inter coder agreement if that is critical)

Choose language with care – consider the ‘baggage’ e.g. ‘hypothesis’, ‘subjects’


Further information

The ‘small scale enquiry’ was conducted with Dr Rachael Luck of Reading University and is published in Design Studies 27 (2006) pp 141-166.

DTRS7 work is published as a book About:Designing J.McDonnell and P.Lloyd (eds), 2009 and parts of it in two journal special issues Design Studies 30 (2) and CoDesign 5 (1) both in March 2009.

Janet McDonnell Central Saint MartinsSouthampton RowLondon WC1B [email protected]

www.csm.arts.ac.uk/janet-mcdonnell.htm

Innovative Design & Manufacturing Research Centre University of Bath World-leading research in...

Documents

Transcript of Innovative Design & Manufacturing Research Centre University of Bath World-leading research in...