Post on 01-Nov-2014
description
Research methods, EngD course, 2012 Slide 1
Research methods for socio-technical analysis
Ian Sommerville
Research methods, EngD course, 2012 Slide 2
Human and organisational issues
• When designing, testing or evaluating critical systems we need to know about how that system will be used in real settings
– What do users really do and why?– What potential is there for error?– How are errors avoided or mitigated?
• There are many useful generic insights into human factors – For example the GEMS model
• But systems engineers will inevitably need to understand human factors that are specific to the system they are constructing or maintaining
– Specifics emerge from: the location, the people, their work, the technologies in use, etc.
• Process models are too abstract to tell us what we need to know about what users do
Research methods, EngD course, 2012 Slide 3
Human and organisational issues
• Because of the importance of human and organisational factors, systems engineering draws from fields such as
– psychology – sociology – organisational studies
• But draws from their methods as much as their results– Don’t expect research done in other disciplines to provide all the
necessary answers for questions in systems engineering – Don’t expect that researchers from other disciplines will understand
or be interested in technology systems. Some are, many aren’t
• Systems engineers have to– Find out about human and organisational factors themselves– Work closely with people from other disciplines
Research methods, EngD course, 2012 Slide 4
The devil is in the detail• The fundamental principle of software design
is abstraction– Find the commonalities between the entities in the
system and represent these as software elements• This is based on the assumption that the
common features are the most important and that stakeholders can adapt to them
• Social science perspective is rather different. The focus is on the detail of a situation
– The key argument is that the details are very significant indeed and that failure to pay attention to them leads to problems
Research methods, EngD course, 2012 Slide 5
Studying socio-technical systems
• Systems engineers need to find out about about human, social and organisational issue that are specific to the system they are working on and to the class of systems being developed
• There are several useful ways to do this1. Experiments2. Archive research / Case studies3. Interviews and focus groups4. Simulation5. Fieldwork
• And several less useful ways (which are better than nothing)1. Questionnaires and surveys2. Imagining what users do
Research methods, EngD course, 2012 Slide 6
Case study analysis• Retrospective analysis, looking back at a
system or incident (a case) that has occurred• In-depth study, based on a detailed analysis of
a set of information sources• Based around one or more research questions
e.g.– What were the factors in system X that led to
catastrophic failure?– What management decisions contributed to the cost
overrun in system Y?• Intention is to gain generally useful insights
from specific cases
Research methods, EngD course, 2012 Slide 7
Examples of case studies• “Normal accidents’
– The problem of nuclear reactor overheating at 3-mile Island
• Denver airport baggage system• The London Ambulance Computer-aided
Dispatch System• Airbus accident – Warsaw airport• Case studies tend to focus on system failure
rather than system successes
Research methods, EngD course, 2012 Slide 8
Stages of case study analysis
• Determine and define the research questions• Select the cases and determine data gathering
and analysis techniques• Prepare and collect the data• Collect data in the ‘field’ (if appropriate to do
so)• Evaluate and analyse the data• Prepare the report of the case study
Research methods, EngD course, 2012 Slide 9
Research questions• Establish the focus of the study by forming questions
about the situation or the problem to be studied and so defining the study’s purpose
• The question or questions are then used to select material for analysis
• Accident enquiries are focused case studies where the question is decided in advance i.e.
– Why did accident X occur?• Danger of bias in establishing the research question
– What management problems led to the collape of Lehmans?
– What were the systemic factors that led to the collapse of Lehmans?
Research methods, EngD course, 2012 Slide 10
Data gathering and analysis techniques
• Case study selection– Often the most ‘typical’ case study does not reveal the
greatest insights– Choose case studies that are
• Extreme - representative of an exceptional situation e.g. an accident
• Critical - having strategic importance with respect to the general problem
• Data to be analysed– Essential to use multiple sources of data
• Interviews
• Archives and reports
• Fieldwork
Research methods, EngD course, 2012 Slide 11
Data collection and analysis• Typically, case studies generate large amounts of
qualitative data– Transcripts of interviews– Reports and associated notes
• Look for opportunities for triangulation– Evidence from diverse sources that supports the same
conclusions• Field data is also important
– Look at the object of the case study as it is– The system in operation
• Analyse the data in different ways and from different perspectives
– Use quantitative data to corroborate qualitative data
Research methods, EngD course, 2012 Slide 12
Case study issues• Generalisability
– Much of our learning is through case studies so we can make generalisations from them. However, we have to be aware of the possibility that specific factors may be dominant in any one study and be careful about generalisation
– Case studies can be useful for finding ‘black swans’ i.e. contradicting received wisdom
• Hindsight bias– Seeing events as more significant than they actually
were because there is a clear link between these events and the research questions
• Obsolescence of results
Research methods, EngD course, 2012 Slide 13
Fieldwork - Get Out of the Office!
Research methods, EngD course, 2012 Slide 14
Fieldwork• Generic name for a set of techniques which
involve collecting data directly from a worksite (the “field”), rather than relying on reports from other people
• Encompasses a range of techniques, which have much in common:
– Direct observation and discussion– Ethnography– Cultural and technology probes
Research methods, EngD course, 2012 Slide 15
Ethnography of work• Situated observation of work with a view to
developing a detailed understanding of the way that work is done and how the actors respond to other actors and the context of the work
• The aim is to develop a ‘rich picture’ of the work done
• Ethnomethodological ethnography– Ethnography that is intended to describe what work is
rather than explain work in terms of some social theory
– Approach proposed in the 1980s at Xerox, developed at Lancaster University in the 1990s.
– Work at Lancaster related ethnography explicitly to systems design.
Research methods, EngD course, 2012 Slide 16
Ethnography in practice• Looks at work in ‘real-time’ rather than
retrospectively• The ethnographer ‘lives in the workplace’ and
experiences it in the same way as the people involved in the work
• Examines the processes and artefacts that take place in the workplace
• The aim is to understand work as it really is rather than work as defined or retrospectively described
Research methods, EngD course, 2012 Slide 17
Examples of ethnography• The UK en route air traffic control system• The London Underground control room• Preparing and printing government documents• Health systems
– Managers in the health service– Breast cancer screening– NHS Direct
Research methods, EngD course, 2012 Slide 18
Fieldwork Methods• Get out of the office
– Talk to people about their work– Observe practices– Visit settings of use
• This can be part of an informal or formal study– How formal or informal you are depends on:
• The formality of the results you wish to produce (Are you writing a study to be read by others, or just trying to get a feel for a setting?)
• Your relationship to the people you research (Do they work in the same organisation as you? Are they customers?)
• The setting (Some workplaces require different kinds of formalities, e.g. steel mill workers vs. hospital workers)
Research methods, EngD course, 2012 Slide 19
Observe “ordinary practice”• Don’t wait for things to go wrong
– The routine and mundane are interesting (safe technology should be mundane and routine!)
– Error is a judgement, are you in a position to judge what is an error?
• What is supposed to happen and what really happens are often different
– People break the rules– People have to go to extra effort to make the rules work
• How is the system used in practice?
• What extra things are necessary to make the system work?
• Look and listen, but don’t judge. Users know their work better than you do
• Observation is sometimes referred to as “Ethnography”
Research methods, EngD course, 2012 Slide 20
Fieldwork and systems engineering
• “Traditional’ ethnography is based on the ethnographer spending prolonged periods in the workplace
– Problematical when there are deadlines for procurement and system specification
• Various techniques have been developed to address this issue
– Concurrent fieldwork– Evaluative fieldwork– Quick and dirty fieldwork
Research methods, EngD course, 2012 Slide 21
Concurrent fieldwork
Debriefing meetings
Ethnographicstudy
Systemsdevelopment
System prototype
Research methods, EngD course, 2012 Slide 22
Concurrent fieldwork• An approach that can be used in conjunction
with iterative development or system prototyping
– The field worker can act as a ‘surrogate user’ because of their knowledge of the work and workplace
– However, unlike users, they can be more objective about the system
• The debriefing meetings involve the fieldworkers and system developers
• The developers set questions for the fieldworker based on the prototype system
Research methods, EngD course, 2012 Slide 23
Evaluative fieldwork
Debriefing meetings
Ethnographicstudy
Amended specification or
design
Initial specification or
design
Research methods, EngD course, 2012 Slide 24
Evaluative fieldwork• An approach which is geared to ‘sanity
checking’ assumptions about a system or a system specification
• The field worker reads and understands the implications of the specification then makes observations of the processes to be suppprted
• During the debriefing meetings, he or she discusses the issues and problems that arose during the fieldwork and the team discusses possible changes to the specification to address these
Research methods, EngD course, 2012 Slide 25
Quick and dirty fieldwork
Debriefing meetings
Quick and dirty fieldwork
More detailed requirements
Outline requirements
Requirements engineering
Research methods, EngD course, 2012 Slide 26
Quick and dirty fieldwork• Short, focused studies to highlight the key
issues of concern in the ways in which work is done
• Aim of the work is to refine an initial specification
• Fieldwork is concentrated in a short period (1 day-1 week), with no iterations after debriefing meetings
Research methods, EngD course, 2012 Slide 27
Planning Fieldwork• Plan
– But don’t plan too much. Allow for the unexpected– How long you need is contingent on what you are looking at and
how long people will put up with you– Who you observe is also contingent on how long you have, and who
will have you. Don’t always assume the ‘important’ people are important (e.g. secretaries can often play critical roles)
– Whether you make recordings or take notes depends on the setting, and whether and how you want to write up examples afterwards
– Be ethical! This is not just about bureaucracy but about morality and common sense
• Access Issues– You may need to approach and negotiate with people: managers,
staff, trades-unions
Research methods, EngD course, 2012 Slide 28
Writing up results• What you do after a field study depends on who the results are
relevant to and how formal the work was– If it was sensitisation they may be nothing to do immediately, or
you may need to do a presentation or write a report and discuss it (Discussion is very helpful)
• The value of the sensitisation should be relevant later - it will influence subsequent decisions
– If the work was more formal and for wider dissemination (i.e. you work in a large team, a long project and/or your work has wide relevance) you will need to write more formal reports.
• First step should be to transcribe or write up what you have found
• Analysis can be done in a number of ways and is usually qualitative rather than quantitative
– Field-studies often lead to useful contextual information and often negative information rather than any functional or non-functional requirements.
Research methods, EngD course, 2012 Slide 29
• She works over constant periods – maybe a day, maybe 10 minutes.• Has done collaborations in past, but not now. Works for a ‘boss’.
Seems like the usual working arrangement. She has two supervisors, one bioinformatician, one biologist. Biologist doesn’t mind about the methods (as in paper)
• The blasting etc will just be there in a paragraph. If you find errors you re-submit correct versions of sequences.
• A lot of stuff looks similar as you work on it so a lab-book helps.• When is it useful? When writing up. Very useful when writing up
rather than wading through files. When have collaborators for presentations or talks. When you have a student you can just show them the process. Time- you can see if someone published a suggestion before you went ahead and did it. When you want to submit a sequence. When you have competitors. A collaborator did ask the date once – I could go back. In bioinformatics I’ve never had that.
Informal notes about lab-books
Research methods, EngD course, 2012 Slide 30
Key points
• Case studies are a retrospective analysis technique that examine a single case in depth
• Case studies rely on diverse information sources including (sometimes) fieldwork
• Fieldwork is a way of finding out about human and organisational factors relevant to a system
• Talk to users (stakeholders, developers, etc) and spend some time observing what they do
• Shut up, look, listen. They are the experts, not you.• Discuss your findings informally with other developers, and
write up your findings more formally if they are useful to a wide number of people