Metcalfe critique2

CRITIQUING ARTICLES

by

Mike Metcalfe

©June 2002

“New Perspectives provides new knowledge…” [Churchman 1979]

1

CONTENTS

Preface: Why this book came about, and how to critique it.

Why Critique? Why Articles?

Simple Argument Perspective

Problem Solving Systems Perspective

Picturing Perspective

Things, Cultures and People Perspective

Perspectival Thinking

Empirics as Action Perspective

Metaphoric Perspective

Dialectic Perspective

Evolutionary Perspective

Power Perspective

Fuller Argumentation Perspective

Conclusion

2

Preface

Good critiquing requires insight, asking thoughtful questions and seeing behind the text. How can someone practice this art? This book aims to provide some pointers. Having been involved with critiquing articles with systems thinking PhD candidates for a number of years, I have attempted to expand on the ancient, yet wonderful, “What is their argument, and is it justified?” as a critiquing method using perspectives such as systems thinking, evolution theory, picturing, dialectic and the perspectival thinking.

My perspective on critiquing articles comes from trying to develop ways to appreciate, and if possible solve, human activity problems. Learning critiquing skills is one approach. The way to judge this book is in terms of your own thinking; critique your own learning. The book aligns with C. West Churchman (ex editor of the Journal of Philosophy) that learning is about finding new perspectives, to give you more choices. I hope this book gives you new perspectives and new choices.

The book is laid out in 11 main chapters, each with a page or two explaining the perspective followed by a set of sample questions that may help your thinking about whatever article you wish to critique.

Regards,

Mike Metcalfe, June 2002

Further advice can be found at the web site of “The Writing Centre” at Harvard University. They, understandably, focus on extracting a justified argument1.“The Writing Centre” at Harvard University, http://www.fas.harvard.edu/~wricntr/html/tools.htm

1 The word argument is being used here in the sense of a one line conclusion, proposition, conjecture or claim that needs to be justified by supporting evidence. It is not a quarrel, but well reasoned, nor pure logic but structured conversation.

3

http://www.fas.harvard.edu/~wricntr/html/tools.htm

WHY CRITIQUE?

Why do I use the word ‘critique?’

I am not assuming the word ‘critique’ has an immoveable definition. I am using it in the sense of ‘constructive evaluation’ or ‘literary review’, hoping to include concepts like, ‘lessons learnt’ and ‘how it might be done differently next time’. I have chosen it over words like ‘review’, ‘analyse’, ‘evaluate’ and ‘discuss’, because I find them too vague or too mathematical. This booklet is my explanation of ‘critique’ by example.

In terms of process, I am assuming undertaking a critique involves reading an article, using the questions listed in this book to think of an overall argument you wish to make and writing it into a ‘critique essay’ which itself has a very explicit, well justified, conclusion (the argument).

Why undertake a critique?

The intent of a critique is for learning to take place, yours and, where feedback is possible, the author of article. So, the way you go about critiquing an article needs to include methodically opening up the way you think. This means two ‘learnings’ need to take place. One is about the article being critiqued and the other is how best to undertake the critiquing process. This booklet provides examples for beginners but you should develop your own critique perspective and resulting set of questions. Alternatively you may think about how the questions can be used. By improving the critique process you will increase or get a better understanding of what you learn from the articles.

WHY ‘ARTICLES?’

The critique methods mentioned in this booklet can be applied also to books, chapters, lectures, courses, projects, human inquiry, human activity problems, and policy. The term ‘article’ was used only for brevity.

4

THE SIMPLE-ARGUMENT PERSPECTVE

Justification

This style of critique draws on the ‘argument as inquiry’ perspective, which in modern times is attributed to Popper’s [1963] book ‘Conjecture and Refutations’, Perelman’s [1989], ‘The New Rhetoric’, Walton’s [1998] ‘The New Dialectic’ and to van Eemeren’s [1987] the ‘Handbook of Argumentation’. By argument it not meant formal logic nor quarrels but rather reasoned debate where an article is required to have an explicit upfront conclusion (the argument line) that needs to be justified with supporting evidence. The article should lay out this evidence in the form of reasoning and/or empirics (descriptive or measurement). Counter arguments need to be anticipated and satisfactorily dealt with otherwise the conclusion will not be convincing to the critiquer. Critiquing an article using this perspective involves evaluating the argument and supporting evidence.

Start by looking for the one sentence argument (conclusion, point) of an article – it should be presented in the abstract, the introduction and the conclusion. The conclusion is often the safest place to look. Care needs to be taken to compare the explicit and implicit argument. The authors may clearly state an argument but you may finish reading the article under the impression there was an alternative implicit argument. The argument also needs to have some surprise value, be a little insightful or, as Popper argues, be risky, falsifiable. Arguing the sky is blue would not be very insightful, arguing that it was red in the early stages of the earth’s development might be. The innovation may be in the evidence. If you had novel evidence that the sky was blue, that may be convincing.

An article should not fail to convince you that its conclusion is justified merely because of poor definition of some key words. Technology, sociology and medicine are disciplines that have developed their own extensive vocabulary. History has not. It is the job of the author to communicate clearly with the intended audience, so an article can be criticised if it uses ill-defined terms. The greatest danger occurs when a word has several meanings and the reader is not alerted to that which the author is using. For example, the word ‘critical’ means negative, exact, nuclear, urgent and emancipatory. Scientists have spent many centuries defining their ‘technical ‘ terms but in social inquiry an author may need to spend some space defining, bounding and contrasting terms and concepts.

5

A reasoned argument needs at least two people. With article critique, this will be the author(s) presenting his or her argument to you. The arguers need to introduce themselves, their expertise in this area, their motivation for writing the paper, the motivation for why you might want to bother reading the article and to acknowledge they are presenting a justified conclusion. This background may assist in your acceptance of their evidence. It should also assist you in anticipating where their evidence may be weak.

There are two basic types of research articles, the discovery story and the justified conclusion. People who watch the New York TV series “Law and Order” will appreciate the difference. The cops discover who did it, the lawyers convince a jury. Analogous to the lawyers, in the justified conclusion style of article, the argument is presented upfront so you (or the jury) can decide if the evidence is convincing or not. Analogous to the ‘cops’, in the discovery style, very common in scientific writing, the conclusion may be presented as a discovery only at the end of the article. This seems a little bit misleading if the author knew this ending before he or she started writing the article! It is really just a style thing, as the cop story, like the science experiment, like the ‘who-done-it’ novel, is still trying to convince you who did it. But be careful of the lure of the detective novel. Weick (ref or who he is) warns that most academic topics are so boring that the audience leaves before the ending. A good critique will discuss argument style – the lawyer style.

Having located the argument (conclusion), the next part of the argument critique method is to evaluate the supporting evidence. There are also two (I think) types of evidence; reasoning and empirics. The classic use of reasoning evidence is Einstein’s [1939?] ‘Theory of Relativity’. He argues the presence and importance of relativity by using only reasoning. It is in the form of asking you to imagine you are in a moving train and throw an object out of the window. The path it appears to take is compared to the path it would appear to take if you were standing watching the train pass-by. The other sort of evidence, empirics, involves perceptions that have passed through the senses of the author, typically his or her eyes, sometimes as measurement, sometime as years of experience. Of course, an author can call on the experiences of other writers and their reasoning through the use of ‘the literature review’. The quality of evidence is not a simple or absolute thing. Science likes very exact measurement, social inquiry likes real insight and to treat the collection of empirics (experience through the senses) as a learning- by-doing action that assist the brain to find insights. The only suggestion I have to help you decide whether or not the evidence is adequate is to simply ask if it would be convincing to a knowledgeable audience.

6

Argument Questions

Ask yourself the following questions about the article.

Argument What is the explicit or implicit argument (conclusion) of the

paper? Was it stated upfront? What was their insight, ie was the argument novel, risky, open

to falsification?

Definitions Are all key words well defined (described)?

Arguers Who are the authors? Have they established their expertise? Why have they selected this particular argument?

Evidence What evidence is brought to support the argument

(conclusion)? Was this evidence convincing, novel, insightful? Was the counter argument fully considered? Were there any empirics? If so, why? Should there be?

Audience Who is the intended audience? Is the paper explicitly persuasive to this audience?

Motivation Is the importance of the argument fully explained? What was the problem? Is it an important problem? What did you learn from the article? How could you use it to improve people’s lives?

8

THE PROBLEM-SOLVING-SYSTEMS PERSPECTIVE

There are numerous species to this now very diverse genus, systems thinking. The species I am interested in seems to be traceable in the US to Boulding’s ‘General Systems Theory: the Skeleton of Science’ [1954], Churchman’s ’The Design of Inquiry Systems’ [1971] and his students Mason, Mitroff, Ackoff and Ulrich who have produced work similar to Weick and Argyris. On the other side of the Atlantic, Checkland [2000] strengthen the meme with his soft systems. The important point being that these writers see systems thinking as a broad scope, fairly generic approach, to appreciating human activity problems. An article can be seen as a human activity problem and its critique as a problem solving exercise. If the article is about research into some physical problem, like the overall mass of the universe, rather than a social one, like saving the environment, the systems thinking critique method may be less appropriate.

While still a developing concept, my interpretation of systems thinking for solving social problems is that it encourages these types of problems to be seen using five constructs. These include purpose, interconnectivity, connectivity, boundary, seeking new perspectives and learning from doing. These problem solving constructs can be used to critique an article.

PurposeOne distinguishing factor of a human activity is that it is purposeful. All the stakeholders are going to give it a purpose. Students may give a University the purpose of getting an education, the lecturers of being able to learn more about their discipline and the administrators to processing student numbers. The construct ‘purpose’ is used in systems thinking in place of the more common term ‘outcomes’ as it the there can be a range of acceptable outcomes that are still compatible with a single overall purpose. For example, students can select different courses as they learn from doing courses but still achieve their purpose of getting an education.

In a critique, the construct, purpose, can be used to ask questions like, what purpose did the authors have in writing the article? What purpose will the critiquer give to the paper? Will it merely be a stepping stone to a qualification or will it inform their inquires. If the authors are participating in the article, then what was their purpose in participating and what purpose will they give to the article? Moving on to the flexibility assumption in purpose, how else might the authors’ purpose have been achieved?

ConnectivityA human activity system is made up of interrelated sub-systems and is itself a sub-system of something else. One of the core constructs

9

of systems thinking is this appreciation of hugely complex, impossible to predict, inter-connectivity of systems- this is the context of the system under consideration. The article has been written with the wider international academic article publishing system, including the subject discipline system. Adding one more article to this complex of inter-related systems, or taking one, out will have hard to predict ripple-effects throughout this wider system.

The connectivity construct encourages questioning about what the article is connected to, for example, its place in the literature. “What else has the author done?” “Is the conclusion unique?” “What does the paper remind you of, how does it sit with what else you know?” “What other evidence is available?” “What ripple effects will it have on wider systems?” “What do you see as the wider system on which it will have the largest impact?” Put another way, “What genre is it from, what inquiry tradition, what school of thought?”

BoundariesWhile systems thinking encourages a more holistic approach to problems, what separates it from the universality of science is that lets people put arbitrary boundaries around problem domains. In this case it may be the article, the discipline literature or the research methodology. Boundary questions may include, “Iis the article complete?” “Are all the issues and concepts raised well defined and scoped?” “Does it present a self-contained story?”

Learning from ActionThinking of something as a system rather than as a thing encourages thinking about changes over time. What comes into the system, is processed, and what goes out? This is sometimes called the transformation construct in systems thinking. It should encourage temporal (time) thinking such as, “Does the date of the article matter and what is the pattern of events that lead to this article?” “What does this article change in my thinking (lessons learnt)?” Internal to the article the processes might be questioned. “What were the inputs to the article? What skill level was required of the reader? What data (empirical and previous literature) was the article founded upon? Does the article process these inputs well? Then what are the outputs of the article, the conclusion and recommendations? Are they fully justified?”

Stakeholders’ PerspectiveThe last construct of systems thinking for solving human activity problems can be very challenging to some people’s definition of knowledge. New knowledge can come from new perspectives. Put another way, human problems are often the result of different stakeholders having very different perspectives on the truth of the situation. Neighbours can disagree over the suitability of a fence.

10

Both opinions are fully justified, errors of fact can be sorted out but differences of opinion cannot be sorted out by measurement, no matter how exact. This leads to the suggestion that problem solving involves seeking the perspective of all stakeholders. “Does the article give all stakeholders a voice; or rather does the author talk for his or her subjects?” Historical articles about warfare are a good example here. “Are the perspectives of both sides, the leaders and the lead given?” To avoid relativism, these perspectives need to investigated and fully justified.

11

Systems Questions


Purpose What purpose has the author had in writing the article? How else might the author’s purpose have been achieved? What purpose will the critiquer give to the paper? What purpose do you think those participating had? What purpose will they give to the article?

Inter-Connectivity What is the article connected to? What is its place in the literature, disciple or topic? What else has the author done? Is the conclusion unique? What does the paper remind you of, how does it sit with what

else you know? What other evidence is available? What ripple effects will it have on wider systems? What do you see as the wider system on which it will have the

largest impact? What genre, inquiry tradition, and school of thought is it from?

Boundary Is the article complete, does it present a self-contained story? Are all the issues and concepts raised well defined and

scoped?

Transformation (Action and Learning) Does the date of the article matter? What is the pattern of events that lead to this article? What does this article change in my thinking (lessons learnt)? What were the inputs to the article? What skill level was required of the reader? What data (empirical and previous literature) was the article

founded upon? Does the article process these inputs well? Are the outputs of the article, the conclusion and

recommendations fully justified?

Stakeholders’ Perspectives Does the article give all stakeholders a voice, does it seek the

perspective of all involved? Are the stakeholders’ perspectives critiqued and/or justified?

12

PICTURING PERSPECTIVE

This approach to critiquing an article involves drawing a picture of the problem represented by the article. You may want to explore different ways of visually representing articles; one possibility is presented. It is based on Checkland’s [2000] ‘rich pictures’. The usefulness of picturing is usually in the process which encourages appreciation of the problem in the article rather than the finished picture being a useful means of communicating with others. However, picturing done as a group, say using a white board, ‘… improve(s) communication amongst people in the problem situation and build a deeper, more shared understanding’ Bronte-Stewart [1999].

An ExampleThe example of the “quarrelling bus conductor” will be used. This is a problem described by Ackoff (2000; www.judgelink.org) where a dispute has arisen between bus conductors and bus drivers as each tries to maximise their conflicting bonus scheme.

1) Identify the problem the article is addressing; this may involve restating the conclusion of the article. A problem is the gap between what is happening and what is sought. The problem makes up the picture’s title, in this case, the quarrel between the conductors and drivers.

2) Draw a large circle to denote what you identify as the system that bounds the problem, eg. the bus transport system. Write in the purpose of that system, eg. cheap, reliable transport around town.

3) Draw as nodes around the circle, the stakeholders, e.g. bus driver, bus conductor, passengers, inspector and so on…

13

Bus transport system

Cheap reliable transport

PROBLEM:The

Conductor and Driver

are quarrelling.

http://www.judgelink.org/

4) Put in the various stakeholders’ main concerns as text in their thought bubbles.

5) Draw as nodes on the circle the objects or things involved in the problem, e.g. bus stops, the bus, the depot clock, and so on…

6) Draw as nodes the organisations (cultures?) involved, eg the Unions, the Bus Company and the Local Government.

7) Ask yourself what tensions can you identify from the picture? Draw as a line between the nodes any tensions that you feel are relevant between the nodes, eg the inspector is checking up on the conductor; the driver wants to get back to the depot on time; the conductor wants more time to issue tickets; passengers don’t want to stop between their stops and the conductor wants fares from the passengers.

In an academic article, containing an experiment, the tension lines may represent validity or reliability issues.

8) Ask yourself how these tensions might be reduced?

14


Cheap reliable transport

Trip Clock

Collect tickets

check conductor

home, work…

Inspector

Conductor

Passengers

Driver

15

driver

Conductor

customersconductor

must not miss tickets

I’m on the clock

home

watch conductor Purpose

= cheap reliable …


unions

council

Bus Co.

Trip Clock

THINGS, ORGANISATIONS AND PEOPLE PERSPECTIVE

There has been a long tradition in dividing the world of human activities into three, the things (T), organisations (O) and people (P). Linstone, [1984, 1993, 1999] sometimes with Mitroff [1993], has expanded on this TOP approach, providing a useful way to critique articles. The complication is that TOP can be perceived a few different ways.

One perspective of T O P is that it represents different ways of knowing (epistemology). T represents the rational scientific way of defining knowledge as seeking objective facts, now associated with positivism and objectivism, which typically means quantification, analysis, reduction of problem into parts and calls for repeatability. O and P can be combined to represent a way of knowing that defines new knowledge as seeking new perspectives. It is an approach identified by Kant, Popper, Kuhn, Churchman and many other theory of knowledge writers. When critiquing, this can be used to ask if the article assumes new knowledge to be found by seeking objective, usually measured facts. Alternatively it may be seeking new perspectives.

The TOP division can be used to appreciate that people (P) have different perspectives on problems depending on their values (ethics) and experiences, each possibly available to learn from. National, departmental and organisational cultures (O) can over-ride these personal perspectives as situational responses overwhelm personality. It is assumed that things (T) don’t have a perspective; but some writers, especially in history pieces, can try to pretend there is no perspective being taken but merely THE truth is being presented. This may be labelled a T perspective. For critiquing, this means that you can reflect on which perspective the article has been written from, and how it would be different if written from another perspective. For example, the space shuttle Challenger disaster can be presented as a mechanical (T) failure, or as a failure of a few individuals (P) making bad decisions or as an organisational culture (O) issue with managerialism overriding the engineering culture. This example brings out the important difference between an organisational perspective and the personal one. The decision to launch is made by a spokesperson representing the organisation, so can be seen as a personal. However, the organisation as a level of system is more than a collection of individuals if you believe as a group certain emergent properties exist. Look for group effects like synergy and emergent properties in your article.

16

T.O.P Questions


Is the article about people, organisational cultures or things? How would it have been different if it were from a different

perspective? Have the different perspectives of the people and/or cultures

being provided? Have the people or cultural norms been treated as ‘things?’ or

as intelligent and experienced people who can inform the author?

Have all stakeholders had a voice?

Has the author treated the problem addressed in the article as one that can usefully be solved by using scientific methods, ie taken a technical perspective? If so:

o Is the problem usefully subdivided into parts that can be measured?

o Is the situation repeatable so the measurements can be confirmed?

o Is it realistic to exclude any variables that have been excluded?

o Which stakeholder is to judge the solution to be valid? o Is the evidence provided direct empirical or experiential?

Has the author treated the problem addressed in the article as best being solved by appreciating the perspectives of the stakeholders (P)? If so:

o Do you get to hear their perspectives in their own words?

o How were opinions justified?o Was the author cynical of whether the stakeholders

understood their own minds or actions?o Was any confirming evidence sought?o What did stakeholders think of other stakeholders’

perspectives?

If a cultural norms perspective is being taken, is there a clear distinction between cultural and personality? For example,

o Does the author assume the culture is more than the personality of a few dominant leaders?

o Are the emergent properties of the culture identified and relevant?

o How universal is the culture, are there contradictory sub cultures?

17

PERSPECTIVAL PERSPECTIVES Generalising from the TOP approach, the more generic perspective draws on the work of Kuhn’s [1970] ideas of theory laden observation, Popper’s expectations [1963], Churchman [1972] multiple inquiry methods, Linstone’s [1999] multiple perspectives concepts and Haynes’ [2000] Perspectival Thinking which draws on Polaymi, Hegel and Heiddegger. The underlying theory of knowledge is that it is useful to identify two types of knowledge. The scientific type that produces ‘objective facts’ is well enough known. The alternative is that knowledge is perspectival. New knowledge means finding a new perspective. Kuhn uses the word ‘paradigm’ to describe this.

This dual knowledge approach believes that it is informative to separate inquiry into the ‘thing’ being inquired about from how it is being perceived. To take a simple example, I might study an organisation (the object) from a managerial efficiency perspective, a learning perspective or as a source of reliable income (3 possible perspectives). Critique involves attempting to separate these two types of knowledge, the thing being studied from the perspective.

Two broad types of perspective have been identified. One is like that used by Linestone [1999], Kuhn [1970] and the Critical Social Theorist. It is externally provided. For example, Linstone suggests engineers use his TOP to remind themselves to not only think of their normal technical (T) perspective, but also to think of the social (O or P) issues related to whatever project they are working on. Kuhn, and many others, argue that the dominant scientific perspective is imposed without discussion. At one time, it was a clockwork universe, then it was one full of attractive forces, now it is a relativistic one. Another example, one from the social inquiry, is Critical Social Theory which advocates seeing human activity through an emanicipatory perspective. So when critiquing, you can ask if the author is trying to suggest a new perspective.

The other sort of perspective is more implicit and personal. Unprompted, people perceive new problem situations in different ways depending on their past experiences and their values. Terms like real interests, worries, theories of action and people’s concerns align with this concept of an implicit perspective formation that make us pre-judge, or appreciate situations in certain ways. A

18

Perspectives

Object

Mode of Inquiry

critique might try to identify the concerns or primary perspective of the author (or maybe some participants in the article).

19

Perspectival Questions


What is the article about, what is the thing, the object under consideration? Think of this thing as being a system and ask yourself the system critique questions:

o Are its boundaries well defined, what can it be contrasted with?

o What is its purpose?o What does it change over time?o What is it connected to? o Who are the stakeholders?

What perspective is being taken of the object under study? o What are the origins of this perspective?o Is it an external perspective or a concern?o What are the limits of this perspective?o Is it part of a wider perspective? o What other perspectives might have been taken of the

object under study?

20

EMPIRICS-AS-ACTION PERSPECTIVE

There appears to be some very separate understandings about the role of empirics in research. Some philosophers and critical theorists seem to think empirics are more trouble than they are worth, rarely using them in research. Reasoning and thought experiments are considered sufficient to create useful knowledge. Measurement problems, the lack of reliability of the human senses and a lack of explanation of why the observation occurred contributes to this attitude. Moreover, Marx’s, who has influenced many social theorists, has a particular position on empirics [Sowell, 1985, chp.2]. While Marx called his work empirical, he was not interested in the mere appearance of something. Rather first it is important to understand the underlying processes in tension going on behind the appearance of something. For example, a married couple can be living, eating, sleeping together but not be happy with their relationship, to understand the marriage you need to understand the underlying tensions. The same is true of a caterpillar; you cannot explain the appearance of a caterpillar unless you understand it is about to change into a butterfly. Put another way we have to learn to be able to see. A baby has to learn which sets of colors and shapes makes up a tree and a surgeon has to learn he or she is seeing when he/she opens up a body. Social researchers have to reason what is going on behind an observation. This can be used to critique empirics by asking what underlying tensions has caused the thing you are looking at to be there as it is.

This lack of centrality of empirics to research can also be found in some areas of science. Einstein’s work was mainly mathematical and mind experiments (anologies). His 1939 book only draws on analogies as aid to reasoning. How the author perceives or uses empirics provides a perspective with which to critique an article. You can ask whether the author explicitly uses reasoning or empirics as evidence to convince you of the conclusion. However, in the positivist tradition, empirics are central, especially the experiment. So you can ask if precision of measurement is considered insightful observation. Does the author display observation or measurement skills greater than a lay person or common sense? If so, then these are the hall-marks of the positivist scientist. Further, does the style of the author suggest that they are reporting the truth not a perspective, and that there is only one correct perspective? This is usually associated with the assumption that the observations can be repeated to get the same result and the observer is independent and unbiased.

The American Pragmatism approach to empirics seems to be slightly different. It sees empirics as problem solving by action, an experience, which enables the brain to draw on more analogies.

21

The emphasis is not on seeking independent objective observations but to improve reasoning from action, including think of different ways to perceive the problem. This emphasis has been taken up in the action science, action research and learning from doing literature. In particular, Weick [19??] Argyris and Schon [1996], Ackoff [2000] and Checkland [2000] have taken it up as a problem appreciation method characterised by the learning or reflective loops approach. It advocates learning from a series of small trial and error learning loops rather than the ‘one big plan’ approach. Look for these assumptions in the empirics in an article.

Moreover, if root cause or double loop learning is to take place, then the effects of the action (eg collecting empirics) need to be reflected upon both in terms of the object being studied and on the perspective being taken of that object (see perspectival thinking). So, if an organisation is the object, the perspective is adaptability, and the action (node of inquiry, the empirics) is the introduction of a new information system, then the empirics need to be used to revise our understanding of both organisations and adaptability. This introduction might be undertaken in small stages so that reflective loops on both of these factors can take place. When critiquing an article look for empirics being used to provide actions to reflect on an upfront expectation.

22

Empirics Questions


Why Empirics? Why does the article have empirics, why does it not? Are the empirics intended to provide objective knowledge or

merely to assist thinking? Could any empirics be replaced with an analogy, reasoning or

mind experiment? Do the empirics convince you of the articles conclusion any

more than the reasoning evidence did?

Underlying Tensions Empirics What are the underlying tensions or forces that produced the

‘thing’ you are looking at as it now appears before you?

Empirics for objective knowledge Are the authors seeking the one truth or do they acknowledge

alternative interpretations of the empirics? Do the empirics produce convincing, objective knowledge? Are the empirics repeatable? Are the empirics generalisable to many other situations, will

they remain valid across time and universally around the world?

Are the empirics all the evidence that is provided towards the conclusion?

Is there any acknowledgement that empirics are ‘theory laden’?

Empirics for learning from action Were the actions that produced the empirics seen as an

exercise in learning from action? Was there any attempt to undertake a series of small actions

and reflect on each so as to redirect future actions to collect empirics?

Is there any evidence of the author or others changing their perspective as a result of the empirics?

Have all stakeholders been given an opportunity to provide their perspective on the actions that created the empirics?

What other modes of inquiry (action) might have provided other learning?

Was there any reflection as a result of the actions against the perspective (intellectual frame, theoretical construct) informing the empirics?

23

ROOT-METAPHORIC PERSPECTIVE

Morgan [1986], in his ‘Images of Organisations’, classified the literature on organisational theory by root metaphor, a concept discussed earlier by Pepper [1942]. His book suggests that, when people wrote about organisations, it was possible to identify certain common root metaphors. For example, Taylorism assumed a machine root metaphor making people like cogs in that machine. Another example, those in education often complain about, is the assumed root metaphor by some education administrators that knowledge is an object that lecturers pass over to students for money. He identifies a number of these assumed root metaphors, suggesting it is a useful critiquing exercise to see if you can spot the root metaphor in an article. Social scientists often use a root metaphor that the subjects of their research are less intelligent than themselves. Johnson and Lakoff [] provided a popular discussion on metaphors while Ortney [] discusses how metaphors affect our thinking. There was a flood of papers that appeared in the 1980’s and early 1990’s trying to identify root metaphors in various human activities. It is hard to provide exact direction on how to spot root metaphors. All I can think to suggest is that you read through the article and write down any metaphors you can identify. From this list ask yourself if there is a collective name, theme or root metaphor that could be used to sum up the list. For example, I recently asked some fellow academics for metaphors of a literature review. Their suggestions are listed below. I have separated them into two as I felt there were two root metaphors.

Group 1These metaphors of a literature review have been allocated to the root metaphor of ‘investigation’ (seeking out, discovery, detection, journey). Undertaking a literature review is a search for (in) the unknown, going out to find out something you don’t know.

map ("go where you like") detective (seeking clues for new research directions) beachcomber (sifting through others' castoffs) prospector (seeking gems or nuggets). lens (to focus readers), pathfinders (layout an area and point to key sources) sifting and winnowing funnel (funnelling in) signpost (to find your empirics) concertina (narrowing and enlarging your search, like a

concertina windbag) focus mechanism (you are talking about this, not that) releasing the imagination

Group 2

25

These metaphors of a literature review have been allocated to the root metaphor of ‘prosecution’ (justification, evidence, proof, argumentation). The literature is used to support a reasoned argument, justify a conjecture, or validate a hunch.

expert witnesses (courtroom evidence... ). currency (to buy credibility) building blocks, concrete foundation, the history, mirror (to see oneself in context) requirements document credibility filter (whose work do you draw on, what is the

source of your ideas?) due diligence (have you acknowledged others whose ideas

you draw on?) social courtesy (can you be relied on not to steal ideas?) situational context (what conversation is this paper a part

of?) family tree foundation (you are educated and building on a tradition) puzzle (your idea fits into this larger puzzle)

26

Metaphor Questions:

Having listed the metaphors in the article ask yourself the following questions about the article in order to find a root metaphor that suits you.

People: Are the people in the article treated like ‘things’ that need to

be organised or as sources of knowledge? Is any human activity treated as though it occurred in a linear

sequence (from A to B to C) rather than in a recursive, or interactive one?

Are the suppliers of capital assumed to be superior to people investing their time (labour)?

Is gender considered relevant? Does the article take a personal perspective or organisational

one? Is technology assumed superior to humans?

Organisations Are any organisations presented as machine like? Are any organisations like adapting, evolving organisms

interacting with their environment? Are any organisations liberating or oppressive to

stakeholders? Is there any indication of the dependency or organisations on

the broader community? Are any organisations depicted as flows of information rather

than people interacting? Are the interpersonal and/or interdepartmental power issues

made explicit? Is efficiency and effectiveness a communal problem, not

related to the desires of particular stakeholders? Is Government treated as more powerful than the market

place?

Things Do any machines have gender, or are they analogous to any

body parts? Do problems exist separately from the people who have the

problem? Is information and/or education treated as a commodity? Is equal access to information assumed?

27

DIALECTIC PERSPECTIVE

The word ‘dialectic’ is being used here in the same way it is used in the organisational change literature [Nielsen, 1996; Mason 1996; Morgan 1986] to refer to “change that emerges from the interplay of conflict and among differences and affirmation of areas of agreement”. Nielsen [1996] identifies five types of dialectic processes which he explains using terms like ‘iteration’, ’spiralling’, ’up building’, and ‘transformation’ which can be involved when there is a social reconstruction of human activities. He argues (in a dialectic with the reader?) that these change processes differ in terms of emotionality, they do not have to be quarrelsome, aggressive or conflictive but they can be. Identifying these types of dialectic can form a further method of critiquing articles.

In ‘interaction dialectic’, an idea (conjecture) is constructively refuted with the intent of developing an improved idea. The action science dialectic is analogous to Argyris and Schon’s [1983] idea of experiment, reflect, and experiment learning loops. The up-building dialectic is similar to the ‘not invented here syndrome’ where one human group develops a culture of how change should occur that an outsider does not respect. There is then tension between these two groups. For strategic dialectic, Nielsin draws on the work of Mason [1969] who suggested that alternative plans of the future can be used to develop a synthesis or improved third plan.

The transformation dialectic is about the underlying social and political forces between group vying for resources. It is hard to mention this version of ‘dialectic’ process without mentioning the hugely influential work of Karl Marx. Sowell [1985], a Marxist economist, argues that their inquiry approach was to consider the ‘underlying dialectic forces’. In their case, this was mainly the underlying political and social forces associated with the class struggle. Nielsen identifies more ‘localised’ struggles that go between participants of human groups engaged in a common activity. For article critique, this can be translated to suggest you think about the underlying tensions inherent in the situation outlined by the article.

28

Dialectic Questions


Iterative Dialectic Does the author systematically question the article’s

conclusion? Is the author suitably suspicious of his or her own conclusion? Is there a specific perspective guiding this suspicion? Are improvements to the ideas under suspicion being

suggested?

Action-Learning Was there any action undertaken simply as a lets “try it and

see what happens”? Were any empirics (research) seen as a learning-from-doing

exercise? Was there a full and systematically post-mortem of all actions? Was there any response to unanticipated results? If so, did it

lead to learning?

Up-building Is there any suggestion of tradition or culture about how the

article was developed? Was there any consideration of how ‘radicals’ or ‘a devil’s

advocate’ would have responded to the activities depicted in the article?

Have all the possible responses to the article been given a voice?

Strategic Does the article set up any sort of dialectic between two

alternatives? Do they get equal consideration? Can you think of an alternative that might (also) be set to

what is used in the article? Is there any evidence of the article trying to learn by setting

up an interaction between two alternatives?

Transformation Can you identify any underlying tensions between human

groups vying for resources?

29

EVOLUTION-METAPHOR PERSPECTIVE

This perspective draws on the metaphor of an article being a species, a biological entity. More specifically, an article is to be thought of as the reproductive young of some species; something’s baby. The perspective is inspired by the work of Darwin, Dawkins [1989] and Dennett [1996]. The basic tenet of evolution is that the young are a random genetic variation on the genetic possibilities of the parents. The environment at the time of its growing to maturity determines whether the variation is able to reproduce its particular attributes to another generation. In most species, many more are born than reproduce. The appropriate level of analysis is the gene pool or the survival of the species not the individual. Birth and death is seen as being essential to improving adaptability to the environment.

The article can be thought of as the newborn. The parents being the major influences on the author, the genus being the school of thought the article comes from. The degree of variation is how different the article is from the parents, who may or may not admire their offspring’s features, but the environment will decide success, whether later authors are influenced (cite) by it. Publication may be analogous to birth; working papers and drafts can be seen as failures to reach birth and the environment the audience. Some species can alter their environment in order to survive, others cannot. So, when critiquing an article, examine the references etc

30

Evolution Questions


From the references can you identify the genus of the paper? From the references can you identify the parents? Would the parents have admired this article or not? Checks the citations index, do later authors use this article? Were the authors influential enough to alter their

environment? Would the environment in which this article was published be

hostile, friendly or manipulated by the article? What environment might have treated the article very

differently?

31

POWER PERSPECTIVE

Boulding [19??] talks of the three faces of interpersonal power; economic, destructive and integrative. Economic power comes from being able to allocate resources, destructive power comes from being willing to not participate in some activity, and integrative power comes from manipulating the human need to belong to social groups. Critical Social Theory [Alvesson and Skodberg, 2000] focuses on institutional power. Members of institutions have power from their formal positions. Journal editors accept and reject certain research methods, politicians have the power to influence legislation, the police from having to process law-breakers, doctors by having patients seen as victims, scientists by determining what is good knowledge and teachers by claiming to be more knowledgeable than their student. Markus [19??] identifies power from technical knowledge that can be extended to knowledge over regulations and processes (bureaucratic power). The feminist movement has been very effective in increasing awareness of the implicit power given to certain groups through language and Kuhn talks of the power of paradigms inside which people act while not appreciating many of their own assumptions.

These perspectives can be used to critique articles, look for the implicit power and persuasion in an article both between the author and the reader but also between any participants in the article. Be emancipatory, help the disadvantaged to see the causes of their stress, look behind the curtain of institutionalised power, assume there are no tall poppies, but rather some groups of people exploiting another group of people.

32

Power Questions


Who determined that the topic of the article is important? Do the authors use their position in an organisation, or

superior insight, to convince rather than using reasoning or empirics?

Is the journal pushing any particular topics of research or research methods?

Is science assumed to be the only reliable source of knowledge?

Are all contributors to the article’s conclusion acknowledged? Does the field of study have a small group of ‘experts’ who

gatekeep on journal article acceptances? Are there any rival journals that intentionally take an

alternative approach?

Within the Article Does the article reveal any struggle for resources from the

point of view of all those involved? Does every point of view get a voice? Is the language exclusive, acting as a barrier to outsiders and

the application of common experience? Are there any appeals to authority? Are references justified in terms of what the author’s studies

or on the basis of quality of journal? Are all the counter arguments fully explored? How would low-income people view the article and its

contents?

33

FULLER ARGUMENTATIVE PERSPECTIVE

The author has a dominant perspective that argument is the basis of human communication, the creation and testing of knowledge. This perspective has been well argued by Pereleman, van Emmeren, Walton and Rehg. Moreover, I need a way to wrap up all the diverse critique perspectives presented in this booklet! Therefore, this last section will attempt to combine all the perspectives presented. The umbrella perspective will be simple argument, with the others providing detail to expand this critique perspective.

34

Argumentation Questions

Arguers Who are the authors? What is the basis of their expertise? What else have they published?

Audience What is the intended audience? What audience would it not work for? Is the paper explicitly persuasive to this audience?

Object or Thing Under Study What is the object or ‘thing’ the article is studying? Did they discuss, define this ‘thing’ in system thinking terms;

its boundaries, relationships (context), how it changes over time, and its purpose?

Concern What is the authors’ concern (perspective, lens, frame) on the

objects under study? Did they scope this concern or perspective? What is the problem they seek to address?

Argument What is the explicit or implicit argument (conclusion) of the

article, was it stated upfront? What was their fresh insight, i.e. was the argument novel? Can you draw a picture of the article?

Definitions Are all key words well defined (described, bounded, scoped)?

Motivation Is the importance of the study fully explained? What is the purpose of the authors writing the article?

Evidence How were they (implicitly) defining knowledge? How did they know things? What evidence is brought to support the

argument/conclusion? Was this evidence convincing, novel, insightful? Was the counter argument fully considered? Where there any empirics if so why, if not why not?

Empirics as Learning

35

Were they treating empirics as an exercise in precise measurement to produce objective knowledge or as action to learn?

Did they build and test something, what and how? Was the testing integrated in the building and evaluated using

learning loops and multiple stakeholders’ perspectives? Was the evaluation convincing? Were all stakeholders given a voice? Were they searching for ‘the truth’ or a range of perspectives?

Implications Were the implications (the so-what) of accepting their

argument/conclusion fully explained? What good knowledge was created? What actionable knowledge (rules of thumb,

recommendations) was created to aid future decision-making?

Lessons Learnt What did you learn from the article? What purpose do you give to the article, how could you use it

in your own studies?

© Mike Metcalfe, June 2002

36

REFERENCES

Argyris C. and Schon D.A. (1996) Organisational Learning II, Mass: Addison Wesley.Arygris C. (1996) Actionable Knowledge, Journal of Applied Behavioural Science, 32(4), 390- Boulding Bronte-Stewart, M. (1999), Regarding Rich Pictures as Tools for Communication in Information Systems Development, Computing and Information Systems, Vol 6, pp83-102. Chalmers AF 1982, What is this thing called science, U of Queensland Press

Checkland, P., (2000), “Soft Systems Methodology: A Thirty Year Retrospective”, Systems Research and Behavioural Science, Vol 17 Number 1, pp S11-S58,

Churchman, C. W., (1971) The Design of Inquiring Systems, Wiley, New York.Churchman C.W. (1979) The Systems Approach and Its Enemies, Basic BooksCohen H.F. (1994) The Scientific Revolution, University of Chicago Press.Dawkins, R. (1989), The Selfish Gene, Oxford: Oxford university pressDennett D.C, (1996), Darwin’s Dangerous Idea, NY: TouchstoneDillon, J., (1988), “Questioning in Science”, in Meyer (ed) Questions and questioning, Walter de Gruyter & Co, Berlin

Eemeren, van F.H., Grootendorst, R. and Kruiger, T., (1987), Handbook of Argumentation Theory, Dordrecht: Foris Publications.

Haynes J, 2000, Perspectival Thinking, NZ: OneIdea Company

Johnson and Lakoff …

Kuhn T.S., (1970), The Structure of Scientific Revolutions, 2nd Ed., University of Chicago Press.Linstone, H. (1984) Multiple Perspectives for Decision Making: Bridging the Gap Between Analysis and Action, Elsevier Science Pub Co, NY. Linstone H A. [1999] Decision Making for Technology Executives : Using Multiple Perspectives, Artech House Publishing: BostonMarkus L. (19??) … powerMason, R O [1969],“A Dialectical Approach to Strategic Planning,” Management Science, Vol 15, B-403 – B-414, Mason, R.O. and Mitroff, I.I. Challenging Strategic Planning Assumption, John Wiley and Sons, New York, NY, 1981. Mason, R O [1969],“A Dialectical Approach to Strategic Planning,” Management Science, Vol

37

Mitroff, I and Linstone H, The Unbounded Mind, Oxford University Press, 1993.Morgan G., (1986), Images of Organisations, Calif.: Sage Publications.Ortney…Perelman Ch. and Olbrechts -Tyteca L., (1969), The New Rhetoric: A treatise on argumentation, University of Notre Dame.Popper K.R. (1963) Conjectures and Refutations, London: RoutledgeRehg W., (1999) Argumentation Theory and the Philosophy of Science Since Kuhn, Inquiry, Summer Schon, D. (1983), The Reflective Practioner, NY Basic Books.Sowell T (1985), Marxism, London: UnwinUlrich, W. (1983) Critical Heuristics of Social Planning, Wiley, New York.Walton D., (1998), The New Dialectic, Toronto: Tronton University Press.

38

Metcalfe critique2

Education

Transcript of Metcalfe critique2