Open Research Onlineoro.open.ac.uk/41879/1/__userdata_documents6_mdr66...regarding the take-up of...

Open Research OnlineThe Open University’s repository of research publicationsand other research outputs

(Breaking) The iron triangle of evaluationJournal ItemHow to cite:

Reynolds, Martin (2015). (Breaking) The iron triangle of evaluation. IDS Bulletin, 46(1) pp. 71–86.

For guidance on citations see FAQs.

c© 2015 The Author

Version: Version of Record

Link(s) to article on publisher’s website:http://dx.doi.org/doi:10.1111/1759-5436.12122

Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.

oro.open.ac.uk

http://oro.open.ac.uk/help/helpfaq.html

http://dx.doi.org/doi:10.1111/1759-5436.12122

http://oro.open.ac.uk/policies.html

1 IntroductionI use it [systems thinking] often. But I can see whypeople surrounding me don’t use it often. They thinkit’s abstract. It’s not solving problems the way theywould like…

Gets a bit atomistic with people proselytising theirfavourite words and models and flogging old horseswith new names, losing its pluralistic origins…

The statements come from some free textresponses on a small (though global) onlinesurvey (86 respondents) amongst professionalswith knowledge of systems thinking in differentdomains (Lachica 2013). Despite a clear need forcountering prevailing reductionist approaches toimpact evaluation, the comments flag issuesregarding the take-up of systems ideas inevaluation practice. One issue is a reluctance toemphasise the systems idea as indeed an abstractidea – as though abstracting from reality issomehow neither normal nor helpful. A secondissue is that systems thinking is often regardedas something quite separate from, rather thanpotentially integral to, other practices. Bothissues can lend themselves towards maintaininga model of business-as-usual in evaluationthinking in practice.

Formulating evaluative questions of impact fromreal-world complexities requires a process ofabstraction. The problem of impact evaluation asdescribed through articles in this IDS Bulletinmight be summarised in terms of a broadeningof the questions to be addressed; from concernsof impact in narrow terms of outputs to concernsabout wider outcomes being effected. A shift inattention amongst evaluators is generallyacknowledged; from exploring questionsregarding the linear net effect of an intervention(project, programme or policy) towards questionsregarding whether an intervention made adifference to the situation, what the differencewas, how the difference was made, and whatdifferences might have been made elsewhere. Itrepresents a shift from being systematic towardsbeing more systemic. Such a shift in questioningactually involves going up a level of abstractionin order to deal with the complexities ofinterventions.

Flood and Carson (1993) suggested that thepurpose of applied systems thinking is to dealwith situations of complexity. A cherishedhallmark of good systems thinking is theaspiration to make simple the complex. Such anaspiration may not sit comfortably amongst

(Breaking) The Iron Triangle ofEvaluation*

Martin Reynolds

Abstract Ideas from complexity science and systems thinking are demonstrably helpful in a shift from

exploring (systematic) linear net effects of an intervention towards exploring wider (systemic) effects

occurring elsewhere. But where these ideas of ‘impact’ are coupled with a narrow use of the contingency

approach, some less helpful ‘triangulated’ relationships might be evident. These relationships might be

regarded in terms of an ‘iron triangle’, a metaphor used frequently to exemplify pernicious relations of

power. The most notable expression of the iron triangle is the ‘military–industrial complex’. This article

briefly outlines generic features of the iron triangle in terms of ‘systemic triangulation’ – an idea linking

three core systems concepts of interrelationships, perspectives and boundaries. Drawing on a tradition of

systems thinking in practice, an associated systemic triangulator is introduced as both a diagnostic and

planning heuristic; a device for not only diagnosing symptoms of an evaluation–industrial complex but for

prompting ideas towards a more benign evaluation–adaptive complex for impact evaluation.

IDS Bulletin Volume 46 Number 1 January 2015 © 2015 The Author. IDS Bulletin © 2015 Institute of Development Studies

Published by John Wiley & Sons Ltd, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA

71

4 IDSB46.1 Reynolds.qxd 17/12/2014 09:13 Page 71

those who sometimes confuse systems with thereality of a situation; or as the age-old systemsadage proclaims – those who confuse the map forthe territory (Korzybski 1995 [1933]). Thedownside of an aspiration to simplify is a rebukethat systems thinking can be too abstract. At onelevel the rebuke is well founded. When a systembecomes too divorced from the realities that itneeds to address, its reliability can bequestioned. There is clearly something wrongwhen, say, a health system assumes that sicknesscan be fixed solely by increasing medication, or apenal system is assumed to work purely on abasis of punishing offenders, or an economicsystem assumes profit motive to be the soledriver of economic success. Likewise, whensystems of impact evaluation are simplified tothe extent of assuming a reality of linearcausality with independent parts – such as withsome extreme versions of randomised controlledtrials (RCTs) – then the impacts measured arelikely to be over-simplified and hence bear littlesense of resemblance to reality.

But let us be clear and unambiguous, systemsthinking is an explicit endeavour of abstractingfrom reality. The challenge is to generateabstracts that have not just reliability towardsmapping realities of the real world, butresonance and relevance to the situations beingaddressed. So rather than having systems ofimpact evaluation based on, say, simpleabstractions of linear, mechanistic cause-and-effect, it may be more appropriate to go up afurther level of abstraction – mirroring the shiftfrom systematic to systemic impact questioningnoted previously – for example, to considersystems with parts interrelated in non-linearways, creating feedback loops, tipping points,path dependencies, time delays, and so on, allcontributing to impacts that may not be so easilypredicted. The contributions to this issue of theIDS Bulletin provide different means of using thisunderstanding of reality. Many draw on thehelpful idea of reality abstracted as a complexadaptive system (CAS) (cf. Patton 2010; Forss,Marra and Schwartz 2011; Ramalingam 2013).

A key feature shared by many authors in the useof CAS is the assumption that not all situationscan be identified as complex, and therefore notall situations demonstrate features of a complexadaptive system. There has emerged a ritualisedpractice in evaluation with supposing that

situations can be mapped from the outset intobeing either simple, complicated, or complex –an idea initiated by Ralph Stacey in terms oforganisational management (Stacey 1992) butlater adopted in the evaluation field(cf. Glouberman and Zimmerman 2002). Suchthinking invokes a contingency approachwhereby the type of method adopted for anevaluation is contingent upon the type ofsituation. A popular metaphoric expression ofsuch situations is in distinguishing between threedifferent activities: (i) baking a cake; (ii) sendinga rocket to the moon; and (iii) bringing upchildren. Whereas the former two activitiesmight respectively be regarded as being ‘simple’(following recipe instructions) and ‘complicated’(requiring address to many variables, butnevertheless predictable in outcomes), it is onlythe latter skill in parenting that invokes‘complexity’ where the situation is volatile,uncertain and highly unpredictable.

Chris Mowles (2014) provides an insightfulcritique of contingency theory in relation to theuse of complexity ideas in evaluation. His coreargument draws explicitly on Stacey’s laterrejection of his own tripartite heuristic (Stacey2001) by further suggesting that it might bebetter to assume that all situations are complex.Situations comprise humans and humaninteraction is always complex and emergent. Asystems thinking perspective might go furtherand suggest that all situations consist ofinterrelationships that may be regarded at anyone time as being both (i) complicated –interconnected – as evidenced in, for example,chaos theory, and (ii) complex – involving people,even if only as mere observers, and thereforeinviting multiple perspectives. Furthermore, acritical systems thinking perspective might regardall situations as (iii) conflictual – with differencesin relations of power circumscribing what isdeemed to be important with regard to boundariesof interconnections (what’s in and what’s out)and boundaries of perspectives (whose viewpointsare more privileged and whose viewpoints aremore marginalised) (Reynolds 2011).

There are two points of departure here fromcontingency theory. Firstly, as with Mowles, withsystems thinking in practice we would argue thatthere are no actual ‘simple’ situations, given that‘simplicity’ represents a particular perspective on asituation rather than ‘the’ actual attribute of the

Reynolds (Breaking) The Iron Triangle of Evaluation72


situation. An artefact from a situation such as adocumented project, programme or policy may ofitself be regarded as ‘an’ attribute. As ‘an’ artefactit may be regarded as simple or even complicated(depending on the relative number of variables –including different perspectives – being dealtwith). But the artefact in and of itself is notcomplex. The complexity arises in the auditing,planning, implementing and evaluating of theintervention – that is, where and when differentperspectives are actually at play. Secondly, whilstany situation might helpfully be regarded ascomprising all three interlocking features ofcomplicatedness (through interrelationships),complexity (through multiple perspectives) andconflict (through boundary judgements), thereremains a question on whether it is helpful todelineate these situational features in order toalign particular evaluative methods – the‘horses-for-courses’ thinking underpinning muchof contingency theory. Whilst there are now avast array of different evaluative methods,including different systems and complexity tools,it is perhaps helpful to remind ourselves thatsuch tools are conceptual devices rather thanreified objects; and hence open to change,adaptation and innovation depending on the userand context of use. Effectiveness may notnecessarily be contingent on ‘the method’ andthe ability to adopt new tools and associatedlanguage, but rather the user’s experience andcraft skills in dealing with all three features of anevaluand – complicatedness, complexity andconflict – using whatever tools with which s/hemay have experience.

The distinction between situations and systemsis key to diagnosing potential problems ofcontingency thinking and the helpfulness ofcomplexity thinking in impact evaluation, whilstavoiding the pitfalls of seemingly being tooabstract and/or too atomistic. Situations mightbe referred to as ‘messes’ as distinct from‘systems of interest’ (Ackoff 1974), or as ‘wickedproblems’ – or even ‘super-wicked problems’ suchas with climate change (Levin et al. 2012) – asdistinct from ‘tame problems’ (Rittel andWebber 1973). Systems thinking is therefore aprocess of making simple the complex; a processof abstracting from reality.

This article will take up a further level ofabstraction. Whereas a complex adaptive systemcan provide a helpful abstract proxy to individual

interventions being evaluated, a similar systemsidea can be used as an abstract heuristic forunderstanding the wider set of influencesassociated with the political economy of evaluation(see also Winckler Andersen in IDS Bulletin 45.6,2014). The simplicity of the system capturesrather than detracts from the reality of theevaluation process. The proposed device is basedon ideas of ‘systemic triangulation’; a genericdevice that we might call a systemic triangulator.

After briefly introducing the idea of systemictriangulation and considering its relationshipwith the iron triangle metaphor, the politicaleconomy of evaluation expressed in terms of anevaluation–industrial complex will be explored.By using systems thinking in practice somequestions are prompted regarding the influencesbetween an evaluand, evaluators and evaluationswith particular attention to contemporaryimpact evaluation. The article finishes with somethoughts on what a more benign system ofimpact evaluation might look like.

2 From systems concepts to systemic triangulationThe three core concepts associated with systemsthinking – interrelationships, (multiple)perspectives and boundary (judgements) – marka significant improvement in understanding andengaging with essential features of systemsthinking. These are generally regarded asstandalone entities which may resonate withentities of value and learning (Hummelbrunnerand Reynolds 2013; see also article byHummelbrunner in this IDS Bulletin). But theseentities might also be transformed into an activeendeavour – a heuristic of systems thinking inpractice – comprising verbs as well as nouns;understanding interrelationships, engaging withmultiple perspectives and reflecting on boundaryjudgements (Reynolds 2011, 2013; Williams, thisIDS Bulletin). In this article the heuristic isadapted further to provide a template forevaluating the political economy of evaluationprocesses, something that was also suggested byWinckler Andersen in IDS Bulletin 45.6 (WincklerAndersen 2014). To do this I will draw on one ofthe most profound generic expressions of systemsthinking – the notion of systemic triangulationdeveloped by Werner Ulrich (1998, 2000, 2003).

Systemic triangulation represents an extensionof ‘triangulation’ familiar in the empirical socialsciences. The extension is to rely not only on

IDS Bulletin Volume 46 Number 1 January 2015 73



Figure 1 The systemic triangulator

Boundary judgements of systems thinking in the partialworld of human activity

3 Systems for change(Reflecting on boundary judgements)

2 Practitioners makingchange

Value judgements in‘multiverse’ world of multiple

stakeholders with multipleperspectives

1 Situations of change

Factual judgements in the‘universe’ of an interdependent

and interrelated world ofrealpolitik

(Engaging with multipleperspectives)

(Understandinginterrelationships)

different methods and theories for understandingreality (interrelationships), but also for engagingwith alternative value sets (perspectives) andreflecting on reference systems (boundaries) as asource of assessing real-world practice (cf. Ulrich2003: 334). Systemic triangulation refers to whatUlrich calls an ‘eternal triangle’ ofinterdependence between judgements of ‘fact’,value judgements and boundary judgements(Ulrich 1998, 2000):

Thinking through the triangle means toconsider each of its corners in the light of theother two. For example, what new facts becomerelevant if we expand the boundaries of thereference system or modify our valuejudgments? How do our valuations look if weconsider new facts that refer to a modifiedreference system? In what way may ourreference system fail to do justice to theperspective of different stakeholder groups?Any claim that does not reflect on theunderpinning ‘triangle’ of boundary judgments,judgments of facts, and value judgments, risksclaiming too much, by not disclosing its built-in selectivity (Ulrich 2003: 334).

Figure 1 illustrates the influences between thesethree entities. The systemic triangulator buildson Ulrich’s original figurative triangle of facts,

values and boundary judgements (Ulrich 2000:252). In my rendition of the triangle, ‘facts’relate to the situations of change in the realworld associated with infinite interrelationships,‘values’ relate to practitioners managing change,associated with the core idea of multipleperspectives, and ‘boundaries’ relate to theideas/tools/models used for decision-makingabout managing change, associated with the coreidea of reflecting on boundary judgements.

Making boundary judgements is, as Ulrich pointsout, a ‘partial’ activity in two senses of the word:(i) partial in representing only a section ratherthan the whole of the total universe ofinterrelationships (what’s in and what’s out?); and(ii) partial in serving some stakeholderperspectives or interests – including practitioners’interests – better than others (whose views areprivileged and whose are marginalised). Theinevitable partiality in systems thinking makesclear the critical limits of being ‘holistic’ – gettingthe big picture – (cf. Williams, this IDS Bulletin)and of being ‘pluralistic’ – being inclusive ofstakeholders whilst claiming an unbiased view orvalue judgement on different perspectives(cf. Hummelbrunner, this IDS Bulletin).

The idea of systemic triangulation is itselfstrongly influenced by traditions of critical social

Source Adapted from Reynolds (2013) and the original ‘eternal triangle’ of Ulrich (2000: 252).



theory – particularly Jurgen Habermas – andAmerican pragmatism – particularly the works ofCharles Peirce, William James, John Dewey,Edward Singer and C. West Churchman (seeUlrich 1983 for a detailed scholarly account of anearlier synthesis of these ideas). The remainder

of this article explores the resonance thatsystemic triangulation may have on the metaphorof the iron triangle used in political science, andhow this metaphor might be helpful to exploresystemic failures generally, and systemic failurein (impact) evaluation practice more specifically.

Table 1 Expressions of the iron triangle compared

The iron triangle: three (original) Paris Peace Military–industrial complex Evaluation–industrialgeneric dimensions Treaty (1919) (1961) complex (2014)

i Interrelationships (on the ground) Bureaucracy Evaluand… including (contexts) professional soldiers managers/administrators

(militarism) implementing interventions*

ii Multiple perspectives Interests of military industry Interest groups Evaluators… with the task of(people) (materialism) representing (and inevitably)

privileging different interests

iii Boundary judgements Decisions of politicians Decisions of Congress Evaluations… as expressions (ideas/tools) (bourbonism)** of advice to decision-makers

(commissioners ofinterventions)

*Interventions can include any intended action on the real world, ranging from simple plans or projects, to widerprogrammes or policies, and from local to global levels. **Bourbonism depicts extreme political and social conservatism. It draws on subservience to support for the rule ofthe Bourbons, a European royal line that ruled in France from 1589–1793 and 1815–48, and in Spain (1700–1808;1813–1931) and Naples and Sicily (1734–1806; 1815–1860).1

Source Author’s own.

Figure 2 Evaluation-in-practice represented as possibly an evaluation–industrial complex: an influence diagramillustrating six activities – (a) auditing; (b) planning; (c) evaluating (understanding); (d) evaluating (practising);(e) commissioning; and (f) learning

Decision-making… focusing on sponsors andcommissioners of evaluations

3 Evaluations(Reflecting on boundary judgements)

2 Evaluators

(Engaging with multipleperspectives and value

judgements)

Advisory support… focusing onmediating different interests for

effective implementation

1 Evaluand

(Understandinginterrelationships and factual

judgements)

Implementation of decisions…focusing on project/programmemanagers and administrators


a

b

c

d

f

e


3 The iron triangle and an evaluation–industrialcomplexThe iron triangle metaphor was first expressedby Ralph Pulitzer, a political journalist reportingcritically on the Paris Peace Conference amongstvictorious allied governments following the FirstWorld War in 1919. Pulitzer warned against theinsipid confluence of interests amongst threeexclusive sets of actors – the military, industryand politicians making the decisions (Pulitzerand Grasty 1919). The iron triangle is helpful inspeaking to a wider sense of systemic failure; onepopular expression being the ‘military–industrialcomplex’ used by American President DwightEisenhower during his 1961 presidentialresignation speech. Political activists like Angela

Davis and Arundhati Roy have adapted ideas of‘industrial complex’ and ‘iron triangle’respectively to surface pernicious confluences ofinterest in more specific domains. Similarly,radical economists like J.K. Galbraith amongstother academics have used and developed theseideas in studies of political economy. Elsewhere,I have used the iron triangle metaphor to analysesystemic failure amongst academic economists inthe midst of the global economic crisis of 2008(Reynolds 2014).

The metaphor is generically used to describeinteraction between three entities: (i) someloosely defined ‘bureaucratic’ entity whichrepresents the site of real-world implementation


Table 2 Two contrasting views of evaluation-in-practice

Activities Evaluation–industrial complex Evaluation–adaptive complex

a Auditing – overview the site Situations of an evaluand are Situations of an evaluand are of implementing an systematically recognised at outset as systemically viewed as comprising all ofintervention (project, either tame (simple or complicated (i) complicatedness (interrelationships), programme, policy, plan…) with linear cause-and-effect attributes), (ii) complexity (multiple perspectives),

or wicked (complex with non-linear, and (iii) conflict (contrasting boundary unpredictable effects and impacts). judgements). Side-effects are inevitable.

b Planning – design of terms Planning is purposive – based on a Planning is purposeful – based on ‘agile’ of reference (ToR) for the perceived need for rather fixed goals measures of success that are not rigidly intervention and targets. Where need be, with defined by fixed quantitative indices

‘complex’ situations, a dominant alone, but also qualitative indices imperative is to ‘tame’ any ‘wicked-ness’ (stories, narratives, case studies, in the situations depending, for testimonies, etc.); both adaptable to example, on quantitative measures of changing situations in the evaluand and success. Little room for flexibility and hence being flexible, adaptive and adjusting measures over time. imaginative.

c Evaluating 1: understanding Evaluator unquestionably adopts Evaluator able to question ToR ofmeasures of success passed on through evaluand, including its own measures ToR of the evaluand. Regards evaluand of success. Regards evaluand as a as a fixed operational system learning system adapting to changing (ontological mechanistic reality). Adopts circumstances. Adopts radical disinterested (quasi) positivist epistemology constructivist epistemology involving self-involving as much as possible objective awareness of biases in shaping the valuation with prime attention to evaluation. Able to continually juggle avoiding bias. Focus more on efficiency between focus on efficacy, efficiency in relation to efficacy and effectiveness. and effectiveness (including ethical and

political notions of equity and sustainability).

d Evaluating 2: practice Practitioner role as measuring of value, Practitioner regarded as an agile either as specialist user of same tools – ‘bricoleur’, a craftsperson formatively ‘bestpracticitis’, or seeking an ever developing value (instrumental, intrinsicgrowing ‘toolbox’ adopting new tools and critical) through adapting tools as ‘best fit’ for purpose (‘horses-for- with new ideas during the course ofcourses’). an evaluation. /cont.

4 IDSB46.1 Reynolds.qxd 17/12/2014 09:13 6

of decisions (e.g. civil servants, managers andadministrators); (ii) interest groups/individualswho stand to benefit from the implementation ofdecisions (e.g. commercial and corporateinterests of various kinds, or commissionedadvisory groups whose task is to capture differentinterests); and (iii) decision-makers themselvesresponsible for making and justifying decisions(e.g. Congress or Parliament, or at a lower level,commissioners of interventions).

For the purpose of exploring the industry ofevaluation, the iron triangle metaphor might beused to describe interaction between threedomains: (i) some site of real-world impact of anintervention – project, programme or policy –with managers and administrators responsiblefor implementing the intervention;(ii) evaluators as an interest group active in theprocess of evaluating interventions – principallyand ideally capturing different values fromdifferent perspectives; and (iii) decision-makers– commissioners, foundations, donor agencies –ultimately responsible for interventions withimplicit and/or explicit models of decision-making to justify actions taken, including thecommissioning of evaluations. Table 1 maps out

the correspondence between differentexpressions of the iron triangle.

Figure 2 is a broad representation of the triadicrelationships between these three domains forwhat might be called ‘evaluation-in-practice’ andwhich may have particular expression throughan evaluation–industrial complex.

The influence diagram might be used as adiagnostic systemic tool for identifying flaws andopportunities associated with evaluation –whether impact evaluation or any otherformalised set of evaluation practices, and at anylevel of intervention – whether evaluating a policy,programme or project directly, or evaluating anevaluation as an intervention in itself.

The notes below give some guidance on how thearrows of influence may contribute towardssystemic failure of an evaluation–industrialcomplex. Although the notes attempt to focus onimpact evaluation, they may be deemed relevantto evaluation more generically, particularlyevaluation based on a contingency approach.Readers may find reference to Table 2 helpful forfurther understanding of activities (a) – (f).


Table 2 Two contrasting views of evaluation-in-practice (cont.)

Activities Evaluation–industrial complex Evaluation–adaptive complex

e Commissioning ToR for Rigour narrowly defined in terms of Rigour understood as three sets ofevaluators of interventions objective reliable and replicable use of co-guarantor attributes: (i) transparency requiring some guarantee evaluation tools as fixed commodities; of limitations in different contexts, or assurance of proven through application in case rather than some claim towards trustworthiness studies understood as being suitable achieving ‘objective truth’;

for either simple/complicated (tame) or (ii) complementarity or resonance – complex (wicked). Assurances relating to communicability with other disciplines/resonance with existing practices tend groups/cultures, etc.; and (iii) social and not to be given importance. Rigour ecological responsibility or relevance – associated with demonstrating internal coherence in dealing with allresponsibility is limited to providing inevitable issues of complicatedness, assurances of accountability to decision- complexity and conflict, and external makers rather than serving wider social coherence in addressing issues of social and ecological imperatives. and ecological responsibility.

f Learning… gained by decision- Enables single-loop and sometimes Enables single-loop, double-loop and makers from evaluations double-loop learning, but generally triple-loop learning from an evaluation

less reflective of power relations of an evaluand, including learning that circumscribing the primary intervention all evaluands have degrees ofand/or circumscribing the secondary complexity and conflict. Evaluation intervention in terms of evaluating the regarded as political, but also involving intervention. Evaluation regarded as humility in dealing with uncertainty.apolitical.



a Auditing situations A proposed interventioninvolves some initial auditing in making(boundary) judgements associated with thefactual reality of the intervention situation(the evaluand). A first audit based oncontingency thinking would determinewhether the situation (evaluand) is one wherethe interrelationships might be bounded assimple or complicated or more complex. Is thesituation akin to invoking a quick fix solution– like recipe reading in baking a cake? Or is itone requiring lots of logistical support – likesending a rocket to the moon? Or is it veryunpredictable and uncertain – like parentinga child? Many initial audits to guideinterventions are premised on a false pre-supposition that the problem situation can bedelineated at the outset between being tameor wicked… between being a mere difficultyor a mess… between being merelycomplicated or complex. The auditing ignoresthe idea that people – with differentperspectives – make all social situations notonly complex but often conflictual.

The key task for decision-makers responsiblefor initiating interventions2 is to make initialboundary judgements that appear to be ‘true’to the reality of the situation. The problemthough is in assuming that such boundaryjudgements based on some ‘truth’ of thesituation are able to be made from the outset.For example, a problem situation might bedefined in terms of the prevalence of measlesin children within a national population.Eradicating measles amongst the under-fivesmight be regarded as a relatively tameproblem – simple or possibly, complicated, butnot necessarily ‘complex’. However, importantcaveats might be made in regarding this as a‘tame’ rather than a ‘wicked’ problem. It mayonly be regarded as tame if important ‘inputs’or conditions are given – including forexample, availability of effective medications,trained staff and management protocols, etc.Clearly, the risk is in auditing ‘situations’ asthough they are ‘given’ systems – viewingmessy situations as though they are justimplicitly well-defined problems – confusingthe territory for the map. There is an errorhere in factual judgement which regards thereal world as ‘simple’ or ‘complicated’ (i.e.tame) without appreciating the human factorthat makes all social situations complex and

potentially conflictual (i.e. wicked). It ishuman agents – people – that translate‘wicked’ situations into ‘tame’ situations –humans that translate messy situations intosystems. The eradication of measles amongstunder-fives is as complex as preventing thespread of Ebola; it is a situation in whichwider factual judgements of context regardingculture, history, geography and politics clearlyhave a significant role in shifting the problemfrom being ‘tame’ to being more ‘wicked’.

Might it be more useful then to assumecomplexity from the outset, given thatcomplexity is determined not just bycomplicatedness of there being manyinterrelated variables, but by the humanfactor of there being multiple perspectives onthe situation? Moreover, the incidence ofmultiple perspectives prompts alertness tothere being actual and potential conflicts –conflicts in contrasting boundary judgements.A more useful principle might be to start anyaudit more systemically, taking account of notjust interrelated variables in the situation, butdifferent perspectives and potential conflictsamongst key stakeholders.

b Planning (terms of reference) interventions A typicalsequence of tasks in planning an interventioninvolves firstly establishing an overall purpose,then constructing or designing reasonableobjectives, and then defining some measuresof success by which the intervention mightthen be evaluated. Defining the purposerepresents the prime boundary of the systemof intervention being initially designed. Twoquestions arise. First, at what level of planningare the purposes or objectives located – policydesign, programme/project management, or atthe level of operational administration?Second, how much flexibility is there in thesystem – do the purposes/objectives/measuresremain fixed or are they open to change? Forexample, regarding the planning involved withconfronting an issue of disease prevention, is itjust a problem of ‘administering’ vaccinationsat an operational level of planning, or might itbe a set of objectives at a higher level ofplanning, possibly involving new managementstructures or even a change in national healthpolicy? However initially perceived, it is helpfulfor evaluators of an intervention to regardthese objectives in a wider contextual field as



nested purposes or objectives; that is,circumscribed by a supra-level system ofpurpose or worldview, and drivers of impliedsub-level systems of more tactical ‘targets’.Regarding purposes in this way enablesevaluators to see the intervention as beingpurposeful rather than purposive (see alsoHummelbrunner in this IDS Bulletin). Apurposive system is one where the purposesremain rigid and unfluctuating, morecommonly associated with mechanicallydesigned systems like washing machines andwatches. Human systems are, or ought to be,purposeful in the sense that purposes are fluidand subject to change given inevitable changein circumstances (Ackoff and Emery 1972). Akey problem shared amongst many evaluatorsis that they see little room for manoeuvre inadvising against proffered objectives of anintervention (see also Williams in this IDSBulletin).

‘Measures of success’ against which to monitorand evaluate ‘impact’ of intervention objectivesare themselves boundary judgements. Ideallysuch measures ought therefore not to be fixed.A common critique of the most prevalentplanning tool in international development forsurfacing such measures – logical frameworkanalysis (LFA) – is the assumed inflexibilityand rigidity. The LFA approach tracks apathway from resources inputs to activities,outputs, outcomes and impacts in a sequentialunilateral manner helpful at the operational/administrative level of planning, where clearmeasures in terms of performance indicatorscan be identified. But it has less value forcomplex situations. When applied to suchsituations, the messy realities can becomedangerously entrenched as being simple andsubject to linear cause-and-effect explanations.

c Evaluating 1: Understanding (interrelationships) Twoaspects of evaluation might usefully beconceptually demarcated – firstly, understandingthe context of the evaluand, and secondly,engaging in practice with the various agenciesof the evaluand. In real-world praxis the twoare, of course, intricately interwoven; as withcorresponding dualities of theory/practice,thinking/action, as well as the thinking inpractice of summative/formative evaluation.Abstracting out understanding and practicecan help reveal the more direct interwoven

sources of influence on evaluators’ praxis.Item (d) will explore further the influences onevaluation practice – a more formativeendeavour of developing value – whilst item(c) here will focus on how understandings ofvalue in the context are influenced (seeWilliams, this IDS Bulletin).

Evaluators are essentially tasked withtranslating factual judgements relating to thereal-world complicatedness, complexity andconflict, into appropriate value judgements.Measures of success – suggested by theintervention’s terms of reference (ToR) – mightbe translated in terms of efficacy and efficiency(has the intervention worked according to itsmandate with good use of resources?) andeffectiveness (what wider effects may havebeen generated by the intervention?). AsHummelbrunner points out (this IDS Bulletin),a third concern for evaluators might also be tounderstand wider critical and political values ofequity and sustainability in relation to theintervention. The task then is to capture theserealities of an intervention in terms ofappropriate values – instrumental, intrinsicand critical/political. Practitioners in theevaluation industry may like to consider theextent to which their evaluations are biasedtowards the instrumental values passed downthrough the ToR.

Evaluation is often regarded normatively asbeing a bias-free pursuit of ‘truth’. The tasktraditionally is to capture the realities ofinterrelationships in as an objective (andhence ironically, value-free) way as possibleusing scientific tools and procedures that canreduce bias. The imperative of objectivity andindependence appears to remain very strongin the evaluation community. For example,Mowles speculates that in using thecontingency approach evaluators ‘continue tocleave to what John Dewey [1929]… referredto as a “spectator theory of knowledge”, whichassumes separation between the observer andthe thing being observed’ (Mowles 2014: 164).Understanding the evaluators’ own valuejudgements in the role of evaluating anintervention requires making an epistemologicalshift. It requires moving from a conventionalview regarding interventions as ontologicalrealities ‘out there’ subject to evaluation, to aview of interventions as conceptual constructs



(helpfully regarded as systems of interest) andan evaluator as a co-creator of such constructsas part of her/his engagement with the reality.3

In sum, ritualised (mis)understandings inevaluation may involve not only: (i) thepiecemeal sectioning of perceived complexsituations from complicated or ‘simple’situations at the outset (that is, failing torecognise that all situations involving humanactivity are complex); but (ii) a possiblemarginalisation of addressing issues of conflictand equity (again, failing to recognise this asan aspect – to varying degrees – of all humanactivity). In addition, the misunderstandingmay involve: (iii) confusing interventions (aswell as evaluations of interventions) asontological realities as against constructivistendeavours; endeavours that speak more tothe practice of formative evaluation.

d Evaluating 2: Practice (engaging with multipleperspectives) The formative side of evaluation isrelated to the actual practice with using toolsin an evaluand context and thus impacting on,and generating value with, the stakeholdersassociated with the evaluand.

In Aid on the Edge of Chaos (Ramalingam 2013)a typology of four sets of challenges is offeredto those working with complex situations –systemic, behavioural, relational and dynamic.Ramalingam then identifies and draws onfour designated families of ‘complex systemsapproaches’ which can ‘fit’ the designatedchallenges respectively; ‘systems approaches,behavioural approaches, network methods anddynamic analysis techniques’. This follows along tradition in the field of managementconsultancy; working with typologies ofsituations and then designating methodologiesin alignment with the situation-type(cf. Ghoshal and Nohria 1993). In effect theprocess involves not just ‘taming’ situations(e.g. simple, complicated, complex), buttaming methods and tools to fit the context aswell. In the systems tradition, a similarprocess gained considerable currency withTotal Systems Intervention (TSI) (Jackson andFlood 1991) in allocating particular systemsapproaches to particular situations. Themapping was done using the Burrell andMorgan (1979) typology of four social scienceparadigms – objectivist ‘functionalism’ and

‘radical structuralism’, and subjectivist‘interpretivism’ and ‘radical humanism’. Sosystem dynamics, for example, was assignedto the paradigm of functionalism, whereas softsystems methodology was assigned to theparadigm of interpretivism.

The similarity with the four groupstypologised in Aid on the Edge of Chaos isperhaps significant, albeit each of the fourapproaches suggested by Ramalingam relateto large families of methods, rather than anysingle best practice. Whilst the latter idea isnicely critiqued within internationaldevelopment in terms of ‘an epidemic ofbestpracticitis’ (Ramalingam 2013), thereremains a pervasive notion of ‘best fit’ –expressed also in the working title of a draftOverseas Development Institute (ODI) paper– ‘From Best Practice to Best Fit’(Ramalingam, Laric and Primrose 2014).

The problem here remains one signalled bycritiques of TSI (cf. Ulrich 2003; Reynolds andHolwell 2010; Reynolds 2011). Systemsapproaches – or indeed any tools, methods,approaches – do not have a pre-determinedfixed use-value, outside of the context of useand the individual(s) doing the valuing oracting as users. To assign such ‘objective’value of tools decoupled from the valuerinvolves reifying the tools; fetishising themethods. To coin a term used by one popularcynic of command and control approachesused in the public sector, might there beslippage within the evaluation industrytowards professionals becoming ‘toolheads’?(Seddon 2003). Despite warnings in the draftODI paper of succumbing to a ‘toolkittemptation’ which may lead to unhelpful‘complexity silver bullets’, there remains arisk in getting fixated on and commoditisingcomplex systems approaches in a ritualisedpractice of promoting an ever bludgeoningchest of evaluation tools.

There is something here that speaks to aquestion from Duncan Green ‘… is the wholepoint of complex systems that you can’t havestandard approaches, only connected, agilepeople able to respond and improvise?’ (Green2014). These ideas of agility and better practiceamongst evaluators is perhaps best captured bya metaphor from the French anthropologist



Levi-Stauss in the 1960s; the art of a bricoleur.As Patton explains, this is a travellingnineteenth-century rural craftsman in Franceskilled in using and adapting tools at hand, butwith, as well, a sense of inventiveness andcreativity for purposeful intervention – the artof bricolage (Patton 2010: 264–304).4

e Commissioning (terms of reference) evaluationsSponsors not unreasonably require someguarantee of rigour from evaluators beingcommissioned. The terms of reference incommissioning evaluations signal the need forsome degree of assurance and trustworthiness.Evidence-based, results-orientated evaluationstypically signal guarantors of objectivity –some sense of reliability based on replicabilityand bias-free recommendations. Thepopularity of, and investment in,experimental approaches such as RCTs wheresuch assurances are paramount is testamentto the power of such guarantors in evaluation.An underlying critique of RCTs from acontingency approach perspective is that theyare inappropriately applied to complexsituations where the variables generating theeffects are often unknown and changeable,and hence difficult to ‘control’. Hence anyresults generated may not always be reliable.

So what guarantors if any might be offered incomplex situations? Glenda Eoyang, thefounding Executive Director of the HumanSystems Dynamics Institute and a long-timepractitioner in the use of complex adaptivesystems for evaluation, commented: ‘The onlything you can be certain of is uncertainty, butyou can’t even be sure how much of that youhave…’. Glenda went on to comment: ‘On theother hand, there’s no escape into constructivstangst and unknowing because there is a world,and we act, and it changes, so there is somegrounding for meaningful conceptual modelsand options for action’ (pers. comm. 2014).Ultimately, of course, there can be no absoluteguarantor of certainty with interventions insituations that are inherently uncertain, butthere remains a need for commissioners ofevaluations to require some sense oftrustworthiness.

Practicing evaluators may like to reflect on aneed for two sets of co-guarantors in additionto traditional scientific (disciplinary) based

criterion of ‘objectivity’, reliability andreplicability. Firstly, there are (moreinterdisciplinary) co-guarantors of‘complementarity’ – a communicativecompetence in facilitating meaningfulconversation and exchanges between andamongst stakeholders in the context of anevaluation (process). The guarantor here isone of generating ‘resonance’ with others.These stakeholders might involve otherrelevant sources of expertise from differentdisciplines, but critically also including the layexpertise of those directly involved in, andaffected by, the intervention being evaluated.A trap in ritualised practice of supportingresonance is in proposing fail-safe techniquesthat have been ‘tried and tested’ in somecontexts and recommended for perceivedsimilar contexts. Unfortunately, the space toexperiment – a culture of safe-fail (Stravinsky2000) – is perhaps an underrated guarantorsparingly afforded to evaluators.

Secondly, there are (more transdisciplinary)co-guarantors of what might be called ‘socialand ecological responsibility’ (Reynolds 2008)– a reflective competence in ‘relevance’; notonly in addressing the internal coherence(interrelationships, multiple perspectives andboundary judgements) of the intervention indealing with complicatedness, complexity andconflict, but also the external legitimacy of theintervention as a whole, in terms of its wideraccountability to social and ecological well-being (equity and sustainability).

‘False’ co-guarantors or assurances mayinvolve either incompetence or incapability inthe use of any one set, or the unreflectiveprivileging of one set of co-guarantors overanother (Reynolds 2001).

f Learning through evaluating For commissioners,the immediate need from evaluations is tolearn whether and how any particularintervention is working (measures of efficacyand efficiency); learning not only on whetherthe intervention is serving its purpose butwhether the purpose being served is the mostappropriate purpose (measure of effectiveness).The learning provided here is in part single-loop – learning to do the thing right – but alsodouble-loop – learning to do the right thing –(see Hummelbrunner, this IDS Bulletin).



Evaluators from a tradition using thecontingency approach are prone also to advisecommissioners in learning not to see allsituations as being tame but rather to view somesituations as complex or wicked and requiringdifferent approaches. Developmental evaluation(Patton 2010) and Aid on the Edge of Chaos(Ramalingam 2013), as well as contributors tothe Evaluating the Complex compilation (Forss etal. 2011), are well endowed with many goodcase studies illustrating the need for double-loop learning in complex situations.

However, choosing case studies that from theoutset are clearly complex involving manyviewpoints and conflicts may lend a false senseof security to commissioners that otherevaluands (sponsored interventions) areactually ‘simple’ or at most merely‘complicated’. Perhaps a more significantlearning point might come from drawing oncase studies that may superficially appear tobe merely simple or complicated but actuallyreveal complexity and conflict in the course ofan evaluation. Commissioners might thenlearn that all evaluands are actually complexto varying degrees. This relates to a possiblemissing third loop of learning regardingpower relations (see again Hummelbrunner,this IDS Bulletin).

Triple-loop learning questions how ‘rightness’is circumscribed by ‘mightiness’ and vice versa(ibid.). Evaluators may like to consider howthe contingency approach has a powerful holdon practice. Arguably, it is an example ofrelatively unquestioned sense (of ‘rightness’)amongst evaluators dominating (themightiness of) decision-making amongstcommissioners. The contingency approachprovides a powerful argument for havingmultiple methods, each with ascribed fixedfunctions and values. It prompts the ritualisedpractice of having an ever increasing‘armoury’ of methods. It prompts decision-makers (commissioners of interventions, forexample) to support an existing toolkit eitherwith an ‘overhaul’ of new tools, or expandedincrementally with a ‘broader menu of tools’.

4 Discussion: breaking the triangle and abstractinga more purposeful system of evaluationWhilst there is much to be commended in allthree publications using complexity ideas for

evaluating interventions (Patton 2010; Forss et al.2011; and Ramalingam 2013), there remainssome doubt as to whether one unholy trinity –one iron triangle – might inadvertently besubstituted by another. The first, being implicitlycritiqued by all three publications, involves adynamic depicting: (i) situations regarded ascomprising factors related only by linearcausality (a judgement of ‘fact’); with (ii) asingle viewpoint (value judgement) dominatedby a mechanistic perspective of the reality (oftencaricatured by LFA); and (iii) circumscribed byan associated mindset of command and control(boundary judgement).

The alternative contingency approach profferedby the authors of the above-mentionedpublications potentially involves another unholytrinity depicting: (i) situations from the outset asbeing (‘factually’) either simple or complicated(tame), or complex (wicked), and so withmeasures of success being ‘fixed’ according totheir initial typological demarcation; with(ii) multiple viewpoints each aligned (in terms of‘value’) with a particular tool, method,methodology or approach; and (iii) associatedwith a mindset of deceptively benign holism incovering all situation ‘types’, and/or benignpluralism (inclusive of ever increasing tools andmethods).

A more purposeful system might have analternative triadic confluence of the six factors:(a) auditing based on seeing all situations ascomprising a mixture of complicated(interrelationships), complex (multipleperspectives), and conflictual (judgements ofboundary); (b) planning based on purposefulsystems design where measures of success areiterative – flexible and open to change;(c) evaluating (understanding) where values ofutility are complemented with intrinsic personalvalues and political critical values; (d) evaluating(practice) where values are regarded as not fixedbut developmental involving the evaluator andother stakeholders in a concerted effort ofdeveloping value in existing practices;(e) commissioning based on a wider set ofcriteria of rigour involving not just reliability andreplicability of tools, but the resonance andrelevance of tools used in an appropriate criticalspace of experimentation; and (f) learning basednot just on received wisdom but social learninginvolving all stakeholders in a continual cycling



of single-, double- and triple-loop learning –inviting not just technical, but ethical as well aspolitical insight.

There is evidence of shifts towards a morepurposeful evaluation thinking in practice. Withrespect to (b) ‘planning’, the draft ODI paper(Ramalingam et al. 2014) signals the importanceof shifting from ‘fixed’ measures of success tomeasures that are open to revision and change –‘looking for multiple intervention points andmoving between them dynamically… workingthrough group interaction and iteration ratherthan ‘back office’ designs… [and] concentratingon flexibility rather than predictability ofsolutions’ (p.11). A key challenge is how theseprinciples might be translated into actualmeasures to inform a more purposefulintervention. How might they challenge theritualised practice of fixing measures of success(for intervention managers and evaluators),irrespective of the inevitable changes occurringin situations of the evaluand? Similarly, withrespect to (d) ‘evaluating practice’, three other‘principles’ from the draft ODI paper illustrate wellthe type of practice required: ‘… accommodatingmultiple alternative perspectives rather thanspecific best practices… generating ownership ofproblem formulation and transparency throughparticipation of stakeholders… [and] developingsystematic visual representations of the problemspace that enable group-based exploration of thesolution space’ (ibid.). Also, there is considerablevariation amongst practitioners using thecontingency approach which does not neatly fit thecaricature described above. Many practitioners,for example, use a wide set of co-guarantorattributes – item (e) – over and above assurancesof reliable tools. Many also focus on learning –item (f) – in dealing very much with issues ofpower relations associated with impact evaluations.

A significant part of the Mowles (2014) paper isgiven to celebrating examples where practitionersin the evaluation field are able to adapt ideas andmethods using principles of systems thinking andcomplexity to their own practice. Amongstpractitioners cited by Mowles in this endeavourare Allen (1988), Callaghan (2008), Sanderson(2009), Westhorp (2012) and the morecontemporary works of Stacey (2012). Otherpractitioner evaluators whose works are cited byMowles in exemplifying the contingency approachare contributing authors to these two issues of the

IDS Bulletin (e.g. Hummelbrunner, Rogers,Williams), but their actual practice is much morecreative, versatile and adaptive in the use ofcomplexity and systems tools than perhaps allowedfor in the main focus of critique. Hummelbrunner(2010), for example, is also cited by Mowles in thesame paper as a demonstration of how even asupposedly rigid tool-set such as the Logframe isable to be cultivated for better practice using, inhis case, ideas from soft systems thinking. One ofthe most promising developments of using ideasof complex adaptive systems without presuming acontingency approach are ideas of ‘adaptiveaction’ (Eoyang and Holladay 2013) based onearlier innovative use of CAS for organisationaldevelopment (Eoyang and Berkas 1998). Adaptiveaction resonates significantly with the wider use ofsystems ideas in our own approach of systemsthinking in practice.

Table 2 illustrates these more progressivefeatures in terms of what we might call anevaluation–adaptive complex, in comparisonwith an evaluation–industrial complex.

5 ConclusionThe identification of three core ideas or conceptsassociated with systems thinking (cf. Williams2013 and Williams, this IDS Bulletin) –interrelationships, perspectives and boundaries –marks a significant shift in abstracting outessential features in order to make systemsthinking more resonant with other practices.The three domains correspond to the threeissues dominating international development inthe last decade (see introduction to IDS Bulletin45.6, 2014). First, there is a growing concernregarding impact evaluation from the policyworld; a concern for more evidence-based,results-orientated or outcomes-orientatedevaluations, raising issues of rigour in evaluation.Second, there is a concern amongst evaluatorswith the complexity of interventions driven bymultiple agencies and multiple perspectives;concerns that have prompted attention toparticipatory and multi-method approaches.Third, there is a concern amongst commissionersand evaluators regarding professionalresponsibility and independence amongstevaluators; a concern leading to an associatedneed to justify particular approaches.

From a systems thinking in practice perspective,a more general source of anxiety is that such



concerns may themselves become systemicallystabilised through ritualised discourse andpractice dominated currently in the evaluationfield by an omnipresent contingency approach.

The approach offers an alternative degree ofrigour towards evaluation, enabling evaluators toadopt particular tools or tool-sets and ideas forparticular situations. But the approach arguablyhas a risk towards stifling the evaluation processby fetishising methods and tools and distractingattention away from craft skills amongstevaluation practitioners and the realpolitik ofany evaluation process.

The systems tool used to make this analysis – thesystemic triangulator – is one based on ideas ofsystemic triangulation. As with any tool or modelor system – it is an abstraction from reality, andas such is neither right nor wrong in terms ofbeing factually correct. For example, the three

triangular indices of the evaluation–industrialcomplex (Figure 2) depicting roles of evaluandmanagers, evaluators and commissioners, may insome contexts actually be embodied in oneperson. The value of any model ought rather tobe measured primarily in terms of its usefulnessand adaptability for inquiry in different contextsby different users.

In particular, practising evaluators may wish toreflect and develop value on the use of systemictriangulation for diagnosing possible instances ofsystemic failure and the role of evaluation insuch failure. The article provides a tentativeoutline model for an evaluation–industrialcomplex to help with such a diagnosis. To whatextent might this be helpful in diagnosingexisting evaluation practices, but also to whatextent might it help through adaptation incultivating value towards a more purposefulmodel of an evaluation–adaptive complex?


Notes* In addition to the helpful feedback from

reviewers on an earlier draft, particularappreciation is extended to Barbara Befani,one of the co-editors of these two specialissues of the IDS Bulletin. Barbara’s insightfulsuggestions and prompts have significantlyenriched the final article. The author remainssolely responsible for the final content andany errors.

1 See www.collinsdictionary.com/dictionary/english/bourbonism#examples_box.

2 That is, the primary intervention – project orprogramme or policy – of the evaluand (a) and

(b), rather than the secondary intervention ofan evaluation (of the primary intervention)represented by (c) through to (f).

3 Callaghan (2008) and Sanderson (2000, 2009)are cited by Mowles (2014: 164) as examplesof more reflective thinking of evaluators’ rolesusing systems and complexity thinking.

4 Notwithstanding a chapter given to thisprogressive idea of an evaluator, Patton’s bookfocuses on evaluators as being able toaccommodate ever more tools and methodsfrom the tradition of complexity rather thanbeing adaptive and inventive with theirexisting tool-sets.

ReferencesAckoff, R. (1974) Redesigning the Future: A Systems

Approach to Societal Problems, New York NY: JohnWiley & Sons

Ackoff, R.L. and Emery, F.E. (1972) On PurposefulSystems, London: Tavistock

Burrell, G. and Morgan, G. (1979) SociologicalParadigms and Organisational Analysis, London:Heinemann

Eoyang, G.H. and Berkas, T. (1998) ‘Evaluationin a Complex Adaptive System’, in M. Lissackand H. Gunz (eds), Managing Complexity inOrganizations, Westport CT: Quorum Books

Eoyang, G.H. and Holladay, R. (2013) AdaptiveAction: Leveraging Uncertainty in YourOrganization, Redwood City CA: StanfordUniversity Press

Flood, R.L. and Carson, E.R. (1993) Dealing withComplexity: An Introduction to the Theory andApplication of Systems Science, New York NY:Plenum

Forss, K.; Marra, M. and Schwartz, R. (eds)(2011) Evaluating the Complex: Attribution,Contribution and Beyond, Comparative PolicyEvaluation 18, New Brunswick: TransactionPublishers

Ghoshal, S. and Nohria, N. (1993) ‘Horses forCourses: Organizational Forms forMultinational Corporations’, Sloan ManagementReview 34: 23–35

Glouberman, S. and Zimmerman, B. (2002)Complicated and Complex Systems: What WouldSuccessful Reform of Medicare Look Like?,Commission on the Future of Health Care in


Canada Discussion Paper 8,www.investisseurautonome.info/PDF-Downloads/COMMENT-INVESTIR-RENDEMENT-INDEX/doc.1590-%20Zimmerman%20Glouberman%202002%20Health_Care_Commission_DP8.pdf(accessed 20 October 2014)

Green, D. (2014) ‘Can Complex SystemsThinking Provide Useful Tools for AidWorkers? Draft Paper on Some DFID PilotExperiments, for your Comments’, From Povertyto Power Blog, 7 May, http://oxfamblogs.org/fp2p/can-complex-systems-thinking-provide-useful-tools-for-aid-workers-draft-paper-on-some-dfid-pilot-experiments-for-your-comments/ (accessed 10 June 2014)

Hummelbrunner, R. (2011) ‘Systems Thinkingand Evaluation’, Evaluation 17.4: 395–403

Hummelbrunner, R. (2010) ‘Beyond Logframe:Critique, Variations and Alternatives’, inN. Fujita (ed.), Beyond Logframe: Using SystemsConcepts in Evaluation, Issues and Prospects forEvaluation of International Development,Series 4, Tokyo: Ministry of Foreign Affairs ofJapan

Hummelbrunner, R. and Reynolds, M. (2013)‘Systems Thinking, Learning and Values inEvaluation’, Evaluation Connections: Newsletter ofthe European Evaluation Society, June: 9–10

Jackson, M.C. and Flood, R.L. (1991) CreativeProblem Solving: Total Systems Intervention,Chichester: John Wiley & Sons

Korzybski, A. (1995) Science and Sanity: A Non-Aristotelian System and its Necessity for Rigour inMathematics and Physics, 5th ed. (1st ed. 1933),Englewood NJ: International Non-AristotelianLibrary

Lachica, R. (2013) ‘Systems Thinking UsageSurvey 2013 Results’, Fuzzzy Blog, 13 March,http://fuzzzyblog.blogspot.no/2013/03/systems-thinking-usage-survey-2013.html (accessed7 June 2014)

Levin, K.; Cashore, B.; Bernstein, S. and Auld, G.(2012) ‘Overcoming the Tragedy of SuperWicked Problems: Constraining Our FutureSelves to Ameliorate Global Climate Change’,Policy Sciences 45.2: 123–52

Mowles, C. (2014) ‘Complex, but Not QuiteComplex Enough: The Turn to theComplexity Sciences in EvaluationScholarship’, Evaluation 20.2: 160–75

Patton, M.Q. (2010) Developmental Evaluation:Applying Complexity Concepts to Enhance Innovationand Use, New York NY: Guilford Press

Pulitzer, R. and Grasty, C.H. (1919) ‘Forces atWar in Peace Conclave’, The New York Times,18 January

Ramalingam, B. (2013) Aid on the Edge of Chaos,Oxford: Oxford University Press

Ramalingam, B.; Laric, M. and Primrose, J.(2014) ‘From Best Practice to Best Fit:Understanding and Navigating WickedProblems in International Development’, ODIWorking Paper May 2014, presented in draftform for comments and feedback,http://oxfamblogs.org/fp2p/wp-content/uploads/2014/05/BestPracticetoBestFitWorkingPaper_DraftforComments_May2014.pdf (accessed9 June 2014)

Reynolds, M. (2014) ‘Systemic Failure inMacroeconomic Modelling’, InternationalJournal of Applied Systemic Studies 5.4

Reynolds, M. (2013) ‘Managing Systemic RiskUsing Systems Thinking in Practice’,AWERProcedia Advances in Applied Sciences 1:217–24

Reynolds, M. (2011) ‘Critical Thinking andSystems Thinking’, in C.P. Horvath and J.M.Forte (eds), Critical Thinking, New York NY:Nova Science Publishers

Reynolds, M. (2008) ‘Reframing Expert Supportfor Development Management’, Journal ofInternational Development 20.6: 768–82

Reynolds, M. (2001) ‘Co-guarantor Attributes:A Systemic Approach to Evaluating ExpertSupport’, 8th European Conference onInformation Technology Evaluation, OrielCollege, Oxford, 17–18 September

Reynolds, M. and Holwell, S. (2010) ‘IntroducingSystems Approaches’, in M. Reynolds and S.Holwell (eds), Systems Approaches to ManagingChange, London: Springer

Rittel, H.W. and Webber, M.M. (1973) ‘PlanningProblems are Wicked’, Polity 4: 155–69

Seddon, J. (2003) Freedom from Command andControl, Buckingham: Vanguard Education

Stacey, R.D. (2001) Complex Responsive Processes inOrganizations: Learning and Knowledge Creation,London and New York NY: Routledge

Stacey, R.D. (1992) Managing the Unknowable:Strategic Boundaries between Order and Chaos inOrganizations, New York NY: John Wiley & Sons

Stravinsky, I. (2000) ‘Writing the Wrongs:Developing a Safe-fail Culture inConservation’, Conservation Biology 14.6:1567–68

Ulrich, W. (2003) ‘Beyond Methodology Choice:Critical Systems Thinking as Critically



Systemic Discourse’, Journal of the OperationalResearch Society 54.4: 325–42

Ulrich, W. (2000) ‘Reflective Practice in the CivilSociety: The Contribution of CriticallySystemic Thinking’, Reflective Practice 1.2:247–68

Ulrich, W. (1998) Systems Thinking as if PeopleMattered: Critical Systems Thinking for Citizens andManagers, Lincoln School of Management,University of Lincolnshire and HumbersideWorking Paper 23, Lincoln: University ofLincolnshire and Humberside

Ulrich, W. (1983) Critical Heuristics of SocialPlanning: A New Approach to Practical Philosophy,Chichester: Haupt, Bern and Stuttgart,Chichester: John Wiley & Sons

Williams, B. (2013) ‘Systemic Thinking inEvaluation: Three Core Concepts’, EvaluationConnections: Newsletter of the European EvaluationSociety, June: 7–8

Winckler Andersen, O.W. (2014) ‘SomeThoughts on Development EvaluationProcesses’, IDS Bulletin 45.6: 77–85



Open Research Onlineoro.open.ac.uk/41879/1/__userdata_documents6_mdr66...regarding the take-up of...

Documents

Transcript of Open Research Onlineoro.open.ac.uk/41879/1/__userdata_documents6_mdr66...regarding the take-up of...