Working Paper Towards Deliverable 6.3 Final Try

1

WORKING PAPER TOWARDS THE EVALUATION

APPROACH OF TCBL

December 2015

Joe Cullen, Kerstin Junge, Giorgia Iacopini

Co#funded)by) Horizon)2020

D 8.2: TCBL Website

646133 - TCBL Textile & Clothing Business Labs

2

TABLE OF CONTENTS

TABLE OF CONTENTS ............................................................................................................ 2)

1.) INTRODUCTION ................................................................................................................ 3)

2.) ABOUT TCBL .................................................................................................................... 3)

2.1) BACKGROUND AND CONTEXT: THE EUROPEAN TEXTILES AND CLOTHING SECTOR ............. 3)

2.2 KEY AIMS AND ACTIVITIES OF TCBL ..................................................................................... 5)

2.2) KEY FEATURES OF TCBL .............................................................................................. 7)

3.) THE APPROACH TO EVALUATING TCBL ..................................................................... 8)

3.2) THE PURPOSES OF THE TCBL EVALUATION ................................................................... 9)

3.2 REFINING THE EVALUATION CONCEPT ................................................................................ 13)

3.2.1 An Ecosystems perspective on Evaluation ............................................................. 13)

3.2.1 Evaluating Complexity ............................................................................................. 19)

3.2.1 Large Scale Change ............................................................................................... 23)

3.3 SUMMARY AND IMPLICATIONS ............................................................................................ 26)

4. THE EVALUATION DESIGN ............................................................................................... 28)

4.1 EVALUATION AUDIENCE AND QUESTIONS ............................................................................ 28)

4.1.1 Evaluation Audiences and their information needs ................................................. 28)

4.1.2 Evaluation Questions .............................................................................................. 31)

4.2 EVALUATION METHODOLOGY ............................................................................................. 37)

4.3 EVALUATION METHODS ...................................................................................................... 41)

5. TOWARDS A BASELINE TCBL THEORY OF CHANGE ................................................... 43)

6. NEXT STEPS: IMPLEMENTING THE EVALUATION ........................................................ 49)

BIBLIOGRAPHY ...................................................................................................................... 50)

3

1. INTRODUCTION

This working paper presents the core ideas for the evaluation of TCBL as a key milestone towards producing D6.3 - Evaluation Approach - in Month 9. The Paper is based on and integrates the following information sources:

information on TCBL from the DOA, the TCBL website (in particular the working group discussions),

the results of two kick-off meetings, a literature review carried out by the evaluation team on key relevant concepts the first round of project partner interviews held in November 2015.

In doing so, it elaborates on the early thinking on the evaluation design presented to partners in the Prato and Brussels meetings.

The document has three main purposes:

To bring together the results from the evaluation scoping work carried out thus far; To develop a conceptual framework for the evaluation design for TCBL as a concrete

step towards the production of D 6.3; To share these ideas with consortium partners and invite feedback that can be used to

a) further advance the evaluation approach, and b) to use some of the content of this paper as a basis for a collaborative exercise on evaluation during the January 2016 partner meeting.

This document is therefore structured as follows:

Chapter 2 sets the scene for the evaluation design. It presents the background and context to the TCBL project as well as a high level overview of its key aims and activities. From this, it extracts the key features of TCBL as relevant for the evaluation task.

Chapter 3 then takes this information to present an outline of the evaluation approach, focusing on the conceptual foundations of the evaluation as well as evaluation purposes.

Chapter 4 outlines the audiences for the evaluation as well as the evaluation questions, and the core data collection methods that will be used.

Chapter 5 presents a baseline theory of change for TCBL, both in diagram and narrative format. This will be further elaborated with partners during the January 2016 coordination meeting.

The concluding Chapter 6 outlines the activities that will be carried out to produce the first evaluation deliverable and the partner input required.

2. ABOUT TCBL

2.1 BACKGROUND AND CONTEXT: THE EUROPEAN TEXTILES AND CLOTHING SECTOR

In line with many established sectors, the Textile & Clothing industry (T&C) reflects a long declining trend in employment, export and production capacity, driven by globalisation and


4

recession. However, it still plays a crucial role for the EUs economy and social well-being. In 2010, the 127,000 active T&C companies in the EU had an overall turnover of 172 billion Euros, employed over 1.9 million workers and represented investments of around 5 billion Euros.1 A key characteristics of the sector is that is shows a negative trade balance, with 34 billion Euros of textile and clothing products exported and 84 billion Euros imported from Third markets in 2010. But if the landscape is broadened to include the entire fashion industry, and high-end products and the non-clothing applications of textiles (where Europe is a worldwide leader), employment goes up to over 5 million persons, which is equivalent to 3.7% of the total non-financial business economy (in 2009), according to Eurostat.

Over the past two decades, the T&C industry has developed three main strategies to meet the global competitive pressure:

A retention strategy for price-sensitive mass-market customers, relying on a cost-oriented approach, which used relocation to low-income countries, including the New Member States, as its major instrument;

A product/service-oriented approach, diversifying the spectrum towards high-quality and specialty textile products;

A productivity-oriented approach, based on automation and IT-based supply chain man-agement, helping improve flexibility and create global sourcing systems.

In recent years, the T&C sector has increasingly looked towards applying research to stimulate innovation, including the setting up of a Technology Platform (http://www.textile-platform.eu/) to meet the requirements of FP7. The T&C sector is also highlighted in the EC Communication For a New Industrial Renaissance of January 2014, which calls for a movement towards innovative, high-added value products; taking full benefit from market liberalisation, notably via Free Trade Agreements; addressing delocalisation by placing more emphasis on creative and high-end products manufactured in the EU; protecting European fashion and high-end companies against counterfeiting; reducing the gap between longer financial cycles and production cycles; retaining traditional skills and know-how; improving consumer trust in online shopping.

The above elements are reflected in an Action Plan for the fashion and high-end industries presented by Commissioner Tajani on 3 December 2013 in London. Although the Action Plan is intended to enhance the competitiveness of the T&C sector, the TCBL approach argues that the Plan will only be implemented effectively if it is supported by business innovation in order to effectively reverse the system of incentives that has supported industrys restructuring pro-cesses over the past two decades. This innovation particularly needs to focus on three areas: taking advantage of the emerging opportunities from the new Making Economy (e.g. person-al robotics, home production, etc.); redirecting the capacities of old artisans and family work-ers or fasonists and re-connecting their knowledge with e.g. new, young, creative people; taking full advantage of the benefits of Future Internet technologies for the T&C global supply chain (diffused ecommerce networks, IoT tracking systems, virtual warehouses, customer en-gagement, etc.) in the light of a new customer-driven approach based on market intelligence.

1 The European Apparel and Textile Confederation


5

2.2 KEY AIMS AND ACTIVITIES OF TCBL

Against this background, the Textiles and Clothing Business Labs (TCBL) project, funded by the European Unions Horizon 2020 programme, aims to transform the European textiles and clothing industry from a reliance on conventional (price driven) manufacturing to one that is more socially and environmentally sustainable (e.g. by using more agile / disruptive and customer driven approaches to production). It aims to do this by encouraging real world experimentation across locations in the eight countries of the TCBL partnership and beyond - via the vehicle of a network of business labs and a number of business pilots (systems).

Business labs are spaces in which actors involved in TCBL can draw on existing and emerging models to freely experiment with new ways of designing, making, producing within specific locations in the countries covered by the TCBL partnership (eight EU member states). They are set up to explore future business scenarios based on emergent societal and technological trends and driven by new market trends. TCBL includes three types of labs:

Design Labs (immaterial value, emotion-oriented) explore tools and methods for designing textiles and clothes, working with professionals, fashion students, or anyone, even working from home;

Making Labs (material value, labour-oriented) experiment with production methods and machinery old and new, from re-discovering traditional tailoring to 3D printing and laser cutting;

Place Labs (spatial value, community and socially-oriented) investigate the local and social dimensions of clothes making, with new modes of organisation of work such as on-demand or home DIY production, community lab spaces, and networks of artisan shops.

The essential purpose of these labs is to produce and transfer knowledge and innovation into Business Systems, motivating potential pilots to emerge. There will be 15 start-up labs, run by existing project partners as an extension of their existing activities. Examples of labs include: exploring new ways for design (co-design, open design, etc); innovative equipment for cutting and sewing (make lab); the re-valorisation of artisan knowledge on a local level (place lab). This will be scaled up to 45 labs in the network via open calls starting in year 3 of the project.

These laboratories interact with a substantial number of sector enterprises of various dimensions pilot businesses who compose innovation elements coming from different Business Labs to identify transition scenarios that can accompany their shift from current ways of working towards more innovative and competitive business models. These business systems (pilots) are essentially established systems and processes that embody existing and concrete supply and value chains which will provide the infrastructure and processes for innovation transfer of business model elements from the Business Labs. These systems will work in two types of activities within TCBL:

Pilot laboratories (small existing structures) will pilot some small scale activities and productions, with an emphasis on developing social innovations that have an impact on the local community.

Pilot factories involve bigger units (factories) that want to experiment with existing innovations or new processes (e.g. reorganisation of working positions, multi-tasking).

The pilots are part of the innovation ecosystem of TCBL services but are external to the TCBL partnership with their engagement entirely driven by the value proposition of TCBL and methods of engagement (DOA, p. 44). The aim is to set up 160 pilots, 100 recruited at a later stage through an open call.


6

This process is supported by interactive knowledge and learning services together with an open repertoire of business services supporting specific moments of the new business models (training, logistics, etc).

Knowledge spaces capture knowledge about the textile and clothing system particularly (innovative) practices of production and including information on what works that can be used to support networking and learning. They are an online, interactive business model repository that hosts and links to embedded and emergent materials and manufacturing knowledge for T&C, as well as market, technology, economic and social trend observations and policy watching (DOA p. 38). The knowledge spaces are to trigger and support learning of its stakeholders about business models, resources and opportunities.

The Business Process Support Services support the satisfaction of market needs by supporting business labs and business systems in accessing, assimilating and adapting knew knowledge and innovative business processes to enable new ways of working in textiles and clothing to be developed and implemented. These business process services will utilise and valorise the evolving knowledge created through the Knowledge Spaces, as well as the additional services brought in by the Associate Service SMEs, for example providers of ICT services to the TCBL ecosystem (estimated to be around 60 at projects end and 100 after 5 years).

Figure 1 below shows how these core components of the TCBL project link together:

Figure 1: TCBL conceptual scheme (DOA p. 10)

The interactive and creative processes in TCBL aim to gradually build an integrated business ecosystem covering the entire value chain. The initial configuration in project year 2 will consist of 15 Business Labs and 60 Pilot Businesses, while by the projects end this will grow to 90 Labs, 240 Pilots, 35 service enterprises, and 15 start-ups. This exponential path of ecosystem growth will be sustained by yearly Calls for Expression of Interest in the TCBL Associates Programme, which will select new members to receive specific support and


7

assistance from project partners. The box below shows the key impacts that TCBL is expected to achieve:

Box 1: TCBL anticipated outcomes and impacts

5 per cent increase in manufacturing capacity within five years after the end of the project

Reduction in the environmental footprint by 20 per cent compared to products produced in the traditional value chains through less stock, less waste and less transportation

Creation of a novel supply network involving at least 100 organisations and individuals at the end of the project and 1000 organisations and individuals within 5 years after the end of the project.

Creation of new embedded services supporting the customer driven supply chain

TCBL is thus expected to have a widespread impact on the T&C industry in Europe, shifting consumer goals, expectations, and even engagement in the processes of designing and making clothes. This in turn will have both social and environmental impacts, as well as influencing attitudes towards responsible fashion and significantly improving the prosperity of Europes diffused system of production.

2.2 KEY FEATURES OF TCBL

From the description of the TCBL project above we can extract a number of key characteristics that are embedded in the project design and that have relevance for choosing a suitable evaluation approach. These are:

Emergence. The project is designed to be implemented in four phases: set-up (where aspects of the projects architecture are being made more specific); internal pilot cycle (where the idea of labs will be tested by actors internal to the TCBL consortium; engagement and impact (where the ecosystem is widened to include external actors as locations for labs and pilot activities); scaling up and out. Before each phase starts, transition times have been built into the project, allowing for the shaping / design of this phase and the taking on board learning from the preceding phase. These transition phases are the focus of project activities and partner commitments (DOA, p. 34). This emergence will shape the TCBL ecosystem itself in unpredictable ways, not only because we cannot foresee what kind of organisations will join (and what innovative transition paths they will bring) but also because through their work they will shape the business services and knowledge spaces themselves the content contributed, and potentially the services offered to pilots.

TCBL has therefore been designed as an open system. The ecosystem [metaphor] used in the DOA to describe the TCBL construct, together with the emergence concept described above, evokes the image of a living and breathing organism in which the different components (consortium partners, labs, associate pilots, stakeholders, technologies,) grow and interact, both with each other and (potentially) the wider social and ecological environment, to change the textiles and clothing industry.


8

TCBL is thus an innovative, if not experimental, project2: it creates an ecosystem that, through networking, aggregates shared knowledge, innovation, creativity and business innovations; this ecosystem combines innovation trends so that they may have an aggregate systemic impact on the T&C sector; uses technology as a tool for this change (rather than imposing it as a driver of change); enhancing existing services to textiles and clothing businesses. To paraphrase one partner interviewed: this is a new approach to pave the way for a different way of doing business, which has not really been done before. To paraphrase another interviewee: TCBL takes a whole industrial sector and messes the whole thing up, sees how the pieces fall and then re-aligns them.

Diversity of experience. TCBL can draw on significant experience and expertise through the composition of the partnership which is not only diverse (combining academics, leading industry actors and technology experts) but has, in parts, worked together before on projects with related topics. This diversity may produce innovation itself, as well as offer a valuable backdrop to the implementation of the project (in the absence of another template or recipe for carrying out this kind of at scale innovation successfully).

Within the framework of activities described in the DOA and the broad impacts to be achieved, TCBL as an intervention into the textiles and clothing system is thus characterised by a significant degree of uncertainty of interim outcomes and causal pathways of how these and final impacts will be achieved. It is a project / intervention that is organic shaping itself and being shaped by the environment with which it interacts. This makes TCBL a complex intervention3, and the approach to the evaluation needs to be able to account for these key features. This is discussed further in the next section.

3. THE APPROACH TO EVALUATING TCBL

Evaluation, broadly speaking, is a process of making a judgement on an intervention. It goes beyond monitoring (which usually involves the collection of information to track the progress of an intervention against targets and can also provide data on spend and outputs) in that it assesses process, outcomes and impacts of an intervention in relation to its intended aims. Evaluations tend to be done for any or all of the following purposes: accountability (to make sure money is spent appropriately); learning (about what works); or knowledge generation about effectiveness for the future.

Depending on the kind of intervention that is to be evaluated, the purpose of the evaluation and the kind of question(s) to be answered, an evaluation will typically be based on one of the following strategies:

A before and after design, where the only purpose is accountability and evaluation questions seek to measure the extent to which predicted outcomes have been achieved. This only works in simple interventions where cause and effect

2 From Consortium interviews, November 2015. 3 Rogers, P Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions Evaluation, Vol 14 No 1, p 31


9

relationships are easily detectable (e.g. the effect of speed cameras) as essentially what is involved is a comparison of monitoring information from before the intervention to the state after the introduction of an intervention to understand the scale of change achieved.

Experimental designs specifically test causal relationships and answer questions about whether the intervention has achieved the anticipated results. To this end, the evaluator compares two groups of people or situations: one that received a treatment (for example a new drug), another that did not (the control group or situation). Differences in outcomes between the two groups/situations are attributed to the intervention. Using this approach requires: a highly specified intervention that is implemented identically in different situations or locations; a high level of existing knowledge about likely cause and effects; ideally, randomisation of the treatment and control groups; and ideally a relatively short implementation period.

Theory based evaluations focus on exploring whether an intervention has worked and on explaining why and under what conditions change has been achieved in different contexts. They are particularly strong for evaluations of very diverse and long term interventions that display a mix of activities, target groups, delivery mechanisms and settings (i.e. interventions that are very complex). That is, they tend to be used in situations where the above approaches would not deliver reliable results because impact pathways are not straightforward (e.g. an intervention contains multiple strands of activities that may or may not interact), a number of external (or intervening) variables may influence the results of an intervention or where there is little prior knowledge about causality. In these instances, theory based evaluations work by articulating the implicit and explicit change theory underpinning an intervention (often represented in a theory of change or intervention logic model) and interrogating this theory throughout the evaluation process. The focus is on improving knowledge about causal pathways that lead towards anticipated outcomes and impacts whilst also capturing unintended consequences of the intervention. Theory-based approaches also allow inferences to be made about the possible long term impacts of an intervention.

It is clear from the short description above and the characterization of TCBL in the preceding section, that the TCBL evaluation approach needs to follow a theory based design as the other two designs are not suitable for the way the project has been conceptualised. In this context, the following section discusses what are the key purposes of the TCBL evaluation.

3.2 THE PURPOSES OF THE TCBL EVALUATION

According to Task 6.3 - Process and Impact Evaluation - of the DOA, the remit of the TCBL evaluation is as follows:

Box 2: TCBL evaluation task according to the DOA

This activity carries out ex ante, process and ex post evaluation of the innovation dynamics amongst the different stakeholders of the TCBL ecosystem and their different goals, objectives, and criteria of success. In the broader framework of the objectives of the project (), it carries out the benchmark and process evaluation required to make a reliable estimate of the longer term impacts of the project (p. 49)


10

This will include: the ongoing collection and assessment of business model innovations, the analysis of the manifest and hidden interdependencies between these innovations and the broader sectorial, socio-economic and policy related aspects, and creation of a set of strategic and operational guidelines for the various stakeholders involved (pp 18-19 DOA).

When asked what they see as the purpose of the TCBL evaluation, partners gave a range of responses, speaking to all three evaluation purposes outlined above:

Accountability: understanding whether project goals are being achieved and to this end monitor what partners are doing by tracking progress along KPIs on tangible project outcomes and suggesting improvements / alignments to project delivery (and avoid deviations).

Learning: from activities undertaken to influence and improve project delivery. This includes helping the project partners understand how others outside the immediate partnership see TCBL.

Knowledge generation: to create an evidence base about what works (business models, what kind of companies join the ecosystem, how they work together) and to generate and share best practices.

The Box 3 below presents the range of answers partners gave when asked about the purpose of evaluating TCBL.

Box 3: Partner thoughts on the purpose of evaluating TCBL

Accountability is the TCBL grant spent well?

We need to understand what has been achieved, so we understand whether we achieve goals, defining indicators to measure our performance very important.

Developing key performance indicators on tangible project outcomes and tracking progress along those.

Find out whether what we think we are doing we are actually doing and do it efficiently.

Of course we need to know if we are going to reach our goals.

For example, it will be important to see if we have reached the certain number of factories that we have wanted to reach.

At KPI level, evaluation of the different instruments that are aimed at achieving these objectives. Degree to which all of these internally run efficiently and contribute to the overall objectives of the project is something that deserves quite some evaluation. Look at KPIs and make them more tangible. What do they mean? E.g. carbon footprint: what does it mean, how do you get there.

Main purpose is to monitor what were doing and give us guidelines on solving some of the issues that are not performing as well.

To provide us with information about how much we have deviated from some of our original tasks in order to assess how we are doing and if necessary to go back and re-plan our path, in line with what is really possible when confronted with reality. So this is the formative element of the evaluation which has a very important role in monitoring of the project.

An embedded facility to support progress and alignment of activities

A mirror to be used as early warning for deviations, recurrent mistakes etc.

A management activity needs to inform the project Steering Committee about effectiveness and potential issues.


11

Learning what works and how can TCBL delivery be improved?

To feed back into and influence ongoing work.

Learning and reflecting. About questioning and understanding. Both in terms of working together and what we are achieving. Its not much about measuring. Because if you start an experiment, you dont really know how to measure. It is a creative process, it is not measurable in the same way as other things, there are different outcomes than more standard things. For example, failure in an experimentation is not necessarily a bad outcome.

To orient and guide us as a partnership and guide the activities within the project and then to also help partners to implement the right activities and the expected results.

To help us understand if we have a common image, vision of what we are doing.

Together with this I think we need an evaluation of how we are perceived outside [externally] because we might have the possibility of a situation of us having a good understanding internally that all is going well but in some distance outside actors are thinking of us differently. When I say outside I mean pilots in TCBL and those who are not involved but who are part of our audience in terms of dissemination and external comms. So, industry associations; big industry players, policy-makers at regional and national and EU level. This will be important so that we can correctly handle external comms to be more convincing about the worth of what we are doing and so we can organise accordingly.

Needs to provide evidence-based advice on the different choices that are available, and which choices are likely to work best.

Knowledge generation contributing to the evidence based of effectiveness for the future

To demonstrate that outcomes and impacts have been caused by the intervention of the project.

For example, it will be important to see if we have reached the certain number of factories that we have wanted to reach. But it is more than that. So, it is not only about companies joining the platform, but also about them working together. So it will also be interesting to understand/evaluate how we can see what kind of companies, entrepreneurs are ready to join the TCBL project.

To identify and share best practices.

To enable the results of the project particularly the generation of new business models to be appropriated especially by smaller and newer businesses in the T&C sector. In essence evaluation is financial advice. TCBL expects small and new businesses to adopt new business models. This will have an economic impact which could possibly be negative if the business model doesnt work. Its crucial that evaluation data provides evidence about which business models will work for whom in what circumstances.

All of this suggests that both a formative (focusing on processes) and a summative evaluation (focusing on results) are needed in TCBL. These each have two purposes.

The formative evaluation has both a design and a developmental purpose:


12

The design purpose is to help clarify the intended aims and outcomes of the project plan. This corresponds to the ex-ante phase in the project and evaluation life cycle and ex-ante evaluation mode. However, as TCBL is an emergent project, the project life cycle does not have a conventional linear phasing path, but follows a series of transition paths. Ex-ante evaluation is not therefore confined to the beginning (or planning) stage of the project but follows these transition paths. These transition paths mark stages in innovation that the Business Pilots in TCBL are expected to go through as they move towards new forms of business models which, in turn, will be transferred into the wider T&C sector through established and evolving Business Systems. The pilots provide the evidence base for the transferability of new knowledge and practices into real Business Systems. There are three transition points in these transition paths: T1 - existing business models and standardised formulations, i.e. from set-up to pilot; T2 - the integration of one or more elements from the Business Laboratories pilots into real businesses; T3 from individual to scaled up results.

The developmental purpose is about following the project process and supporting the different stakeholders involved in assessing how the initiative is doing and whether it is on track. This focuses not only on collecting ongoing data as the project evolves for example collection and assessment of business model innovations but also assessing these data and feeding the results back to help the Business Labs, pilot labs and pilot factories to work more effectively (i.e. evaluation in operational mode).

The formative element therefore speaks to both the accountability and the learning purposes required of the TCBL evaluation. It addresses the requirement in the DOA for a process component to evaluating TCBL. Indeed, this is particularly important because of the ecosystems design of TCBL. Ecosystems ecology tries to understand how the system as a whole (rather than individual parts) operates: This means that, rather than worrying mainly about particular species, we try to focus on major functional aspects of the system. These functional aspects include such things as the amount of energy that is produced by photosynthesis, how energy or materials flow along the many steps in a food chain, or what controls the rate of decomposition of materials or the rate at which nutrients are recycled in the system.4 The analogy to the TCBL ecosystem is that the task of the process evaluation is to investigate what the major functional aspects of the TCBL ecosystem are and how they work.

The summative evaluation has a learning and knowledge generation purpose:

The knowledge purpose is about providing evidence on the outcomes and impacts of TCBL, including the likely long-term impacts on key targets like reducing the industrys environmental footprint and increasing manufacturing capacity. This includes in particular exploring the granularity of the TCBL ecosystem the interactions between ecosystem actors, their context and TCBL activities and whether and in what ways these lead to change. The summative evaluation also needs to produce meaningful indicators that enable us to assess the extent to which the project has achieved its intended objectives and outcomes. It needs to demonstrate what is likely to have happened to the T&C sector if the TCBL ecosystem had not been

4 The Concept of the Ecosystem: http://www.globalchange.umich.edu/globalchange1/current/lectures/kling/ecosystem/ecosystem.html


13

established the counter-factual perspective. There is therefore also an accountability dimension to this purpose.

The learning purpose of the TCBL evaluation is about both assessing the transferability of the results of TCBL to similar initiatives in the future and contributing to supporting the replication and sustainability of TCBL innovations.

Again, these purposes support the use of a theory based evaluation design. The question now is how to shape this design so it matches with the key concepts embedded in TCBL. This is the aim of the next section.

3.2 REFINING THE EVALUATION CONCEPT

This section presents the results of a review of three key concepts that shape the TCBL approach and which therefore need, in turn, to be reflected in the evaluation approach and design. The three key concepts are: ecosystems; complexity and large scale change. The discussion below is based on a narrative review of the literature in these three fields.

3.2.1 AN ECOSYSTEMS PERSPECTIVE ON EVALUATION

There is not much literature on designing evaluations from an ecosystems perspective. The literature is dominated, perhaps unsurprisingly, by work carried out in the field of natural resources management focusing in particular on two areas: first, methodologies developed to assess the ecological status of a specific territory or habitat (for example carrying out an environmental audit to identify the extent to which an areas species are threatened); second, methodologies developed to assess environmental impact. However, work in this mainstream field of environmental ecosystems has stimulated work in other fields, borrowing and adapting concepts and methods from natural resource management, for example, to develop a body of knowledge in fields of interest for TCBL notably in business ecosystems. This branch of the literature presents similar preoccupations to the work in environmental ecosystems, utilising concepts and methods drawn from systems theory and complexity theory.

ENVIRONMENTAL ECOSYSTEMS AND RELEVANCE FOR TCBL

One important strand in this literature is the work that has developed around the Convention on Biological Diversity. This has focused on promoting standardisation in ecosystems management, and has involved a range of stakeholders including government agencies, NGOs and scientists, working within the umbrella of the United Nations Environment Programme (UNEP) to explore the need for an international convention on biological diversity. The Convention identifies 12 principles that are recommended to be applied in ecosystems management, represented in the box below.

Box 4: Principles for ecosystems management5:

5 https://www.cbd.int/ecosystem/principles.shtml


14

1. The objectives of management and land, water and living resources are a matter of societal choices this focuses on the need for all stakeholders voices to be represented in any intervention that affects the ecosystem. Ecosystems need to be managed for their intrinsic values and in a fair and equitable way.

2. Management should be decentralised to the lowest possible level this emphasises the need to apply participatory approaches in ecosystem management that balance local interests with the wider public interest, and the use of local knowledge in designing and implementing interventions.

3. Ecosystem managers should consider the effects of their activities on adjacent and other systems this highlights the need to consider the displacement and multiplier effects of interventions.

4. Understand and manage the ecosystem in an economic context this highlights the need to understand how ecosystem interventions have significant, and often complex, economic impacts. For example, introducing alternative methods of land use can trigger market distortions that ultimately can have a negative effect on local living standards.

5. Conservation of ecosystem structure and functioning should be a priority target of the ecosystem approach this argues that it is not enough to focus simply on protecting species. Ecosystem functioning and resilience depends on the dynamic relationships between, within and among species and their environment.

6. Ecosystems must be managed within the limits of their functioning this emphasises the need for caution in designing and implementing interventions. You need to know the limits of the ecosystem and the constraints to its development.

7. The ecosystem approach should be taken at the appropriate spatial and temporal scales this essentially means defining the boundaries.

8. Recognising the variability of temporal scales means that ecosystem management should focus on the long-term perspective this highlights the tendency for humans to prioritise short-term gains. Instead, the perspective should focus on the long-term.

9. Management must recognise that change is inevitable this emphasises the need to anticipate how changes to the ecosystem will inevitably create systemic and often unpredictable changes throughout the system. Management processes therefore need to adopt systems and processes to both predict possible evolutionary trajectories and to track these trajectories.

10. The ecosystem approach needs to balance conservation and diversity there is a need to shift to more flexible approaches that recognise the need for preservation, but can also incorporate human intervention.

11. The ecosystem approach should consider all forms of relevant information effective ecosystem management strategies need to draw on information and knowledge from a wide range of sources, combining scientific with indigenous and local knowledge.

12. The ecosystem approach should involve relevant sectors of society and scientific disciplines this reinforces principle 11: the knowledge and expertise of all stakeholders are equally valuable.

The literature on evaluation of ecosystems though limited reflects the principles set out in the Convention. The James Hutton Institute has developed an evaluation framework for assessing what has been termed the ecosystem approach (Waylen et al, 2014). The framework highlights the need to understand how trade-offs work in ecosystems. Because ecosystems are complex systems, in which a wide range of actors work in a continually changing environment, the key to evaluation lies in capturing how these different actors


15

negotiate and manage transactions. Echoing the principles of the Convention on Biological Diversity, the Hutton Framework sets out a number of principles for ecosystem evaluation:

Decentralisation and empowerment of stakeholders is important to the ethos of the Ecosystem Approach.

Iterative monitoring and evaluation is necessary to allow adaptive management and learning, so the project should ideally plan for monitoring, and monitor a range of social, economic and cultural issues as well as ecological issues

The complexities of natural systems present several challenges for evaluation: since natural systems are very complex, and sometimes poorly understood, it is often hard to track or understand changes resulting from management actions.

Ecological processes sometimes take decades to reflect the effects of changes in management practices.

Single indicators or measures are unlikely to be sufficient as it becomes harder to define what success is, let alone how to measure it.

The Ecosystem Approach has social goals that are not directly about the environment: not only does it attempt to conserve resources, it is explicit that there should be equitable use and benefits from those natural resources.

These social goals may be about project processes e.g. raising awareness or empowering vulnerable groups, or may be about outcomes e.g. ensuring stakeholders will receive economic benefit. Thus it is necessary for evaluators to study process as well as outcomes.

Different goals may even be in tension: for example some environmental goals could be achieved efficiently without meeting goals for stakeholder involvement or social benefit. Thus, any single project will have more than one goal or objective to consider when evaluating the overall success of the project

Evaluation will never result in a simple objective binary distinction of success or failure. This is because different social and ecological goals may or may not all be achieved, for a variety of reasons, and because there will be multiple points of view as to whether success has been achieved.

However, tracking multiple dimensions of project design, context, progress and outcomes can help to produce a balanced and fair understanding of what a project has achieved and why.

These evaluation principles mirror to a large extent the broader principles of ecosystem management embodied in the Convention: the centralisation of stakeholders and local knowledge in evaluation; the need to capture complexity; an evolutionary, formative focus for evaluation; the need to apply multiple indicators, reflecting the different perspectives and constructions of different stakeholders.

An example of the implementation of this kind of approach in practice is an evaluation of a tropical fishery initiative, the Mombasa coral reef and seagrass ecosystem, undertaken by Daw et. al. (2014). This focused on surfacing, mapping and exploring how the complexity of interactions between a wide range of different actors involved in the initiative were expressed in the form of different trade-offs between economic, cultural, ecological and societal considerations. Using an evaluation approach that combined ecological simulation with participatory evaluation, the evaluation showed that food production, employment, and well-being of marginalized stakeholders were differentially influenced by management decisions leading to trade-offs. Some of these trade-offs were suggested to be taboo trade-offs between morally incommensurable values, such as between profits and the well-being of marginalized women. These were not previously recognized as management issues.


16

Stakeholders explored and deliberated over trade-offs supported by an interactive toy model representing key system trade-offs, alongside qualitative narrative scenarios of the future. The concept of taboo trade-offs suggests that psychological bias and social sensitivity may exclude key issues from decision making, which can result in policies that are difficult to implement.

BUSINESS ECOSYSTEMS

A central strand in the literature on business ecosystems and their evaluation focuses on exploring the relationship between business entities and business models and how these inter-relate to form business networks. This is particularly relevant for TCBL, which seeks to promote new kinds of networks, and hence new kinds of value chains within the Textile and Clothing sector, by stimulating the production, diffusion and replication of business innovations within the Business Labs and pilots. In this context a business, or service, ecosystem is defined as a value co-creation configuration of people, technology, shared information, and value propositions connecting internal and external service systems (Allee, 2013). Allee describes relationships between business entities by three types of value transactions: goods, services, and revenue; knowledge; and intangible value. Business ecosystem evaluation has concentrated on identifying how these relationships and transactions work and what added value they contribute. In value network research and evaluation, a business model is typically used to describe the roles and relationships of a company, its customers, partners, and suppliers, including the flow of goods, information, and money among these parties and the financial benefits for those involved. Business networks describe the infrastructure that underpins an ecosystem connecting these different business models.

Work on exploring how these networks operate has drawn extensively on game theory and gaming models (Baron, 2012). An important part of modelling and analysis of a business ecosystem is to capture the dynamic interactions among ecosystem business entities.

On example is the BEAM approach (Tian et al, 2008). BEAM developed a comprehensive framework to integrate business ecosystem modeling and analysis capabilities using game theory as the main entity behavior model, and value network modelling as a systematic modelling method. A case study of a retail B2B service platform was used to demonstrate how actual system performance can differ from expected behaviours, which do not take into account the effects of interactions among business entities. Figure 2 below shows the components used in the BEAM approach to analyse the value chains in a business ecosystem in order to assess impact.

Figure 2: Ecosystem attributes: the BEAM model (Tian et al, 2008)

Class Concept Properties

Resource Such elements as monetary, human capacity, machine, software, and power that can be consumed in the execution of business activiteis or invested to realize a service

Owner

Unit cost

Activity A task that uses resources Resource consumption

Decision The selection of a course of action among variations, such as pricing and capabitliy allocation

Objective

Decision variable set

Constraint set


17

Related decision variable

Metric Performance indicator of a business object (activity, business entity or service ecosystem)

Business object

Value

Role A set of connected activities and decisions in a service ecosystem

Activity list

Decision list

Metrics list

Business entity

A general term used for enterprises, business units and regulators

Goals (e.g. payoff funciton, risk attitudes)

Demographic properties (e.g. capacity, size, and location)

Business model

The roles and relationships of a company, its customers, partners, and suppliers as well as the flow of goods, information, and money among these parties and the financial benefits for those involved.

Parntership

Decision-making structure

Decision making mechanism

As the Figure shows, the key attributes that need to be evaluated in a business ecosystem, according to this model, are resources (e.g. human capacity, money, ICTs), activities that use these resources, decisions (that directly affect business activities), roles (the set of connected activities and decisions within the service ecosystem), metrics (used to measure the performance of the role, a decision list that lists all the decisions the role can make, and an activity list that lists all the business activities played by the role, where activities, metrics, and decisions are each elements of the ecosystem model). Data collected for these entities are then input into a simulation model that calculates the expected outcomes and impacts of the ecosystem.

The BEAM example shows how business ecosystems because of the complex interactions that take place between divergent actors within an environment that is constantly changing depart from predictive models of organisational behaviour that focus primarily on rational strategies that are rooted in economic decisions.

Rong and Shi (2014), in their analysis of business ecosystems that have evolved around the hi-tech sector in China, convincingly show how pointless it is to isolate individual organisational entities from their wider eco-sphere. Taking as one example the (unregulated, and therefore illegal) growth of the electric vehicle sector in China, Rong and Shi demonstrate how supply chains cannot be separated from a much wider range of business, environmental, societal and regulatory systems each of which combine to produce the different value propositions of the ecosystem.

The challenge for evaluation in this field is to capture the dynamism of these rapidly evolving networks and value chains whilst at the same time imposing some degree of boundary and structure on the evolving ecosystem as a process in order to demonstrate value added, effectiveness and impact. Li et al (2013) provide one evaluation model that attempts to do this within the field of health ecosystems. This model builds on a framework initially developed by Guo et al (2012) which suggests five attributes of the business ecosystem that need to shape evaluation: logical community structure; favourable non-physical environment; efficient system productivity; sustainability; co-ordination of management. Li et al expand this framework to


18

cover eleven ecosystem attributes: energy, vitality, resilience, service to maintain business ecosystem, transmission on innovation, investment reduction, harm to neighbouring systems, effect on human health, power performance, inclusiveness and robustness. The model was tested in a real world health system using ecological and complex adaptive systems theory. Table 1 below shows how these attributes are translated into evaluation criteria and indicators and outcomes.

Table 1: Ecosystem Evaluation Model (Li et al, 2013)

Evaluation criteria Indicators Outcomes

Ecological attributes Ability to promote performance by strategic arrangement and resource integration

Completion level from price to brand

Efficient and innovative management

Emergence

Synergetic evolution

Self organisation

Adaptability

No. of supportive enterprises

No. of complementary enterprises

Basic enterprises

Structural attributes Financial contribution of suppliers to enterprises

Expanded enterprises

Technical accumulation

Investment prospects

Supporting institutions

Promotion of products by independent institutions

Complementary organisations

Industrial policy and regulatory support

Correlative environment

Functional attributes Rate of return on equity

Technical capital growth rate

Productivity

Enterprise survival rate

New enterprise success rate

New enterprise growth rate

Vitality

Differentiated enterprises growth rate

Creativity

Operational attributes Transparency and confirmation of value platform strategy

Strategic clarity

Commercial opportunities of value platform strategy

Platform compatibility

Trust level and co-operative distribution

Stability


19

Conflict resolution Conflict co-ordination

Lifecycle attributes National economy and technical level of ecosystem

Life stage staged development

IMPLICATIONS FOR TCBL EVALUATION DESIGN

The above presentation of key issues for evaluating from an ecosystems perspective identified a number of recurring themes that reflect the distinctive attributes of ecosystems and which need to shape the TCBL evaluation. It seems clear that the TCBL evaluation approach should embody:

A participatory evaluation methodology that incorporates an element of action research, based on engaging with key stakeholders in particular real world business entities who will access and adapt business innovations emerging from the Labs and pilots. In this context, the evaluation role becomes one of co-creation of knowledge and action.

This also means that the evaluation needs to combine a broad range of data and information that reflects the spectrum of involved stakeholders using scientific data as well as local knowledge. This in turn requires the construction of multiple sets of evaluation indicators. Whilst these need to be in line with the evaluation questions, these could follow the model developed by Li et al. (see Annex 1).

It follows that, since stakeholders and their different constructions of and positions on TCBL are centralised within the evaluation, it needs to devote attention to ensuring that all stakeholder voices are represented particular those who are less powerful. This will be a real challenge, since there is a clear tension in TCBL between the interests of small entities (and individuals), communities and major players like the large T&C retailers. Sensemaking actions will therefore be crucial in the evaluation.

The evaluation should prioritise formative (process and action-driven) evaluation over summative evaluation.

Summative evaluation needs to consider the long-term view. The evaluation needs to incorporate methods to calculate the expected impacts of TCBL in the long term, with a time horizon of 2020.

Process evaluation needs to focus in particular on the following: how entities within the ecosystem access and engage with business innovations; how they make decisions on the utility of these innovations; how they trade-off different aspects of innovation utility to arrive at a calculation of value for their business; how different entities make trade-offs between themselves within the ecosystem; how the extracted value of business innovation is then applied in practice; whether this extracted value is short term or long term.

3.2.1 EVALUATING COMPLEXITY

Working with the ecosystems idea in the TCBL evaluation means also working from a complexity perspective (and we have already characterised TCBL as a complex intervention).

A COMPLEXITY PERSPECTIVE ON THE TCBL EVALUATION

Complexity theory is an interdisciplinary field of research that has become recognised as a new field of inquiry focusing on understanding and better describing dynamics and processes


20

of change found in a range of physical and biological phenomena. The notion of complexity and complexity theory gradually absorbed elements of general systems theory, cybernetics, chaos theory and information theory (ODI: 2008).

Following initial applications to the physical sciences, increasing attention is now being paid to how the key ideas and concepts of complexity science (e.g. notions of emergence, nonlinearity, dynamic systems, interactions and interrelations) can help researchers and practitioners understand (and influence) social, economic and political phenomena. The underpinning assumption is that much change in contemporary situations is not linear (Rogers 2008). Within this shift, complex systems thinking was applied in management and to the study of social phenomena by the social science disciplines, to development and to policy analysis. Complexity theory has also increasingly been discussed and applied within evaluation theory and practice over the past decade (Ling: 2012) in order to evaluate what are defined as complex interventions. In particular, the interest in complexity centres on a growing recognition of the disjunction between the non-linearity and unpredictability of change processes and the protocols and procedures that govern development interventions that assume otherwise. The box below summarises 10 key concepts of complexity theory.

Box 5: 10 principles of complexity theory

Complexity and systems. These first three concepts relate to the features of systems which can be described as complex:

1. Systems characterised by interconnected and interdependent elements and dimensions are a key starting point for understanding complexity science.

2. Feedback processes crucially shape how change happens within a complex system.

3. Emergence describes how the behaviour of systems emerges often unpredictably from the interaction of the parts, such that the whole is different to the sum of the parts. In complex interventions, the notion of emergence means not that certain patterns emerge as our understanding of them improves (knowledge which can then be used to predict similar interventions in the future), but that the specific outcomes, and the means to achieve them, emerge during implementation of an intervention.

Complexity and change. The next four concepts relate to phenomena through which complexity manifests itself:

4. Within complex systems, relationships between dimensions are frequently nonlinear, i.e., when change happens, it is frequently disproportionate and unpredictable.

5. Sensitivity to initial conditions highlights how small differences in the initial state of a system can lead to massive differences later (butterfly effect).

6. Phase space is a way to describe complex systems because it does not seek to establish known relationships between selected variables, but instead attempts to shed light on the overall shape of the system by looking at the patterns apparent when looking across all of the key dimensions to build a picture of the dimensions of a system, and how they change over time. This enables understanding of how systems move and evolve over time.

7. Chaos and edge of chaos describe the order underlying the seemingly random behaviours exhibited by certain complex systems.

Complexity and agency. The final three concepts relate to the notion of adaptive agents, and how their behaviours are manifested in complex systems:


21

8. Adaptive agents react to the system and to each other, leading to a number of phenomena (contextual sensitivity).

9. Self-organisation characterises a particular form of emergent property that can occur in systems of adaptive agents.

10. Co-evolution describes how, within a system of adaptive agents, co-evolution occurs, such that the overall system and the agents within it evolve together, or co-evolve, over time.

Evaluating from a complexity perspective also means thinking through what a complex intervention is. In line with the principles above, complex interventions are not predictable, involve emergent and responsive interventions and causal processes which cannot be completely controlled or predicted in advance. The interactions between determinants of the sub-problems can lead to non-linear causal relations between potential causes and outcomes. Also, context-sensitivity can make a problem complex. As a consequence, outcomes are unpredictable. To solve complex problems, formulae and standardised solutions that proved effective in the past provide little guidance. Instead, complex problems are solved through safe-fail experiments that allow learning by doing or by making sense of events post facto.

A particularly useful report on evaluating complexity, by Preskill and Gopal (2014) seems to encompass most aspects of complexity concepts. They outline some key characteristics of complex systems and, from them, infer propositions for evaluating complexity. In coming up with the following characteristics of complex interventions, they have borrowed from a variety of complexity scientists and theorists, as well as evaluation practitioners.

Table 2: Features of complex interventions

A complex system is always changing, often in unpredictable ways; it is never static

A complex intervention has multiple components, interacting with each other; everything is connected; events in one part of the system affect all other parts

Interventions are open systems and change through learning as stakeholders come to understand them.

Context matters

Each situation is unique; best principles are more likely to be seen than best practices

Relationships between entities are equally if not more important than the entities themselves

Cause and effect is not a linear, predicable, or one-directional process; it is much more iterative

Patterns emerge from several semi-independent and diverse agents who are free to act in autonomous ways

Complexity theory thus represents a challenge to the linear assumptions about how change happens that underpins many funded programmes. These tend to be based on a predictive logic, i.e. if a particular set of actions are taken these will result in a particular set of outcomes. This works very well where there are known solutions to problems, and stable and replicable contexts, but is often problematic in real-world social contexts which are constantly changing and in which multiple actions are combining with each other time and again to produce a highly unpredictable environment, such as the case of TCBL.


22

IMPLICATIONS FOR TCBL EVALUATION DESIGN

The text above has the following implications for the design of the TCBL evaluation:

Designing an evaluation approach that is iterative and able to capture data in real-time. Because complex interventions are not fixed, their evaluation requires real-time data collection, to support ongoing learning and reflexivity. In this sense, outcomes and impact can only be assessed against the decisions made during the course of the programme, not against the programming logic at the start of the programme. This means that process and content (including any changes in direction and the rationale for them) is important and must be documented as we go along. The iterative nature of the process enables to: understand what has changed in the system, and what specifically has changed as a result of an action taken; adjust actions/goals etc to maximise impacts; set new baselines and indicators of success against those baselines. Overall, complexity calls for a more adaptive mode, similar to the intentions of developmental evaluation (Patton: 2006, cited on: http://betterevaluation.org/plan/approach/developmental_evaluation), where real-time evaluative practice stays close to unfolding intervention so that it is part of it, or longitudinal study of impact evaluation (CDI conference paper, 2011).

Inclusion and sense making. In conditions of complexity, multiple competing explanations of why and how change happens exist, including tensions between the large number and diversity of stakeholders involved. This has two key implications for the TCBL evaluation. The first is that evaluation will need to include all TCBL stakeholders at all levels of the project to enable the multiple perspectives to emerge. Second, the evaluation will need to build in sense-making opportunities. Much of the literature reviewed specifically used the term in cooperation with partners rather than participation with partners, in ways that allow multiple perspectives to inform the findings while being mindful of power relations among participants. In general, the two points discussed above (the iterative nature of an evaluation design that seeks to evaluate a complex intervention and the importance of sense-making in the evaluation process) tend to stress the usefulness of adopting principles of Action Research methodologies when assessing impact in complex and highly dynamic environments.

Strengthening the formative role for the evaluation. Perhaps similarly to the points above, a common message from the literature is that when evaluating a complex intervention, there is value in, or indeed a necessity for, shifting towards an evaluative practice that emphasises reflective practice and organisational learning in the form of a formative evaluation. Attached to this, is the notion that there is often a blurring or an overlap between the impact evaluation and the creation of evidence by the project itself as it learns and adapts (formative). Due to the emergent nature of complex interventions, it is not uncommon for the balance between the two to shift towards a more formative role because it takes on, as noted above, more of the characteristics of a real-time evaluation (Ling: 2012).

The importance of context. Context in evaluation typically involves understanding five specific dimensions: demographic characteristics of the setting and the people in it, material and economic features, institutional and organisational climate, interpersonal dimensions or typical means of interaction and norms for the relationships in the setting, and political dynamics of the setting, including issues and interests (Preskill and Gopal: 2014). Because complex initiatives tend to involve multiple actors and organisations, are implemented over multiple years, and naturally adapt in response to changing conditions (challenges and opportunities, as well as negative and positive stimuli), evaluations need to capture information on how the initiative and its context are co-evolving. In other words, the evaluation should not only study the context and its influence, but also measure the ways in


23

which the initiative affects the context. Evaluation findings should be interpreted and grounded in context to support claims about an initiatives progress and impact.

Theory of change and counterfactual. Understanding the projects Theory of Change remains an important feature of evaluations of complex programmes (Rogers: 2012; Ling: 2012; Burns 2014). In emergent projects designed to accommodate the uncertainties associated with complexity, the Theory of Change will include attention to the importance of learning and adaptation. It will also identify the key dependencies upon systems and subsystems which lie outside the formal structures of the intervention. Burns (2014) also points out that a theory of change should constantly be tested in action through an iterative process, as learning takes place. Whilst in complex interventions it is often much harder to identify the counterfactual, nevertheless it is crucial to pose the core question in an evaluation which is did it make a difference?. Answering this question must in turn involve asking compared with what?. However, the counterfactual for a complex intervention is not a single outcome but a counterfactual space of more or less likely alternative states. This might be produced by scenarios, modelling, simulation, or even expert judgement.

3.2.1 LARGE SCALE CHANGE

Finally, we suggest it is useful to think of TCBL as a large scale change effort, and to consider what this means for evaluation design.

Large scale change (LSC) can be defined as the emergent process of mobilising a large collection of individuals, groups and organisations towards a vision of a fundamentally new future state, by means of:

High leverage key themes A shift in power and a more distributed leadership Massive and active engagement of stakeholders Mutually reinforcing changes in multiple systems and processes.

Done properly, this leads to such deep changes in attitudes, beliefs and behaviours that sustainability becomes largely inherent.6

According to the NHS Institute for Innovation and Improvement7, there are three ways of thinking about large scale change:

1. Kurt Lewins distinction between incremental and transformational (ie large scale) change. Transformational change may be experienced as a series of small (incremental) steps, but has a vision behind it where, by the end of the process, everything would be different.

2. The complex systems lens. Large scale change requires integrated changes in structures, processes and patterns (of behaviour and outcome).

3. Three dimensions of LSC. According to NHS Institute, large scale change is one that is:

6 NHS Institute for Innovation and Improvement (2013) Leading Large scale change: a practical guide, p. 30 www.nhsiq.nhs.uk/download.ashx?mid=8526&nid=8530 7 NHS Institute for Innovation and Improvement (2013) Leading Large scale change: a practical guide, www.nhsiq.nhs.uk/download.ashx?mid=8526&nid=8530, pp18-21


24

a. Widely spread across geographical boundaries, multiple organisations, or multiple distinctive groupings.

b. Deeply challenging to current mental models and ways of thinking (it feels uncomfortable and evokes some push back from others because it is so different from the usual).

c. Broadly impacting on what people do in their lives or time at work and requiring co-ordinated change in multiple systems.

Large scale change normally involves a combination of technological and social systems challenges and is approached from an open systems view, especially when pursued through an ecology or social movement lens.

Figure 3: Dimensions of large scale change

Source: NHS Institute for Innovation and Improvement (2013) Leading Large Scale Change, p. 22

There are 10 key principles that underpin Large Scale Change. These are represented in the box below.

Box 6: 10 Principles of Large Scale Change8

1. Movement towards a new vision that is better and fundamentally different from the status quo. LSC is fuelled by the passion that comes from the fundamental belief that there is something very different and better that is worth striving for.

2. Identification and communication of key themes that people can relate to and that will make a big difference. The vision is out there and in the future. Typically, if it is truly LSC it will seem so distant and so in contrast with the current reality that it may feel overwhelming or impossible. In order for people to get engaged they have to see what they can do, now or soon, that would be clear and meaningful step along the journey.

8 NHS Institute for Innovation and Improvement (2013) Leading Large scale change: a practical guide, p. 26-29, www.nhsiq.nhs.uk/download.ashx?mid=8526&nid=8530


25

3. Multiples of things (lots and lots). LSC is complex with many different stakeholders, agendas (both hidden and open), points of view, needs and wants, details, systems that need to change and so on. Attempts to isolate or work around some groups, or to ring fence some parts of the system to be left alone while others must change, typically result in something less than LSC.

4. Framing the issues in ways that engage and mobilise the imagination, energy and will of a large number of diverse stakeholders in order to create a shift in the balance of power and distribute leadership. Because there are lots of lots, a small band of leaders cannot possibly make the LSC happen. Tight, centralised planning and control actually works against LSC. Instead, multiple leaders from across the system and at all levels, drawn to the vision must engage and commit their will and energy to the effort of achieving LSC. As more distributed leadership emerges and is enabled across the system, cross boundary and partnership working increases and change happens at a massive scale and pace. But the key lies in gaining the commitment of others to act, not merely their compliance in doing what you tell them to do. Experience shows that change based on compliance without commitment is difficult to sustain over time.

5. Mutually reinforcing change across multiple processes / subsystems. If the vision is sufficiently clear and the collection of key themes comprehensive enough, what may seem at first like a chaotic lack of control actually comes together in the form of changes that connect with and build upon, one another.

6. Continually refreshing the story and attracting new, active supporters. While LSC efforts often start small, with just a few people who are switched on by the vision, the life blood of LSC is the continual stream of new supporters who become attracted to the vision when they see it progressing. Without a steady stream of new supporters becoming committed to the change, LSC efforts can plateau or run out of energy.

7. Emergent planning and design, based on monitoring progress and adapting as you go. Because of the complexity and uncertainly involved, LSC outcomes are impossible to predict at a detailed level. Flexibility, adaptability and engagement of others are the keys. It is OK to have detailed plans and milestones () just dont spend too much time on them before you start actually doing something and dont be surprised if every detail does not work out as planned.

8. Many people contribute to the leadership of change, beyond organisational boundaries. As LSC efforts spread and become increasingly complex, more and more leaders need to be recruited for the change effort. Leadership of the LSC effort is not dependent on a small number of key individuals in a hierarchy. This is distributed leadership; a variety of different sources of leadership expertise move into play and are spread around the system. Leaders pool their effort and expertise so that the collective result is significantly greater than the outcomes of individual leadership actions.

9. Transforming mindsets, leading to inherently sustainable change. According to the literature, when LSC is done well, sustainability is the natural by product. If people have really become engaged and believe that that vision is more desirable than the status quo (and you have addressed multiple structures, processes and patterns) they will be committed to and will fight to keep the new way, showing little interest in going back to the old. However, if powerful leaders have simply demanded compliance and


26

not brought others along in the thinking, those on the frontline will put things back to the old ways. The way they still believe was right.

10. Maintaining and refreshing the leaders energy over the long haul. The case study literature on LSC makes it clear that large scale change can take some time to unfold completely. Too many leaders simply run out of steam.

It is easy to see that these resonate quite well with the TCBL project design, in particular themes 1, 3, 4, 6, 7 and 8. Large scale change thus offers a practical lens for the evaluation that is compatible with ecosystems and complexity thinking: the ten principles articulated above appear particularly useful to shape the formative and process evaluations of TCBL by offering some concrete markers against which to analyse TCBL activities and where necessary suggest improvements.

3.3 SUMMARY AND IMPLICATIONS

The table below summarises the key elements that each of the three concepts brings to the TCBL evaluation task. It shows that the concepts complement each other and highlights the specific dimensions that each of the three perspectives has to offer. The next step is to work these requirements into the details of the evaluation design: the questions, process and data collection methods. This is done in the next chapter.

Working Paper Towards Deliverable 6.3 Final Try

Documents

Transcript of Working Paper Towards Deliverable 6.3 Final Try