Vol.:(0123456789)1 3

Sustainability Science (2019) 14:405–419 https://doi.org/10.1007/s11625-018-0532-y

ORIGINAL ARTICLE

How transdisciplinary projects influence participants’ ways of thinking: a case study on future landscape development

Silvia Tobias1  · Maarit F. Ströbele2 · Tobias Buser1,3

Received: 24 May 2017 / Accepted: 17 January 2018 / Published online: 23 January 2018 © The Author(s) 2018. This article is an open access publication

AbstractTransdisciplinary (TD) approaches have increasingly been promoted in the field of land-use research. However, the theoretical discourse about transdisciplinarity is far more advanced than its implementation in practice. In particular, empirical stud-ies about the effects of concrete TD projects on the participants are rare. We evaluated joint knowledge generation among researchers and non-academics in a TD research programme on urban and landscape development. For the assessment we used standardised questionnaires, our own observations, and a simplified implementation of the ‘most significant change’ method. The evaluation revealed that the participants gained considerable TD knowledge through encountering different thought-styles and problem owners. They together developed a feeling of companionship, broadened their views on the issue and, consequently, attributed increased legitimacy to it. The following aspects of TD research were found to be successful as the programme: offered opportunities for enthusiasm; used a form of communication that promotes mutual trust; and provided boundary objects. Similar to other studies, we observed the creation of hybrid spaces and communities of research and practice where the participants could build up mutual trust, interact with other thought-styles, and jointly develop their TD knowledge.

Keywords TD knowledge · Stakeholder view · Thought-styles · Most significant change method · Questionnaire · Hybrid meeting place

Introduction

Transdisciplinary (TD) research is often promoted as the key to solving complex societal problems, and is typically applied when normative problems arise in which valuation

and decision-making are major issues. Consequently, TD approaches have increasingly been promoted in the field of land-use research. However, in a comprehensive literature review, Zscheischler and Rogga (2015) found that the theo-retical discourse about transdisciplinarity in land-use science is far more advanced than its implementation in practice, as Brandt et al. (2013) also noted for the TD research in sus-tainability science. No consistent definition of transdiscipli-narity is, though, available even in the theoretical literature, but three characteristics of TD research can be identified: (1) the research seeks to contribute to solving a problem in practice; (2) researchers from different disciplines and practitioners collaborate; and (3) the methodology is devel-oped during the research process and in a problem-specific way (Wickson et al. 2006; Lang et al. 2012; Zscheischler and Rogga 2015). This definition is unfortunately limited as the focus is only on the process with little consideration of the results or effects of this kind of research (see e.g. Klein 2010; Lang et al. 2012). Moreover, the TD literature takes a very science-oriented perspective with little mention of the

Handled by Dr. Eefje Cuppen, Delft University of Technology, Netherlands.

Electronic supplementary material The online version of this article (https://doi.org/10.1007/s11625-018-0532-y) contains supplementary material, which is available to authorized users.

* Silvia Tobias silvia.tobias@wsl.ch

1 Swiss Federal Research Institute WSL, Landscape Ecology Group, Zürcherstrasse 111, 8903 Birmensdorf, Switzerland

2 Swiss Federal Research Institute WSL, Social Sciences in Landscape Research Group, Zürcherstrasse 111 8903 Birmensdorf, Switzerland

3 Network for Transdisciplinary Research td-net, Laupenstrasse 7, 3001 Bern, Switzerland

406 Sustainability Science (2019) 14:405–419

1 3

views of practitioners (Polk 2014; Zscheischler and Rogga 2015).

As large environmental research programmes require in their funding schemes the integration of different scientific disciplines or practitioners, an increasing body of literature has emphasised the evaluation of the procedures and the effects of joint knowledge production between scientists of different disciplines and/or between scientists and practition-ers (e.g. Rossi et al. 2003; Stokols et al. 2003; Bergmann et al. 2005; Enengel et al. 2012; De Jong et al. 2016). A number of challenges come along with these types of coop-eration and their evaluation. For a start, participants from different backgrounds tend to have different expectations for the joint project and, consequently, follow different objec-tives, use different criteria to assess the project’s success, and are interested in different results and products (Klein 2008; Edelenbos et al. 2011). As a consequence, researchers tend to collaborate with colleagues from similar disciplines, even if they are supposed to work in interdisciplinary team-science projects (Stokols et al. 2008). Another important challenge is the fact that both social and cognitive factors influence the collaboration. It has been found that intensive communication between the project participants and oppor-tunities for informal exchange support mutual understand-ing. Having a variety of products and involving all parties in writing the project proposal from the beginning enhance the chances that a collaborative project will have a satisfac-tory outcome for all participants (Klein 2008; Gaziulusoy et al. 2016; De Jong et al. 2016; Hessels et al. 2014). Several studies (e.g. Blackstock and Carter 2007; Talwar et al. 2011; Lang et al. 2012) have noted that the practitioners are often insufficiently interested, committed, or willing to participate in TD projects, due to or resulting in unbalanced problem ownership. TD research has, thus, as a prerequisite to ensure good team building and communicating, so that a feeling of joint problem ownership can be established or sustained among the project participants.

What specific kind of knowledge can be created in TD research projects? Summarizing the results of different eval-uation studies of TD projects, it can be concluded that the most important impact is in influencing the project partici-pants’ ways of thinking. Here, we use Fleck’s (1979) concept of thought-styles (Cohen and Schnelle 1986; Pohl 2011), which says that each scientific discipline and each field of policy and practice has developed its particular culture of thinking, i.e. its specific thought-style, according to specific norms, values and attitudes. We adopt this concept for the analysis of our TD process because it explains a variety of potentials and challenges in communication and cooperation between experts from different working fields.

Bringing together different thought-styles and having them interact supports societal debate and reflection and enhances the participants’ capacity for decision-making

(Walter et al. 2007; Pohl 2011; Wiek et al. 2014; Polk 2014; Hegger and Dieperink 2015; Luederitz et al. 2016). Capac-ity building in decision-making occurs in TD research according to Pohl (2011) because the participants collec-tively enhance their own thought-styles when interacting with different thought-styles, by broadening or changing their understanding of an issue and developing new ways of handling the issue in their own real worlds. Such increases in decision-making capacity are not necessarily an imme-diate consequence of participatory events, but are induced by a number of intermediate effects, since the exchange of different thought-styles can influence the knowledge, atti-tudes and behaviour of the participants (Walter et al. 2007; Wiek et al. 2014; Hegger and Dieperink 2015). The authors mention examples such as ‘generating attention and enthusi-asm’, ‘community identification’, and ‘trust in others’. These intermediate effects are also seen as merits, added values or social impacts of TD processes (ibid.).

In this paper, we assessed the effects of a TD research pro-gramme on the participants’ ways of thinking. The research programme had produced scenarios and steering approaches for the future of urban and landscape development in four case study regions in Switzerland. This empirical study is intended to help fill the gap in research on the interaction of different thought-styles in TD projects. We do not present a comprehensive evaluation of all aspects of a TD research process, but rather focus on the participants’ evaluations of how the TD process in this research programme influenced their ways of thinking. Our TD research followed four objec-tives derived of from Pohl (2011), Lang et al. (2012) and Polk (2014):

1. To generate a feeling of joint problem ownership among the project participants.

2. To facilitate the interaction between stakeholders with different thought-styles, i.e. between researchers of dif-ferent disciplines, or practitioners with different profes-sional backgrounds, as well as between scientists and practitioners.

3. To enable the project participants to link abstract (sci-entific) with case-specific (practical) knowledge.

4. To encourage the project participants to incorporate the shared knowledge in their real-world situations.

The evaluation included: (1) checking how far our objec-tives for the TD research had been fulfilled; (2) assessing the intermediate effects of the TD process on the partici-pants’ ways of thinking; and (3) identifying obstacles and challenges in the TD process. We chose a combination of quantitative and qualitative approaches for the evaluation at the end of the research programme using standardised questionnaires on the one hand, and structured and open dis-cussions as well as our own observations on the other hand.

407Sustainability Science (2019) 14:405–419

1 3

Materials and methods

Objectives and structure of the research programme

The research programme ‘Room for People and Nature’ of the Swiss Federal Research Institute WSL ran from 2011 to 2016 and focused on the future landscape development in regions undergoing strong urbanisation pressure. Four rural regions in the sphere of influence of the city-region of Zurich (Switzerland) were selected for the case studies. The regions struggle with the problem of how to develop their villages while safeguarding specific landscape fea-tures of the regions, which are the basis for the regions’ high quality of life. The core objectives of the research programme were to create and evaluate scenarios for the future development of the landscapes in the case study regions, and to derive steering approaches for sustainable landscape development there.

The programme consisted of four natural and social science research projects and the synthesis project, in which the findings of the research projects were compiled and interpreted in the context of the case study regions. The approach was only transdisciplinary in the synthesis project, which was co-designed by researchers and prac-titioners. The scientists selected the main research ques-tions of the other four projects, but the practitioners gave specific inputs to make the projects relevant for the case study regions. This structure enabled the scientists in the individual projects to fulfil the requirements of scientific rigour in their specific communities, and at the same time to cooperate effectively in the TD parts of the research programme.

Participants and their involvement in TD cooperation

Three groups participated in the study: 19 scientists, 10 funding partners (practitioners) and approx. 35 local stake-holders in the case study regions. All the scientists were from the Federal Research Institute WSL but with different research fields: landscape ecology, biodiversity, social sci-ences of human–landscape relations, and governance stud-ies. The programme leader and two scientists experienced in TD research were responsible for the programme syn-thesis and facilitated the TD process. The other scientists took part in the TD process as participants of workshops and meetings.

The funding partners were representatives of cantonal and national spatial planning and environment protection authorities. The case study regions were located in the cantons, which provided funding. The funding partners

were involved in the programme design, implementation and evaluation. TD collaboration between the funding partners and the scientists was mainly in managing aspects of the programme.

The local stakeholders, who participated in each case study region included local politicians, such as mayors or municipal council members as well as representatives of NGOs, such as nature, landscape or bird protection asso-ciations. They were mostly key figures in local politics concerned with urban and landscape development. The municipal politicians were responsible for the zoning and development plans of their municipalities, while the NGOs were particularly interested in contributing to good man-agement of the open landscape. The local stakeholders participated in two workshops in each case study region, and the drop-out rate was surprisingly small. In the first workshop series, they developed a common vision about the future development of the landscape in their specific region. To support the workshops, real-time visualisations were produced (Tobias et al. 2016a; see also Fig. 1). In the second workshop series, the stakeholders developed steering approaches for sustainable landscape development. Some of the funding partners also participated in the workshops with the local stakeholders and thus experienced an additional TD element of the research programme.

TD work was carried out at three levels adapted from Stauffacher et al. (2008) and Schneider and Buser (2018): (1) information, where the researchers informed the prac-titioners; (2) consultation, where the researchers obtained information from the practitioners and vice-versa; and (3) coproduction, where the researchers and the practitioners or different practitioners together developed new knowledge (Fig. 2).

The TD events can be divided into the typical phases of a TD process according to Lang et al. (2012): (1) problem framing and team building; (2) co-creation of solution-ori-ented transferable knowledge; and (3) (re-)integration and application of the created knowledge. The researchers’ and the funding partners’ joint elaboration of the programme’s objectives and research questions was the key element in problem framing and team building (Fig. 2). The explora-tive, guided face-to-face, interviews in the case study regions were the basis for the region-specific problem framing and team building between the programme leader and key local stakeholders.

Typical elements of the phase of co-creation of knowl-edge were the workshops in the study regions. At these workshops, the three scientists responsible for the pro-gramme synthesis facilitated the reflection- and decision-making process for the local stakeholders and provided information from the research projects. The local stakehold-ers actually generated new knowledge together, taking into account research findings.

408 Sustainability Science (2019) 14:405–419

1 3

The process of co-creating knowledge between the researchers and the funding partners was less direct because the researchers worked on their research projects separately and informed the funding partners about their progress. The funding partners provided the researchers with information about the features of the case study regions and the needs of the local stakeholders. Knowledge exchange between these groups of participants took place during field trips to the case study regions, the yearly meetings, the presentation of the visuals, and the conference on the results of the indi-vidual research projects (Fig. 2). The field trips were rather unusual settings for the meetings of the funding partners

and the researchers. We organised bus trips through the case study regions with stops at certain places where selected local stakeholders informed the researchers and funding partners about the urban and landscape development in their regions. After the bus trips, the official meetings between the scientists and the funding partners took place in meeting rooms at local restaurants in the case study regions. The field trips offered the opportunity for a more informal exchange between the researchers and the funding partners, and for them to have direct contact with some local stakeholders.

The events of the final phase of integration and applica-tion of the created knowledge were the evaluation of the

Fig. 1 Examples of the visuals showing visions of future landscape development in the case study regions. The visuals were created during the workshops by scientific illustrators guided by the local stakeholders

409Sustainability Science (2019) 14:405–419

1 3

research programme by the funding partners and researchers, and the information events in the case study regions for the local stakeholders. In this phase, the scientists and funding partners also jointly produced a leaflet (Tobias et al. 2016b), which synthesized the most important issues for practice based on the findings from the research projects and the knowledge generated at the workshops with the local stake-holders. Further products of the research programme were a poster visualising the key points of the leaflet, a synthesis report addressing the funding partners and other practition-ers (Tobias 2015), six articles in magazines for practition-ers, and eight scientific papers for the different scientific communities.

Evaluation procedure for the TD process

We asked each group of participants (scientists, funding partners and local stakeholders) to evaluate the TD process of the research programme at two different times: the local stakeholders filled in a standardised questionnaire immedi-ately after the second workshop in their specific case study regions, while the researchers and funding partners com-pleted an online questionnaire after the last joint meeting.

The qualitative evaluation was conducted with the researchers and funding partners at the last joint meeting using a simplified version of the ‘most significant Change’ (MSC) method according to Davies and Dart (2005). In the MSC method, the participants tell stories about moments, actions or processes that induced a significant change in their way of thinking or doing, and a panel of selected stakeholders or staff selects the most significant of these change stories. At our evaluation workshop, we asked the

researchers and funding partners to write down on a sheet of paper short sentences or key words about what they per-sonally experienced as most important steps or moments that advanced or influenced the research programme sig-nificantly. Afterwards, the participants exchanged their experiences in small groups of three or four people, including both researchers and funding partners. In the subsequent plenary discussion, the groups exchanged their consolidated experiences. The plenary discussion was tape-recorded and transcribed. We analysed the discus-sion transcript and the participants’ notes in a qualitative content analysis (Schreier 2013; Mayring 2014), and com-pared our results to findings from the literature. The MSC method was used to obtain information about intermediate effects and challenges of this TD research on the partici-pants’ ways of thinking. Our emphasis was on the events and the levels of TD cooperation shown in Fig. 2. Moreo-ver, we looked for evidence about the achievement of the objectives of TD research described in the introduction. Our own observations were also fed into the evaluation.

The questionnaires were completed by 26 local stake-holders, but not all respondents answered every question. 12 researchers and 4 funding partners participated in the online survey. For the MSC method, 7 researchers and 5 funding partners were present. As the samples were so small, we did not perform detailed statistical analyses and only ascertained the distributions of the answers to the statements in the questionnaires.

Evaluation framework

From the literature, we derived four objectives for our TD research (cf. “Introduction” and Table 1) and defined specific criteria for each objective as indicators of suc-cess. These then provided the bases for the questions in the questionnaires for the programme participants (Table 1).

For each specific criterion, we formulated one or two statements the interviewees had to respond to on a scale of 1–5 (1 = disagree; 5 = fully agree), with separate standard-ised questionnaires for each group, the researchers, fund-ing partners and local stakeholders. All questionnaires had the same structure, but the individual questions were formulated to take into account the specific perspective of each participant group, resulting in different question-naires for each group. The single statements for each par-ticipant group are listed in the electronic supplementary material S1.

Fig. 2 The TD elements of the research programme assigned to the three levels of TD cooperation. The box colours indicate which agents were mainly involved in the events, with two groups rep-resented in each case. All events were organised by the scientists responsible for the research programme synthesis

410 Sustainability Science (2019) 14:405–419

1 3

Results

Questionnaire results

Objective 1

The first objective was to generate a feeling of joint prob-lem ownership among the project participants. In general, it seems to have been at least partially met (Fig. 3). The median values for the non-academics, i.e. the local stake-holders and the funding partners, were slightly more positive than those of the researchers, particularly if they were asked about the whole group as joint problem owners (Fig. 3b). The questionnaire also included the question whether the respondents had missed the inclusion of important stake-holder groups. Although most answers were negative, some participants mentioned representatives from agriculture, fol-lowed by representatives of industry and trade, investors and landowners, should have been included.

Objective 2

The second objective was to facilitate the interaction between stakeholders with different thought-styles, i.e. between researchers from different disciplines, or practi-tioners with different professional backgrounds, as well as between scientists and practitioners. How much respond-ents valued the interaction with other thought-styles differed between the participants (Fig. 4). According to the median values, the local stakeholders were those who thought they had benefited most from the exchange with other thought-styles, i.e. from discussions with local representatives of

Table 1 Objectives of our TD research derived from the literature and the specific criteria we defined as the basis for the standardised question-naires

a Adapted from Pohl (2011), Lang et al. (2012), Polk (2014)

Objectives of the TD process in the research programmea Criteria specifying the objectives

1. To achieve a feeling of joint problem ownership among the project participants

All group members’ knowledge is considered importantThe whole group is considered to be the collective problem-owner

2. To facilitate the interaction between stakeholders with different thought-styles

New perspectives/ideas are developed due to the thought-styles of other group members

Joint products are developed3. To enable the project participants to link abstract (scientific) with

case-specific (practical) knowledgeExperiences with other knowledge types (both scientific and practical)

are integrated in the real world (working environments)New interfaces between the different types of knowledge (both scientific

and practical) are discovered4. To encourage the project participants to incorporate the shared

knowledge in their real-world situationsThe participants are motivated to disseminate the jointly developed

knowledge in their real worlds (working environments)Ideas are generated for new approaches/activities in the participants’

own real worlds (working environments)Ideas are developed for new cooperations between groups that have not

yet worked together (in this way)

Fig. 3 TD objective 1: to generate a feeling of joint problem owner-ship among the project participants. Research programme partici-pants’ assessments of specific statements. Median values: funding partners 4 (a) and 4.5 (b); researchers 4 (a, b); local stakeholders 4 (a) and 5 (b). N number of respondents

411Sustainability Science (2019) 14:405–419

1 3

other disciplines or policy sectors (Fig. 4a). They were also the most likely to say they had developed joint products during the workshops in the case study regions (Fig. 4b). The funding partners tended to think they had developed new perceptions and ideas through exchanging with the researchers’ thought-styles (median of 4 in Fig. 4a), but were less enthusiastic about the development of joint products (median of 3 in Fig. 4b). The researchers were the most sceptical. They felt they had benefited somewhat from the exchange with the funding partners (median of 4 in Fig. 4a), but they were critical about the development of joint prod-ucts (median of 2.5 in Fig. 4b).

Objective 3

The third objective was to enable the project participants to link abstract (scientific) with case-specific (practical) knowledge. The responses to the questions concerning this objective were rather evenly distributed across the groups (Fig. 5). This indicates that the results depend more on the individual perception of each participant than on differences between the participant groups. The results for the funding partners may suggest more consensus (Fig. 5b), but clustered

around the indifferent level. Only four of the funding part-ners, however, answered the questionnaire. According to the median values, the local stakeholders agreed most with both statements, whereas the researchers were as indifferent as the funding partners.

Objective 4

The fourth objective was to encourage the project partici-pants to incorporate the shared knowledge in their real-world situations. The local stakeholders were far more positive than the other two groups in terms of their motivation to disseminate the new knowledge in their working environ-ment (Fig. 6a), and in terms of ideas considering new coop-erations (Fig. 6c), but they were largely indifferent about introducing new approaches at work (Fig. 6b). Conversely, the researchers and the funding partners were more posi-tive about obtaining ideas for new approaches or activities in their working environments (Fig. 6b) than about feel-ing motivated to disseminate the knowledge or form new cooperations.

The respondents who tended to agree with the statement in Fig. 6c were additionally asked to indicate with whom

Fig. 4 TD objective 2: to facilitate the interaction between stakehold-ers with different thought-styles. Research programme participants’ assessments of specific statements. Median values: funding partners 4 (a) and 3 (b); researchers 4 (a) and 2.5 (b); local stakeholders 4.5 (a) and 4 (b). N number of respondents

Fig. 5 TD objective 3: to enable the project participants to link abstract (scientific) with case-specific (practical) knowledge. Research programme participants’ assessments of specific statements. Median values: funding partners 3.5 (a) and 3 (b); researchers 3.5 (a) and 3 (b); local stakeholders 4 (a, b). N number of respondents

412 Sustainability Science (2019) 14:405–419

1 3

they would like to establish a new or intensified coopera-tion. The local stakeholders mentioned NGOs such as nature protection organisations, local trade associations or local ini-tiatives for sustainable regional development. They further listed regional governmental organisations such as commis-sions for agriculture, regional planning, and landscape pro-tection. The funding partners mentioned authorities within their own or other political departments like the cantonal transport office. The researchers thought of looking for more

scientific contacts with other national and international uni-versities, and for more contacts with the authorities respon-sible for recreation infrastructure.

Summary of the notes and discussion on the most significant changes

In both the written notes and the plenary discussion, direct exchange between research and practice in the case study regions was cited as the most valuable. The funding partners who attended the workshops with the local stakeholders in the case study regions said they personally found them the most valuable during the research programme. In the discus-sion, they explained that, thanks to the scientific context, these workshops provided conditions for building up mutual trust, where the local stakeholders from the different interest groups could freely discuss issues without being in the con-text of municipal politics. Several funding partners greatly appreciated the way the researchers managed to organise meetings where local politics played little role.

The field trips to the case study regions also provided opportunities for science–practice interaction, which all participants appreciated. Both the funding partners and the researchers were enthusiastic about the field trips and found the discussions on-site with representatives of the case study regions very fruitful. Some of the participants mentioned that the field trips were opportunities for team-building between researchers and funding partners. The participants appreciated the way the dialogue between research and prac-tice had been implemented, and said it had fostered mutual learning and trust.

The visuals were seen as playing important role in knowl-edge exchange. The researchers perceived them as a kind of report about the workshops with the local stakeholders. The funding partners considered them to be both the bases for the cooperation among the local stakeholders as well as the products of the cooperation. All said the visuals served as bridging elements between research and practice. Both the researchers and the funding partners remarked in their writ-ten notes that co-creating the leaflet as a joint product was also an important experience.

The researchers and the funding partners reported in retrospect how important the framing phase had been for the research programme. They mentioned, in particular, the workshop on defining the research questions as a crucial event, albeit most of them were not really satisfied with it. The funding partners generally felt they had missed the opportunity to actively design the research questions. Some of them were of the opinion that the direct benefits of the research for practice were necessarily limited because the research topics and methods were too heterogeneous with an unbalanced mix of detailed and strategic perspectives. They thought it was difficult for the practitioners to define

Fig. 6 TD objective 4: to encourage the project participants to incor-porate the shared knowledge in their real world situations. Research programme participants’ assessments of specific statements. Median values: funding partners 3.5 (a), 4 (b) and 2.5 (c); researchers 3 (a), 4 (b) and 3 (c); local stakeholders 4 (a), 3 (b) and 4 (c). N number of respondents

413Sustainability Science (2019) 14:405–419

1 3

their research needs. The researchers, in turn, expressed their disappointment that the funding partners had provided them with too little tangible information about their specific needs.

Several participants complained that not all interest groups from practice had been present at all the meetings. The funding partners felt that the cantonal authorities had been over-represented, whereas there were too few repre-sentatives of the federal and municipal offices even though they had also contributed to financing the research pro-gramme. They interpreted this absence as lack of commit-ment. The researchers, on the other hand, tended to miss rep-resentatives from agriculture and the private economy. One practitioner, however, countered that he had experienced the discussions to be freer and more open than at events where the private economy was strongly represented. The research-ers said they were pleasantly surprised by the strong com-mitment of the practitioners who participated regularly in the research programme.

Finally, at the end of the plenary discussion and in their written notes, the participants indicated some lessons learned from developing shared knowledge. They felt hav-ing found colleagues from other real worlds, what convinced them about the relevance of urban and landscape develop-ment and of their own contributions to this issue. The fund-ing partners said the discussions and the documents (leaflet and synthesis report) provided them with substantial argu-ments for their own work. They encouraged the researchers to continue with this kind of landscape research in other regions. Some researchers reported that they had discovered connections between their research fields and problems in practice that they had not been aware of before, e.g. “land-scape research incorporates planning”.

The statements indicate a number of intermediate effects of this TD research that induced changes in the participants’ ways of thinking. Similar changes have been reported in the literature. They are compiled in Table 2 and discussed in “Discussion”.

Initial effects of the research programme in the case study regions

In addition to the feedback in the written questionnaires, the local stakeholders commented informally on the workshops in the case study regions over food and drink at the end of the event. They particularly appreciated the composition of experts and the holistic way of discussing the topic of urban and landscape development. They enjoyed the joint meetings with local politicians, cantonal authorities and representa-tives of landscape, nature and bird protection associations. They liked being able to discuss freely outside a clear politi-cal context and the way the workshops were embedded in a scientific programme, which, they felt, made the workshop

results more relevant. Some participants said they had told others at work about the new ideas they had encountered at the workshops.

After the second workshop series, two municipal mayors invited the authors to the retreat sessions of their municipal councils in two case study regions. We were asked to repeat our presentations of the workshops as an input to the sub-sequent discussions. With this external input, the mayors wanted to encourage their councils to adopt a more holistic and open-minded approach to the issues. After the end of the research programme, we were also invited to give a talk at a public event organised by the local trade association in a case study region.

Finally, we were surprised about the enthusiasm of the local stakeholders at the closing information events in the case study regions, which had originally been planned as a way to thank the participants for having attended the two workshops, and to promote the leaflet and the poster as the key products for practice. Almost all workshop participants made sure that they could come to the information events and some of them even brought a colleague. In two case study regions, we also invited the local press at the sugges-tion of some stakeholders. We were particularly surprised how motivated the participants were to spread the new ideas among other key players in the case study regions. This became apparent during the discussions, where they almost mandated one another to promote the lessons learned about landscape development among their expert colleagues.

Discussion

Achievement of TD objectives

All participant groups, the local stakeholders, the funding partners, and the researchers were rather positive about having reached the objective of achieving a feeling of joint problem ownership (Fig. 3). In the course of the regular meetings, the teams created an atmosphere of mutual trust, and the great commitment of the practitioners sometimes even surprised the researchers (cf. “Summary of the notes and discussion on the most significant changes”). We also, however, faced the challenge of unbalanced problem own-ership that other researchers have found (Blackstock and Carter 2007; Talwar et al. 2011; Lang et al. 2012). The par-ticipants were more bothered by the lack of commitment or discontinuous participation of a few project participants than by the fact that not all interest groups could be invited. Making a financial commitment is obviously not sufficient to ensure participation in a TD process. The local stakeholders and the funding partners from the cantons where the case study regions are located may have developed a strong feel-ing of joint problem ownership because the research was on

414 Sustainability Science (2019) 14:405–419

1 3

Tabl

e 2

Par

ticip

ants’

sta

tem

ents

in th

e w

ritte

n no

tes

and

the

disc

ussi

on a

bout

the

mos

t sig

nific

ant c

hang

es (1

st co

lum

n), a

nd o

ur in

terp

reta

tion

in te

rms

of th

e in

term

edia

te e

ffect

s of

TD

(TD

) re

sear

ch (2

nd c

olum

n) c

ompa

red

to fi

ndin

gs fr

om th

e lit

erat

ure

(3rd

col

umn)

with

the

corr

espo

ndin

g re

fere

nces

(4th

col

umn)

(1) S

tate

men

ts o

f res

earc

hers

and

fund

ing

partn

ers

(2) I

nter

med

iate

effe

cts a

nd c

halle

nges

of o

ur

TD re

sear

ch (o

wn

inte

rpre

tatio

ns)

(3) C

orre

spon

ding

find

ings

from

the

liter

atur

e(4

) Sou

rces

of fi

ndin

gs in

col

umn

3

The

wor

ksho

ps w

ith th

e lo

cal s

take

hold

ers i

n th

e ca

se st

udy

regi

ons b

uilt

a pl

atfo

rm fo

r m

utua

l lea

rnin

g ou

tsid

e th

e co

ntex

t of l

ocal

po

litic

s

Safe

spac

es fa

cilit

ated

new

hyb

rid c

omm

uniti

es

of re

sear

cher

s and

non

-aca

dem

ics

Trus

t and

ent

husi

asm

wer

e pr

omot

ed

Esta

blis

hing

a h

ybrid

com

mun

ity o

f res

earc

h an

d pr

actic

eN

etw

ork

build

ing;

com

mun

ity id

entifi

catio

n;

sens

e of

bel

ongi

ngTr

ust i

n ot

hers

Heg

ger a

nd D

iepe

rink

(201

5), P

olk

(201

4)W

alte

r et a

l. (2

007)

, Wie

k et

 al.

(201

4)

The

field

trip

s to

the

case

stud

y re

gion

s and

the

on-s

ite d

iscu

ssio

ns w

ith re

pres

enta

tives

of t

he

loca

l sta

keho

lder

s wer

e th

e m

ost v

alua

ble

for

rese

arch

ers a

nd fu

ndin

g pa

rtner

s

Imm

ersi

on in

the

real

wor

ld o

f oth

er st

akeh

old-

ers e

nhan

ces r

eflex

ivity

am

ong

parti

cipa

nts

Enth

usia

sm w

as g

ener

ated

Mor

e re

flexi

vity

am

ong

parti

cipa

nts

Gen

erat

ing

atte

ntio

n an

d en

thus

iasm

Heg

ger a

nd D

iepe

rink

(201

5)W

iek

et a

l. (2

014)

The

dial

ogue

bet

wee

n re

sear

ch a

nd p

ract

ice

was

ap

prec

iate

d as

resp

ectfu

l and

instr

uctiv

eC

aref

ul c

omm

unic

atio

n fo

sters

trus

tN

etw

ork

build

ing

Trus

t in

othe

rsW

alte

r et a

l. (2

007)

, Wie

k et

 al.

(201

4)

The

visu

alis

atio

ns se

rved

as b

ridgi

ng e

lem

ents

be

twee

n re

sear

ch a

nd p

ract

ice

Bou

ndar

y ob

ject

s fac

ilita

te c

omm

unic

atio

n an

d co

oper

atio

nB

ound

ary

obje

cts

Car

lile

(200

2), B

riers

and

Chu

a (2

001)

, La

ng e

t al.

(201

2)C

oope

ratio

n on

the

leafl

et w

as in

struc

tive

for

both

the

rese

arch

ers a

nd th

e fu

ndin

g pa

rtner

sC

opro

duct

ion

and

join

t pro

duct

s fos

ter j

oint

pr

oble

m o

wne

rshi

p an

d pr

ovid

e bo

unda

ry

obje

cts

Com

mun

ity id

entifi

catio

nSe

nse

of b

elon

ging

Bou

ndar

y ob

ject

s

Wal

ter e

t al.

(200

7)B

riers

and

Chu

a (2

001)

, Lan

g et

 al.

(201

2)

The

prog

ram

me

fram

ing

phas

e w

as a

key

cha

l-le

nge.

Pra

ctiti

oner

s mis

sed

havi

ng a

n op

por-

tuni

ty to

be

activ

ely

invo

lved

in d

esig

ning

the

rese

arch

pro

gram

me

and

wer

e ov

erw

helm

ed

by th

e he

tero

gene

ity o

f the

(dis

cipl

inar

y)

rese

arch

app

roac

hes

Proj

ect f

ram

ing

phas

e: “

cold

boo

ting”

for t

he

inte

ract

ion

of th

ough

t-sty

les

Pred

omin

ance

of s

cien

tific

inte

ntio

nsU

nbal

ance

d pr

oble

m o

wne

rshi

p at

this

tim

e

Pred

omin

ance

of s

cien

tific

cons

ider

atio

nsU

nbal

ance

d pr

oble

m o

wne

rshi

pW

uels

er a

nd P

ohl (

2016

)La

ng e

t al.

(201

2)

Rese

arch

ers a

nd p

ract

ition

ers w

ould

hav

e lik

ed

mor

e co

mm

itmen

t of f

undi

ng p

artn

ers f

rom

fe

dera

l offi

ces.

Rese

arch

ers w

ould

hav

e ap

pre-

ciat

ed m

ore

non-

acad

emic

repr

esen

tativ

es o

f ag

ricul

ture

and

the

priv

ate

econ

omy

Unb

alan

ced

invo

lvem

ent/c

omm

itmen

t of i

nter

-es

t gro

ups c

halle

nges

the

feel

ing

of c

omm

on

prob

lem

ow

ners

hip

Unb

alan

ced

invo

lvem

ent/c

omm

itmen

t of i

nter

-es

t gro

ups

Bla

cksto

ck a

nd C

arte

r (20

07),

Talw

ar e

t al.

(201

1), L

ang

et a

l. (2

012)

Rese

arch

ers a

nd n

on-a

cade

mic

s gai

ned

new

kn

owle

dge

and

wer

e m

otiv

ated

to d

isse

min

ate

it in

thei

r ow

n re

al w

orld

s

Parti

cipa

nts d

istrib

ute

the

shar

ed k

now

ledg

e in

th

eir r

eal w

orld

sPa

rtici

pant

s dev

elop

idea

s to

impl

emen

t the

ne

w k

now

ledg

e in

thei

r rea

l wor

lds

Dist

ribut

ion

of k

now

ledg

eD

evel

opin

g tra

nsfo

rmat

ion

know

ledg

eW

alte

r et a

l. (2

007)

415Sustainability Science (2019) 14:405–419

1 3

a concrete issue in the region. These regions did not, how-ever, come directly under the responsibility of the funding partners from the national offices, which may be why the national officers only attended a few of the joint meetings. For the others, a certain feeling of joint problem ownership was probably already around at the beginning of the coop-eration due to the actual research topic, and this feeling was reinforced during the TD process.

The meetings were rather successful in facilitating the interaction of different thought-styles, which helped, in particular, to induce new perceptions and ideas among the participants (Fig. 4). The findings indicate the effective-ness of the different levels of TD cooperation (Fig. 2). TD events involving the coproduction of knowledge or at least intensive consultation with two-way communication appear to be necessary to trigger the interaction of thought-styles. In the questionnaire results (Fig. 4), the local stakeholders, who had attended the workshops in the case study regions, were more positive than the other participant groups. Con-versely, at the ‘most significant change’ workshop, the fund-ing partners and researchers said that the field trips to the case study regions were what had made most impression on them. The field trips enabled the researchers and funding partners to exchange with local stakeholders from the case study regions. For the researchers, this was an interaction between science and practice. For the funding partners, it was an exchange between different governmental levels.

The most striking effect the interaction of thought-styles had for all participants seems to have been becoming aware of people in other real worlds committed to the same issues, i.e. the awareness of joint problem ownership. This made the participants look at the issue from new perspectives, as one researcher at the evaluation workshop put it: “landscape research incorporates planning” (cf. “Summary of the notes and discussion on the most significant changes”). His views shifted from a large landscape ecological perspective to include a spatial planning perspective.

Considering the third objective, the participants did not really feel enabled to link scientific and practical knowledge (Fig. 5). The local stakeholders made the most attempts to integrate the experiences they had with the scientists into their own real worlds (Fig. 5a). This became evident when, for example, they invited us to attend their municipal council retreats. Some funding partners also mentioned that the sci-entific knowledge documented in the leaflet and the synthe-sis report supported their reasoning in their practical work (“ Summary of the notes and discussion on the most signifi-cant changes”). The researchers’ responses to the question-naire varied most. At the final workshop, some researchers said they would have liked more direct exchange with the local stakeholders.

The participants’ motivation to incorporate the shared knowledge in their real worlds (objective 4) became

particularly clear in the statements and actions of the local stakeholders after the workshops (“Initial effects of the research programme in the case study regions”), and in the statements of the researchers and funding partners at the evaluation workshop (“Summary of the notes and discussion on the most significant changes”). The awareness of joint problem owners with other thought-styles encouraged the local stakeholders to work more closely together with other interest groups on their particular political tasks. Conversely, the feeling of having found colleagues from other real worlds encouraged the researchers and funding partners to develop new ideas for their work in their own real worlds.

Intermediate effects and challenges of the TD process

The biggest success factors in this research programme were the field trips to the case study regions and the workshops there because they offered opportunities to experience the real worlds of other stakeholder groups and to experience an “agora”-like context for science and practice. Participants at these events were enthusiastic and able to build up a mutual trust (Table 2).

The effects of the workshops in the case study regions were similar to those in sustainability transition experiments described by Luederitz et al. (2016). As in such experiments, the participants enhanced their decision-making capacities, particularly for incorporating the new knowledge in their own real worlds, what became evident at the final informa-tion events when they mandated one another to spread the lessons learned. The use of visuals may have additionally triggered the participants’ confidence and commitment, as participants are particularly committed in transition experi-ments trying novel practices (Wittmayer ànd Schäpke 2014; Luederitz et al. 2016).

Coproduction of knowledge appears to be particularly effective if joint products are generated that can be used as boundary objects by different stakeholder groups. In our case, the visuals played the role of boundary objects in two ways: (1) as boundary objects for the panels of local stake-holders at the workshops in the case study regions; and (2) as bridging elements for the researchers and funding partners with the local stakeholders. The visuals were ideal boundary objects between the different thought-styles because they met the prerequisites Carlile (2002) identified for boundary objects in organisational development: (1) enable a shared language; (2) serve as a means to learn about the differences and dependencies across the boundaries between different thought-styles; and (3) facilitate the process of conjoint knowledge transformation. Briers and Chua (2001) further noted that visionary boundary objects like joint principles or rules of the game were more efficient than joint data reposi-tories. As our visuals showed ideas and scenarios for future

416 Sustainability Science (2019) 14:405–419

1 3

landscape development, they also had the characteristics of visionary boundary objects.

In the research programme, the existence or coproduc-tion of boundary objects, together with opportunities for generating enthusiasm and trust, fostered among the par-ticipants the feeling of joint problem ownership. Together, they developed a feeling of companionship and belonging to the research, which Walter et al. (2007) also found to be an important intermediate effect of TD research. This feeling of companionship increased the participants’ awareness of the relevance of urban and landscape development and, in turn, they acknowledged the necessary legitimacy of the results of their TD cooperation. Di Iacovo et al. (2016) made simi-lar observations in a TD social farming project, while Lang et al. (2012) identified a lack of legitimacy of TD findings as one of the main impediments for implementing shared knowledge in the real world. Reed et al. (2014) also stress the importance of including stakeholders with decision-mak-ing powers and trusted individuals as knowledge brokers. In our case, the legitimacy of the joint products was not only supported by the positions of the participants in municipal politics or as representing national or cantonal authorities, but also by the exchange of science and practice.

The researchers and funding partners mentioned the project-framing workshop as one of the key challenges of the research programme (cf. “Summary of the notes and discussion on the most significant changes”; Table 2). We assume that, at the time when the workshop took place, both the researchers and the practitioners were unable to cope with the way of thinking of the other agent-group and experienced the workshop as kind of “cold booting”. At this time, all participants saw the problem from their own spe-cific perspectives and were, therefore, not able to capture its complexity. Although they had the opportunity to interact with different thought-styles, they were, at this early stage of the research programme, not able to come to terms with the impressions from the other thought-styles. This is again a typical challenge in TD research, as Wiek (2007), Zeischler et al. (2014) and Di Iacovo et al. (2016) have observed in other TD projects as well. At the beginning of most TD projects, the project leaders and participants put too little emphasis on the very different obligations of scientists and practitioners. This often results in a predominance of scien-tific considerations (Wuelser and Pohl 2016).

The research programme’s structure may have supported an initially more researcher-dominated situation, as it con-sisted of several individual, not TD research projects con-tributing to a TD synthesis (cf. “Objectives and structure of the research programme”). The project framing work-shop was actually intended not only to promote the research ideas of the individual projects to the practitioners, but also to collaboratively develop the research questions of the programme synthesis and modifications of the individual

projects. This structure of the research programme was not just a challenge during the framing phase, as some research-ers also complained that they had received very little infor-mation about the work in the case study regions. Therefore, the field trips to the case study regions were crucial events for the scientists to experience the context and the real worlds of the beneficiaries of their research. In addition, the researchers appreciated the visuals as bridging elements to the case study regions. On the other hand, the structure of the research programme allowed us to work parallel in four case study regions without overloading the research-ers schedules, and it enabled a variety of publications. In contrast to other TD projects (Hegger and Dieperink 2015), this research programme did not result in fewer scientific publications than five individual projects would have done. Simultaneously, we could benefit from opportunities for coproduction and published more articles for practice than five single projects.

All participants were enthusiastic about the lessons they had learned for their own real worlds. The process of the discussion indicated that it was not only a matter of provid-ing useful information but also being able to share enthusi-asm and take pleasure in the learning process. Wiek et al. (2014) also observed in participatory sustainability research that enthusiasm among the project participants supported building their decision-making capacities. The motivation to incorporate the shared knowledge into their real worlds is closely connected with the feeling of joint problem own-ership. The awareness that agents in other real worlds are committed to the same issue was very motivating for all par-ticipants. In addition, adopting the perspective of the other stakeholders on the joint problem enhanced the relevance of the issue and their own contributions.

Our findings confirm the generation of hybrid spaces and communities as others have found in TD projects (Polk 2014; Hegger and Dieperink 2015). The workshops with the local stakeholders in the case study regions were, in particu-lar, appreciated as opportunities for discussion in a neutral environment removed from local politics and disciplinary science, but at the same time embedded in a practical and scientific context. The funding partners and researchers had similar experiences during the field trips to the case study regions and in the workshops for defining the research ques-tions and evaluating the programme. The respective groups, i.e. the local stakeholders as well as the funding partners and researchers, built mixed teams, which developed new TD knowledge. These hybrid spaces were most important for the interaction of thought-styles and for trust building. As Polk (2014) points out, they were both a prerequisite and a prod-uct of the TD research. Although the participants were moti-vated to make use of the personal lessons they learned, these will not necessarily be directly implemented in either the urban and landscape development in the case study regions

417Sustainability Science (2019) 14:405–419

1 3

or in new research projects. The jointly created knowledge needs “translation” or transformation to make it compatible with existing institutional structures and decision-making processes in the specific scientific or practical realms (Wiek et al. 2012; Polk 2014).

Limitations

Our evaluation procedure has some serious limitations. First, the stakeholder groups differed in their sizes and were all rather small. This is why we conducted no statistical analy-ses and relied on combining the quantitative and qualita-tive data for our interpretations. Second, we conducted the evaluation immediately after the end of the workshops or the research programme to reach the maximum number of people possible. However, at this stage in the research programme only the first few effects on the participants’ real worlds had become apparent (cf. “Initial effects of the research programme in the case study regions”). At a later point in time, however, it is not necessarily easier to identify the influence of the TD process because many additional parameters can play a role in the participants’ real worlds. In addition, the participants might not be able to recall the situations, which Wiek et al. (2014) found to be a core chal-lenge in evaluating TD projects. Given these limitations, we compared our results to findings in the literature for the interpretation (Table 2).

Conclusions

In our evaluation, we combined a quantitative approach using targeted questions in questionnaires with a qualitative approach involving a simplified version of the ‘most signifi-cant change’ method, together with our own observations. The two approaches complement each other. The question-naire results gave an overall picture, although the number of participants was small. The qualitative evaluation revealed highlights, key challenges, and the atmosphere during the TD process. In particular, it allowed us to make recom-mendations for successfully organising TD processes. The exchange between different thought-styles should be in two-way communication and preferably at the TD level of copro-duction. Opportunities to experience the real worlds of oth-ers, e.g. on field trips, are a good way to foster participants’ enthusiasm for the project and build up mutual trust. These qualities (enthusiasm and mutual trust) appear to strongly support the interaction of different thought-styles. In addi-tion, boundary objects, such as our visuals, are very useful to facilitate the interaction of thought-styles. Boundary objects may be given as bridging elements between the different thought-styles or developed together as joint products.

In our case, the TD process was successful in the four aspects Belcher et al. (2016) identified for assessing the quality of TD research: (1) relevance, (2) credibility, (3) legitimacy, and (4) effectiveness. (1) The partici-pants became aware of additional problem owners (“col-leagues”) from other thought-styles and consequently were more convinced about the relevance of the issue. (2) The practitioners, in particular, found the results of the process credible because they considered the research-ers to be representatives of intelligence and objectivity. (3) The interaction of different thought-styles helped the participants adopt a broader view on the issue of urban and landscape development. These wider perspectives, together with a feeling of companionship, enhanced the legitimacy of the issues and (4) motivated the participants to incorporate the shared knowledge into their own real worlds.

The evaluation also revealed the importance of hybrid spaces and communities, as Polk (2014) and Hegger and Dieperink (2015) have observed in other TD projects. In such independent spaces, participants can experience enthusiasm and trust and together produce new knowledge. This new TD knowledge includes: (1) the awareness of problem owners with other thought-styles; (2) a mutual feeling of companionship; (3) a broadened view on the issue; (4) increased legitimacy of the issue; and (5) partici-pants motivated to tackle the issue with new approaches or with new cooperations. However, this TD knowledge tends to be tacit knowledge, and it will always be difficult to prove what changes in the participants’ real worlds it induces. In our case, the local stakeholders made some attempts to apply and spread the new knowledge in their real worlds, e.g. bringing it into the process of regional landscape development policies, and they invited us to follow-up presentations to reach further decision makers. But there is still a long way to go before visible changes in their land-use plans become apparent.

Acknowledgements The authors thank Silvia Dingwall for the English corrections. We are also grateful to the three anonymous reviewers for their valuable comments on a previous version of the manuscript. This work was part of the WSL research programme “Room for People and Nature”. We gratefully acknowledge financial support from the Federal Office for the Environment, the authorities of Cantons Aar-gau, Lucerne, St. Gallen, Zug, and Glarus, as well as the municipality of Glarus Nord. Tobias Buser is furthermore financed by the COST Action TD1408 INTREPID in project C16.0042 of the Swiss National Science Foundation. Funding was provided by Bundesamt für Umwelt (10.0036.KP/K345-1617).

Compliance with ethical standards

Conflict of interest The authors declare that they have no conflict of interest.

418 Sustainability Science (2019) 14:405–419

1 3

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecom-mons.org/licenses/by/4.0/), which permits unrestricted use, distribu-tion, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

References

Belcher BM, Rasmussen KE, Kemshaw MR, Zornes DA (2016) Defin-ing and assessing research quality in a TD context. Res Eval 25:1–17. https://doi.org/10.1093/reseval/rvv025

Bergmann M, Brohmann B, Hoffmann E, Loibl MC, Rehaag R, Sch-ramm E, Voss J-P (2005) Qualitätskriterien transdisziplinärer Forschung. ISOE Studientexte 13:76

Blackstock KL, Carter CE (2007) Operationalising sustainability sci-ence for a sustainability directive? Reflecting on there pilot pro-jects. Geogr J 173(4):343–357

Brandt P, Ernst A, Gralla F, Luederitz C, Lang DJ, Newig J, Reinert F, Abson DJ, von Wehrden H (2013) A review of TD research in sustainability science. Ecol Econ 92:1–15

Briers M, Chua WF (2001) The role of actor-networks and bound-ary objects in management accounting change: a field study of an implementation of activity-based costing. Account Org Soc 26:237–269

Carlile PR (2002) A pragmatic view of knowledge and bounda-ries: boundary objects in new product development. Org Sci 13:442–455

Cohen RS, Schnelle T (1986) Cognition and fact: materials on Ludwik Fleck. Reidel, Dordrecht, p 468

Davies R, Dart J (2005) The ‘most significant change’ (MSC) tech-nique. A guide to its use. Version 1.00, p 104. http://www.capaci-ty4dev.eu. Accessed 16 Feb 2017

De Jong SPL, Wardenaar T, Horlings E (2016) Exploring the promises of TD research: a quantitative study of two climate research pro-grammes. Res Policy 45:1397–1409

Di Iacovo F, Moruzzo R. Rossignoli CM, Scarpellini P (2016) Measur-ing the effects of TD research: the case of a social farming project. Futures 75:24–35

Edelenbos J, van Buuren A, van Schie N (2011) Co-producing knowl-edge: joint knowledge production between experts, bureaucrats and stakeholders in Dutch water management projects. Environ Sci Policy 14:67–684

Enengel B, Muhar A, Penker M, Freyer B, Drlik S, Ritter F (2012) Co-production of knowledge in TD doctoral theses on landscape development—an analysis of actor roles and knowledge types in different research phases. Landsc Urban Plan 105:1063–1117

Fleck L (1979) Genesis and development of a scientific fact. University of Chicago Press, Chicago

Gaziulusoy AI, Ryan C, McGrail S, Chandler P, Twomey P (2016) Identifying and addressing challenges faced by TD research teams in climate change research. J Clean Prod 123:55–64

Hegger D, Dieperink C (2015) Joint knowledge production for climate change adaptation: what is in it for science? Ecol Soc 20(4):1. https://doi.org/10.5751/ES-07929-200401

Hessels LK, Wardenaar T, Boon WPC, Ploeg M (2014) The role of knowledge users in public-private research programs: an evalua-tion challenge. Res Eval 23:103–116

Klein JT (2008) Evaluation of interdisciplinary and TD research. Am J Prev Med 35(2):S116–S123

Klein JT (2010) A taxonomy of interdisciplinarity. In: Frodeman R, Thompson Klein J, Mitcham C, Britt Holbrook J (eds) The Oxford handbook of interdisciplinarity. Oxford University Press, New York, pp 15–30

Lang DJ, Wiek A, Bergmann M, Stauffacher M, Martens P, Moll P, Swilling M, Thomas CJ (2012) TD research in sustainability sci-ence: practice, principles, and challenges. Sustain Sci 7:25–43

Luederitz C, Schäpke N, Wiek A, Lang DJ, Bergmann M, Bos JJ, Burch S, Davies A, Evans J, König A, Farrelly MA, Forrest N, Frantzeskaki N, Gibson RB, Kay B, Loorbach D, McCormick K, Parodi O, Rauschmayer F, Scheidewind U, Stauffacher M, Stelzer F, Trencher G, Venjakob J, Vergragt PJ, von Wehrden H, Westley FR (2016) Learning through evaluation—a tentative evaluative scheme for sustainability transition experiments. J Clean Prod. https://doi.org/10.1016/j.jclepro.2016.09.005

Mayring P (2014) Qualitative content analysis: theoretical foundation, basic procedures and software solution. Klagenfurt. http://nbn-resolving.de/urn:nbn:de:0168-ssoar-395173. Accessed 18 Oct 2016

Pohl C (2011) What is progress in TD research? Futures 43:618–626Polk M (2014) Achieving the promise of transdisciplinarity: a critical

exploration oft he relationship between TD research and societal problem solving. Sustain Sci 9:439–451

Reed MS, Stringer LC, Fazey I, Evely AC, Kruijsen JHJ (2014) Five principles for the practice of knowledge exchange in environmen-tal management. J Environ Man 146:337–345

Rossi PH, Freeman HE, Lispey MW (2003) Evaluation: a systematic approach. Sage, Thousand Oaks, p 499

Schneider F, Buser T (2018) Promising degrees of stakeholder interac-tion in research for sustainable development. Sustain Sci 13:129–142 https://doi.org/10.1007/s11625-017-0507-4

Schreier M (2013) Qualitative content analysis. In: Flick U (ed) The SAGE handbook of qualitative data analysis. SAGE, London, pp 170–183

Stauffacher M, Flüeler T, Krütli P, Scholz RW (2008) Analytic and dynamic approach to collaboration: a TD case study on sustain-able landscape development in a Swiss prealpine region. Syst Pract Action Res 21:409–422

Stokols D, Fuqua J, Gress J, Harvey R, Kimari P, Baezonde-Garbanati L, Unger J, Palmer P, Clark MA, Colby SM, Morgan G, Tro-chim W (2003) Evaluating TD science. Nicotine Tobacco Res 5:S21–S39

Stokols D, Hall KL, Taylor BK, Moser RP (2008) The science of team science: overview of the field and introduction to the supplement. Am J Prev Med 35(2):S77–S89

Talwar S, Wiek A, Robinson J (2011) User engagement in sustain-ability research. Sci Public Policy 38(5):379–390

Tobias S (ed) (2015) Raumansprüche von Mensch und Natur. Synthese-bericht des WSL Programmems. WSL Berichte 35:103

Tobias S, Buser T, Buchecker M (2016a) Does real-time visualiza-tion support local stakeholders in developing landscape visions? Environ Plan B Plan Des 43:184–197

Tobias S, Ströbele M, Nobis MP, Obrist MK, Moretti M, Hunziker M, Hersperger AM, Pütz M, Kienast F, Buser T (2016b) Siedlungs- und Landschaftsentwicklung in agglomerationsnahen Räumen. WSL Merkblatt für die Praxis 56:16

Walter AI, Helgenberger S, Wiek A, Scholz RW (2007) Measuring societal effects of TD research projects: design and application of an evaluation method. Eval Progr Plan 30:325–338

Wickson F, Carew AL, Russell AW (2006) TD research: characteris-tics, quandaries and quality. Futures 38:1046–1059

Wiek A (2007) Challenges of TD research as interactive knowledge generation. Gaia 16(1):52–57

Wiek A, Ness B, Schweizer-Ries P, Brand FS, Faioli F (2012) From complex systems analysis to transformational change: a compara-tive appraisal of sustainability science projects. Sustain Sci 7(Sup-plement 1):5–24

Wiek A, Talwar S, O’Shea M, Robinson J (2014) Toward a meth-odological scheme for capturing societal effects of participatory sustainability research. Res Eval 23:117–132

419Sustainability Science (2019) 14:405–419

1 3

Wittmayer JM, Schäpke N (2014) Action, research and participation: roles of researchers in sustainability transition. Sustain Sci 9:483–496. https://doi.org/10.1007/s11625-014-0258-4

Wuelser G, Pohl C (2016) How researchers frame scientific contribu-tions to sustainable development: a typology based on grounded theory. Sustain Sci 11:789–800

Zeischler J, Rogga S, Weith T (2014) Experiences with TD research. Syst Res 31:751–756

Zscheischler J, Rogga S (2015) Transdisciplinarity in land-use science. A review of concepts, empirical findings and current practices. Futures 65:28–44