Benoit Guerin, Stijn Hoorens, Ohid Yaqub, Dmitry Khodyakov* · 2016. 6. 3. · 5th...

5th International Conference on Future-Oriented Technology Analysis (FTA) - Engage today to shape tomorrow Brussels, 27-28 November 2014

THEME 3: CUTTING EDGE FTA APPROACHES

- 1 -

ELICITING EXPERT KNOWLEDGE TO ASSESS GLOBAL SOCIETAL

TRENDS AND THEIR IMPACT ON THE EU

Benoit Guerin*, Stijn Hoorens*, Ohid Yaqub**, Dmitry Khodyakov***

* RAND Europe, Rue la Loi 82, 1040 Brussels, BE: [email protected]/ [email protected]

** University of Sussex - Science Policy Research Unit, Sussex House, Falmer, Brighton, BN1 9RH:

[email protected]

*** RAND Corporation, 1776 Main Street, Santa Monica, CA 90401-3208: [email protected]

Abstract

As part of the Strategy and Policy Analysis System (ESPAS) project,1 RAND, an independent

not-for-profit policy research institute, conducted a study to examine the evidence underpinning

major societal trends in Europe to 2030. A multi-disciplinary team conducted a large-scale online

modified-Delphi exercise to elicit expert opinion on major trends and their impact on EU policy-

making. In this practice-oriented paper, we demonstrate the implementation of the Future-

Oriented Technology Analysis (FTA) approach to solicit input from experts across the world who

participated in one of the six online thematic panels, including demographic change, individual

empowerment, income and equality, migration and mobility, labour and employment, or

technology and education. This paper focuses on the way in which the online modified-Delphi

system was used to solicit expert input on multiple plausible prospects and trends in a

collaborative way, with a view to eliciting policy-relevant insights to help build resilience for future

policymaking in Europe. It outlines how a RAND-developed system called ExpertLens was

implemented to assess the likelihood that different trends materialise by 2030, and how this

contributed to the identification of potential responses to Europe’s societal challenges.

To develop the FTA protocol, the team conducted a trends assessment and analysis using

existing data. First, the literature and existing forecasts were reviewed to describe major trends

identified by experts and to identify their drivers, trajectory, impact, and alternative narratives.

We then used ExpertLens to consult relevant experts worldwide, of which 116 respondents were

included in the analysis. Over a period of about three weeks, experts, divided into six panels,

rated the likelihood of different trends materialising. The ExpertLens process consisted of three

rounds: in Round 1, participants answered a set of questions; in Round 2, they reviewed the

group responses to Round 1 questions and discussed the results with other respondents using

asynchronous and anonymous discussion boards moderated by the research team, and in

Round 3, initial responses were revised in light of insights obtained during the discussion. The

analysis undertaken using the data from the online modified-Delphi approach helped identify

consensus among large and diverse groups of experts on the trends, uncertainties, and trade-

offs, which helped prioritize policy challenges based on their likelihood and potential impact.

1 For more information about ESPAS, see: http://europa.eu/espas/about-espas/index_en.htm



- 2 -

The first section of this paper introduces the study and its policy and institutional background at

the EU level. Section 2 focuses on the ExpertLens tool used for the Delphi exercise and explains

how it was deployed in practice so as to support strategic policy making at the EU level. The

third section of the paper highlights major findings from the study and their implication for the EU

landscape. Finally, the fourth section summarizes study findings and discusses their impact.

Keywords: Delphi, ExpertLens, foresight, trends, futures, expert elicitation, European Union, society, demography,

middle class, migration, technology, education, empowerment, inequality, work, labour

1. Introduction

The European Union is experiencing turbulent times. The global financial crisis, the Eurozone

crisis, soaring unemployment rates, and instability at its Eastern borders are just a few of the

challenges over the past years. They reflect the unstable and fast-changing global environment

of the EU. If left unchecked, these challenges could undermine not only the EU’s economic and

political influence, but also its ideals and values.

The European Strategy and Policy Analysis System (ESPAS) was set up to develop a lasting

framework to assess global trends and to develop policy responses across EU institutions. It

was launched as an inter-institutional effort in 2010 to investigate the global trends that will

prevail in 2030 and to determine the long-term challenges that decision-makers will be faced

with in the next institutional cycle (2014-2019). Its purpose is embedded within a wider context of

building a permanent EU forecasting capacity, relying on the collaboration of various EU

institutions (European Commission, European Parliament and the Council of the EU) and actors

in the individual member states, but also to set up a continuous framework to assess global

trends and to develop policy responses across the EU institutional framework.

Researchers at RAND were commissioned to further investigate the theme of societal changes,

by analysing key global trends in this field and by drawing their implications for the continent.2

The ESPAS Task Force identified the following six main thematic areas to be refined,

documented and analysed:

1. Income and consumption

2. New technologies, media and access to education;

3. Individual empowerment;

4. Changing demography;

5. Migration;

6. Employment and work.

To study these themes, the team sought to assess the evidence base, uncertainties and potential trajectories surrounding trends in the six major themes. The research was divided into

2 Three different research groups were commissioned to investigate global trends in the area of societal issues (by RAND),_the

economy (by a consortium led by the Centre for European Policy Studies) and governance and power (by think tanks Chatham House and FRIDE). The research teams were requested to draw upon the findings of a pilot project, which were documented in a report published by the EU Institute for Security Studies (EUISS, 2012).



- 3 -

three main tasks. Firstly, the team conducted a thematic review of the academic and grey literature on the trends for each theme. Secondly, additional information was collected for all themes using an online Delphi exercise with a large group of international experts. Finally a series of 29 semi-structured interviews were conducted with experts from academia and think tanks, policymakers, and leading thinkers from the private or voluntary sector to further explore the findings from the Delphi exercise and literature review.

This paper will focus on the Delphi exercise conducted using the ExpertLens system - a previously evaluated online platform that uses a modified-Delphi structure to elicit expert opinion and engage stakeholders with different sets and levels of expertise in a structured fashion (Khodyakov et al. 2011). Because of its online nature, ExpertLens helps large groups of geographically and spatially distributed individuals to provide their expert opinion without traveling to a centralized location. This paper explores the application of this on-line Delphi platform in the context of a future-oriented analysis to understand the development and drivers of specific societal trends, with a view to advising policymakers about the most important challenges in the next 20 years.

Delphi approaches, and ExpertLens in particular, have been used extensively in medical and

technological context. The Delphi technique has been used widely in national (technology)

foresight (particularly in Japan and Germany, e.g. Kuwahara 1999; Cuhls et al. 2001) and in

health research (Jones 1995), such as the fields of technology assessment (e.g. Elder and

Andrew 1992), education and training (e.g. Fraser et al. 1992), research prioritization (Claassen

et al.) or nursing and clinical practice (e.g. Passannante et al. 1993). Typically, these

applications tend to have a set of well-defined questions in a relatively field of research that are

answered using an expert panel.

For this paper, we have applied the technique in an online environment to a very diverse set of

broad societal questions. The purpose of using the ExpertLens for ESPAS was to understand

the perspective of experts on the fault-lines in the evidence base gathered through the literature

review. In this sense, ExpertLens not only provided information intended to complement the

literature, but also provided background information for key stakeholder interviews, which were

conducted subsequently. Identifying and assessing long-term trends for policy decisions

requires input from experts from a wide range of disciplines and backgrounds. The results of this

paper may be useful in further refining this approach for such contexts.

The sections below present the process managed by the team to run the elicitations in a way

that would support strategic policymaking, and the substantial results and recommendations

from the study.

2. Methodological approach

a. RAND’s ExpertLens approach

i. A brief introduction to the Delphi method

The Delphi method relies on a core component of military operations research, namely the

reliance on the judgement of experts (Brown, 1968). Developed by researchers at the RAND

Corporation over 60 years ago, Delphi (Dalkey and Helmer, 1963) is a method for gathering

expert opinion and providing structured feedback to participating experts, rather than a way of

“predicting the future” (Ismail, 2009). In a traditional Delphi process, participants anonymously



- 4 -

respond to a survey, the results of which are combined and fed back to the group (Linstone

1978). Upon reviewing the group responses, the participants have the opportunity to refine their

answers through a second survey. This process can be repeated until consensus is reached or

a pre-determined number of rounds has passed (Van de Ven and Delbecq 1974; Rowe and

Wright 1999; 2001). For some issues, such as controversial problems, the Delphi method may

never lead to consensus. However, the reasons participants provide in support of their

responses may be used to see where the major points of disagreement lie. Paul (2011: 13-14)

outlines the richness of building on specialist knowledge as opposed to one-off elicitations. He

explains the value of Delphi exercises:

“As the experts review the justifications and calculations made by the others, they may

recognize factors that they failed to include in their own calculations [or answers] or come

to understand that they have over- or underestimated [a specific factor]. The revised

estimates are likely to be based on more complex calculations, be better calculations [or

estimates], and be closer to each other than were the initial individual expert estimates.”

ii. An online Delphi system

Traditional Delphi studies were conducted using mail surveys and therefore rarely engaged a

large number of participants. An online Delphi, such as the ExpertLens system (Dalal, et al.

2011; Khodyakov et al. 2011), can engage more than 100 participants in a short period of time

and allow them not only to answer questions but also to participate in an online discussion of

their responses. Moreover, the participants can be geographically dispersed and can offer

potentially more efficient use of experts’ time by requesting their insights at their own

convenience, with some deadlines (Bowles et al. 2003).

Other advantages may include the ability to make online discussions anonymous and thus

reduce possible biases based on participant status or personality (Dubrovsky et al. 1991;

Murphy et al. 1998); and the benefit of contributing to the elicitation process at the time

convenient to panellists (Bowles et al. 2003; Dubrovsky, 1991; Murphy et al., 1998).

Facilitating a Delphi process in an online environment, however, may be at the detriment of the

level of participants’ engagement and interaction, potentially caused by their relative unfamiliarity

with online tools in general and a possibility of technical difficulties accessing or using an online

system. This may undermine panellists’ willingness to participate and affect the quality of

deliberations and outputs (Snyder-Halpern et al. 2000; Khodyakov et al. 2011).

ExpertLens is useful when the opinion of a select number of individuals who have training and

experience in areas related to the topic of the study is required. It is therefore the expertise of

the participants that drives the power of analysis, and not their number per se.

iii. Applying a three-round Delphi process

Using ExpertLens, the Delphi process was divided into six parallel but separate expert panels,

covering the six thematic areas outlined above. The study team iteratively consulted experts by

structuring the Delphi exercise into three rounds, which are outlined below and in Figure 1.



- 5 -

a) In Round 1, participants responded to a set of predetermined questions, which varied by

the panel.

b) In Round 2, participants familiarized themselves with the answers given by others and

discussed the group responses via asynchronous and anonymous online discussion

boards moderated by the research team

c) In Round 3, the participants were given an opportunity to modify their original answers in

light of the group discussion.

Figure 1. Schematic illustration of the iterative Delphi process in ExpertLens

ExpertLens offered a structured approach for determining what study participants thought and

an interactive way for finding out why they thought so without requiring a face to face meeting,

thereby reducing the cost of implementing the Delphi exercise. It provided a way to combine

quantitative and qualitative data and offered researchers several comparative advantages over

other survey-based data collection methods, namely convenience, anonymity, and exposure to

diverse perspectives, among others (Dalal, et al. 2011; Khodyakov et al. 2011).

Whilst including a Round 2 consisting of providing a summary of group responses is a common

characteristic of Delphi exercises, offering an opportunity to discuss results through a discussion

board functionality is a relatively new phenomenon. Previous research identified a number of

potential disadvantages of online interaction (Brown 2000; Khodyakov et al. 2011), including

variable participation rates, information overload, and difficulties in following discussion threads

(Wainfan and Davis 2004; Turoff and Hiltz 1996). In addition, studies show that in-person panels

given the same information may come up with different conclusions (Keeney et al. 2001;

Shelleke et al 1998), yet the magnitude of this effect for online panels with larger numbers of

participants is still unknown.

For this study, the team conducted ExpertLens elicitations for each of the six different themes

with a different panel of experts. De facto, six different expert elicitations were conducted: each



- 6 -

theme featured its own panel of experts. Experts were only able to participate in one elicitation,

namely the one where the participant’s expertise was best suited.

iv. Selecting participants for the thematic elicitations for ESPAS

In order to benefit from the most relevant and up-to-date expertise for each of the thematic

areas, the research team invited 1367 experts in total to participate in one of the six Delphi

elicitations. Participants were selected from the research team’s knowledge of the community as

developed through the initial literature review, conference lists, reference lists from seminal

articles and reports, web searches, and personal networks. The initial contact was made through

email. The team received 412 positive responses. Those who agreed to participate in the study

were assigned to one of the six elicitations. Experts were only invited to participate in one of the

six elicitations each, to avoid over-burdening the participants. Each participant received a unique

ExpertLens User ID (unknown to the research team) to allow the system to track individual

responses, discussion posts, and logins. The participants identified themselves from a variety of

affiliations and countries, as shown below.

Table 1. Participants’ affiliations by sector

Affiliation Number of participants

National governments 11

International organisations 15

Private sector 9

Think tanks 45

NGOs and charities 15

Academia 130

N.A. 4

Total 229

In selecting participants, the team paid particular attention to the diversity of the panel members,

including the regional makeup of the participants.

Table 2. Participants’ region of origin

Region Number of participants

Europe 155

North America 41

South America 7

Asia 15



- 7 -

Africa 7

Australasia 4

Total 229

Although the 412 experts were not evenly distributed across the six elicitations, each elicitation

had a minimum of 55 experts participating on the panel. For Round 1, response rates ranged

from 58% to 62% for the six elicitations.

The response rate for Round 2 was lower than that for Round 1. In total, 214 users accessed

ExpertLens during Round 2 (51%), and 91 users (22%) contributed at least 1 comment or

discussion thread. There were a total of 307 posts across 80 threads, giving rise to aRound

25,000 words of discussion for analysis. Although only 22% were actively contributing to the

discussion, substantially more (up to 51%) could have been reading and monitoring the

discussion.

As is common with Delphi studies, Round 3 showed attrition in participation. There were 116

respondents to Round 3 questions (28%), nearly all of whom answered half of the questions

(26%), and three-quarters of whom answered more than 90% of the questions (21%). The

responses of these 116 respondents were used in the analysis, and although not all of them

answered every question, averages were calculated to incorporate the number of respondents

per question individually.

The attrition rate for the 6 elicitations was relatively high compared to what can be observed in

similar studies. The aggregated overall participation rate of 28% (116 respondents out of 412

accepted invitations) tends to be somewhat lower than 45%-50% that can be typically expected

for a traditional Delphi study (Jillson 2002) and 66% for another recent application of an online

Delphi using ExpertLens (Khodyakov 2011). However, the total number of analysed responses

per elicitation did meet the minimum threshold of 15 respondents who completed Round 3 that

we had set ourselves. This is important, although in such exercises, the qualitative insights

offered by ExpertLens emerge not just from the number of participants, but also the quality of

the participants’ expertise and the nature of their interactions.

b. Practical implementation of the Delphi to support policymaking

For each of the six elicitations, researchers identified a set of 3-5 sub-topics to be analysed in

the Delphi. These sub-topics stemmed from scan of the available literature and available

quantitative projections. The nominated topics were more likely to be included when: they were

considered to have high future societal impact; they were characterised by relatively high

uncertainty; or there is a high degree of disagreement in the literature. The inconclusiveness in

the literature around these sub-topics was subsequently formulated into a small set of specific

questions. The Delphi questionnaire had a maximum of 25 questions per elicitation, which were

validated with the ESPAS task force.

Three broad types of questions were used in each panel: estimation, rating, and ranking. An

example of each question type that was asked is provided with a screenshot below.



- 8 -

Estimation: Participants are asked to enter a value to estimate the likelihood of a trend or

to estimate the trend itself (Figure 2a).

Rating: Participants are asked to rate a trend on a Likert-type scale of 1 to 7 (Figure 2b).

Ranking: Participants are asked to rank trends relative to each other by ‘dragging and

dropping’ into a specific order (Figure 2c).

Figure 2a. Example of an “Estimation” question

Figure 2b. Example of a “Rating” question.

Figure 2c. Example of a “Ranking” question.

At the start of Round 1, experts were sent an email link inviting them to join the ExpertLens,

along with background material provided by the researchers. Once they had agreed to

participate and logged in, the expert was asked to answer the set of closed-ended questions

about trends.3 These ranged from questions on the likelihood that Asia as a whole will account

for over 50% of global economic output by 2030 to questions asking experts provide estimates

of the world’s population living in urban areas by 2030.

In Round 2, the experts were provided with a summary comparing their own answers with those

of the entire group: a group median and quartiles were shown, along with the bar charts showing

3 The six ExpertLens elicitations were launched on 23

rd April 2013. We initially planned to have each round running for 1 week,

straight after each other. However, to allow some participants who had requested extra time, we extended some of the rounds for some of the elicitations. All of elicitations were closed on or before 20

th May 2013.



- 9 -

the distribution of responses (See Figure 2). Participants could toggle between a chart view and

a table view. Each expert then had an opportunity to engage in an anonymous online discussion

about the questions with the entire group of experts. A dedicated researcher monitored and

facilitated the discussion during this round on a regular basis, asking open-ended questions

about the trends seen in Round 1 results, engaging participants into discussion by promoting

two-way information exchange, and asking clarifying questions about already-posted comments.

Figure 2. Visualizing the ExpertLens answers

Experts then had an opportunity to engage in an anonymous online discussion about the

questions with the entire group of experts. A dedicated researcher monitored and facilitated the

discussion during this round on a regular basis, seeding questions across the trends, engaging

participants into discussion, and asking clarifying questions about posted comments.

The discussion in Round 2 allowed for exploration of issues in a way that is not possible within a

structured question set. For example, diverging views on strategies were identified in by the

research team, and by directing attention to points of divergence within the group, we were able

to explore why particular views were held. These types of discussions occurred across the

elicitations and questions (prompted where needed by the research team) providing a rich set of

qualitative information to support our analysis.

Finally, in Round 3, experts were asked to revise their Round 1 responses in light of discussions

or further reflections made during Round 3.

To derive insights from the data collected by answers from the participants in each panel, the

research team used three analytical devices, namely averages; agreement; and convergence.



- 10 -

Average ratings and rankings: For each question, median scores were calculated and

reported back to the group at the end of Round 1. Respondents reflected on the group’s

median score in relation to the score they provided as individuals, and this provided a

basis to engage in discussion (Round 2) and then for refinement of answers (Round 3).

In addition to median scores, the research team also used mean scores (which tend to

be more sensitive to extreme values in the group) and took note of the distribution scores

(in particular whether they were uniform – that is even across each value on the scale –

unimodal, bimodal, or skewed)4 in describing study findings. These were not reported

back to the group, but were used in the analysis to draw additional insights.

The agreement/disagreement axis: This is the extent to which participants respond with

similar answers and the distribution of answers is unimodal (i.e. clustered around a single

value). A uniform or bimodal distribution of responses may by contrast be indicative of

disagreement between participants: metrics for agreement or disagreement may include

interquartile ranges between answers, the standard deviation, or the mean absolute

deviation from the median answer (Dalal et al. 2011; Fitch et al. 2001).

The convergence/divergence axis: This is where the group’s responses differ in the

degree of agreement they show between the rounds. A reduction in the absolute

deviation from the median between Round 1 and 3 was taken as a sign of increased

consensus (Dalal et al. 2011; Fitch et al. 2001).

3. Results, discussion and implications

The ultimate aim of the study was to identify the key societal challenges for the incoming

leadership of the EU. The ExpertLens Delphi approach helped delineate expert knowledge on

those issues where the available literature and data are inconclusive. Results from all six

elicitations provided useful advances of knowledge on the themes studied. This enabled the

identification of cross-cutting trends and drivers, their outcomes, key uncertainties and ultimately

the formulation of key policy challenges.

a. Elicitation results from per thematic area

The ExpertLens approach was most useful in answering closed, well-defined questions, such as

quantifying the degree of uncertainty or level of impact. The results also helped identify areas of

sustained contention among experts. Aside from the quantitative results, the moderated

discussions helped shed light on the underlying reasons behind these results.

The elicitation on demographic issues focused on the factors of uncertainty in demographic

projections, such as fertility rates and life expectancy, their drivers and consequences. In the

same way, the elicitation on migration focused on issues such as migration flows between

developing and ageing countries, and the factors affecting migration, ranging from climate

4 When participants respond with similar answers, the results can be said to be unimodal: if one were to plot results on a graph, a

single ‘peak’ would appear. By contrast, a uniform (i.e. flat across each item on the scale) or bimodal (i.e. where two ‘peaks’ appear in the plotted data) distribution of responses is indicative of disagreement between participants in the Delphi exercise on the same scale. The terminology of convergence/divergence is used to reflect the fact that the group’s responses differ in the degree of agreement they show across the rounds. For instance, if the answers provided to a specific question in Round 1 of the Delphi show disagreement, but the group’s answers show a greater degree of agreement by Round 3, it is possible to say that there is a degree of convergence. In some cases, it may be that the group reaches some form of consensus on a given question.



- 11 -

change and growing international mobility to urbanisation. Key issues covered in the panel on

labour included the ageing European workforce, the employment prospects of low-skilled

workers, the skills mismatch and its impact, and youth unemployment. The panel on the global

middle class aimed to assess the growth rate of the global middle class, the factors likely to

impede it, and the tension between middle class growth and within-country inequality.

Insights derived from the Delphi shed new light on the importance of global societal trends for

the EU and the nature of their outcomes. For instance, while experts taking part in the elicitation

on the rise of a global middle class initially rated the likelihood of a doubling in the size of the

middle class by 2030 at 50%. However, a discussion on the presence of safety nets preventing

people from floating in and out of poverty in Round 2 influenced the results, and by the Round 3,

the expert panel agreed that this likelihood was about 20%. Our results in general suggest that a

rising inequality in wealth, labour, and skills are among the biggest future challenges for Europe.

Similarly, responses from the Delphi elicitation on demography suggests that the likelihood of a

rise in the average life expectancy at birth in high-income countries without a ceiling, horizon

2030, is around 75%, about 10% higher than in the initial estimate provided in Round 1 of the

Delphi exercise, with some (albeit little) disagreement among experts – about 10% of experts

thought the likelihood was below 50%. Although the attrition of experts between rounds may

affect the results, the difference in results across rounds may also be accounted for by the

debates taking place in Round 2 of the exercise. In this example, a range of drivers for longevity

were discussed, including the health status of current generations (which are less healthy than

previous younger generations, owing partly to increases in obesity and unhealthy behaviours,

etc.) and therefore perhaps less likely to live as long as current generations. These expert

insights have direct relevance to major policy issues, such as the future cost of welfare systems,

notably in European countries with ageing populations.

b. Discussion and limitations

The research topics to be addressed as part of the wider study for ESPAS were very broad and

ill-defined, the research questions were generally complex and open-ended. The tool works best

for concrete and well-scoped areas of inconclusiveness in the literature and data. Therefore, the

identification and formulation of precise, policy-relevant closed questions requires extensive

preparation. Given the breadth and the nature of the research questions, the actual quantitative

results offer modest help in adding to the existing knowledge base.

However, the process of arriving to these results offered useful insights in the complexity of the

questions. The convergence or divergence effects reflected on whether experts’ level of

agreement increased or decreased between the rounds. And particularly the moderated forum

discussions in Round 2 were very valuable in answering some of the more complex questions.

Moreover, Round 2 discussions seemed to improve the engagement of a sub-set of experts,

who could subsequently be recruited for in-depth interviews.

Study findings highlighted the need to build a resilient policy system by designing policies that

are robust yet adaptable to a changing strategic landscape. Results were synthesised in a final

report (Hoorens et al. 2013), and detailed analysis will be released shortly as part of a series of



- 12 -

evidence reports. The synthesis report produced by RAND was widely circulated among EU

stakeholders, and general outcomes from this phase of the ESPAS project are currently being

communicated to the incoming leadership of the European Commission and European

Parliament to help them coordinate the EU’s response to global challenges.

c. The impact of the Delphi on identifying policy challenges

The results from the six Delphi elicitations were used by the research team as a basis for semi-

structured interviews to develop concrete policy recommendations and policy actions for the EU

within the then forthcoming institutional cycle of 2014-2019.

The study, of which the Delphi exercise was only a part, culminated in the identification of a non-

exhaustive selection of 33 key trends within the themes studied. The Delphi contributed to the

team’s assessment of the evidence, the degree of uncertainty of each of these trends, and their

level of impact.

The study concluded that inequality will be the single most prominent societal challenge for the

EU in the coming decades. The gap between rich and poor in the EU has widened in recent

crisis years and will likely further exacerbate. Not only would this represent a trend break for the

EU, it is also at odds with the EU’s foundation: inclusive growth. The study identified eleven

salient policy challenges for the next Commission, clustered around three themes: 1) Investing in

citizens; 2) A new growth paradigm; and 3) Reinventing government.

4. Conclusions

The research team undertook an innovative exercise by deploying an online Delphi through

ExpertLens on societal issues. This paper explained how this tool was leveraged to derive

insights for decisionmakers.

The results of the Delphi exercise fed into the overall analysis, which was presented in a

synthesis report (Hoorens et al. 2013). Whilst the quantitative results only addressed specific

aspects and a limited number of the many high level research questions, the Delphi approach

usefully complemented inconclusive findings in the literature. The report, along with those of the

research teams studying trends in economy and in governance and power, was made available

to those involved in ESPAS. Staff at Bureau of European Policy Advisors (BEPA) are currently in

the process of preparing a briefing document based on these reports for the new Juncker

Commission, helping the new leadership to shape its long-term policy agenda.

Given the amount of preparation required, and the specificity of the quantitative results, if the team was to engage in a similar exercise addressing such broad societal issues again, we would recommend narrowing the scope of questionnaire considerably and focusing only on areas of disagreement in the literature.

References

Bellamy N, Anastassiades TP, Buchanan WW, Davis L, Lee P, McCain GA, et al. (1991) Rheumatoid arthritis anti-

rheumatic trials. III. Setting the delta for clinical trials of anti-rheumatic drugs–results of a consensus development

(Delphi) exercise. J Rheumatol, 18, pp.1908–15.



- 13 -

Bowles KH, Holmes JH, Naylor MD, Liberatore M, Nydick R (2003) “Expert consensus for discharge referral decisions

using online Delphi.” AMIA Annual Symposium Proceedings 2003, 106-109.

Brown, B.B., (1968), Delphi Process: A Methodology Used for the Elicitation of Opinions of Experts, Santa Monica,

Calif.: RAND Corporation, P-3925. As of August 2012: http://www.rand.org/pubs/papers/P3925

Brown R (2000) Group processes: Dynamics within and between groups. Blackwell Publishing.

Claassen, CA., JL. Pearson, D. Khodyakov, PM. Satow, R Gebbia, AL. Berman, et al. (2014). Reducing the burden of

suicide in the US: the aspirational research goals of the National Action Alliance for Suicide Prevention Research

Prioritization Task Force. American journal of preventive medicine.

Cuhls, K., Blind, K., & Grupp, H. (Eds.). (2001). Innovations for our Future: Delphi'98: New Foresight on Science and

Technology, Vol 13, Springer

Dalal, S., Khodyakov, D., Srinivasan, R., Straus, S., and Adams, J., (2011), “ExpertLens: A system for eliciting

opinions from a large pool of non-collocated experts with diverse knowledge,” Technological Forecasting and Social

Change, 78(8), pp. 1426–1444

Dalkey, N., and Helmer, O., (1963), An experimental application of the Delphi method to use of experts, Manage. Sci.

9, pp. 458–467.

Dubrovsky VJ, S. Kiesler, BN. Sethna (1991) “The equalization phenomenon: Status effects in computer-mediated

and face-to-face decision-making groups.” Human-Computer Interaction, 6, pp. 119-146

Elder OC Jr. and ME Andrew, (1992) Important curriculum content for baccalaureate allied health programs: a survey

of deans. J Allied Health, 21, pp. 105–15

ESPAS, (2013), The World in 2030: Highlights. ESPAS Outreach leaflet, Autumn 2013. As of October 2014:

http://europa.eu/espas/pdf/espas-outreach-leaflet.pdf

EUISS, (2012), European Strategy and Policy Analysis System (ESPAS): Global trends 2030 – Citizens in an

interconnected and polycentric world, Paris: European Union Institute for Security Studies.

Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lazaro P, Loo Mvh, McDonnell J, Vader JP, Kahan JP

(2001) RAND/UCLA Appropriateness Method (RAM). Santa Monica: RAND Corporation, 109.

Fraser CE, Smith QW, Luchi RJ.Geriatric fellows' perceptions of the quality of their research training. (1992) Acad

Med, 67, pp. 696–8

Hoorens, S., B. Guerin, JJ. Ghez, D. Schweppenstedde, T. Hellgren, V. Horvath, M. Graf, B. Janta, S. Drabble and S.

Kobzar, (2013). “Europe's Societal Challenges: An analysis of global societal trends to 2030 and their impact on the

EU.” Santa Monica, CA: RAND Corporation. http://www.rand.org/pubs/research_reports/RR479.

Ismail, S., (2009), “Delphi exercises”, in: Ling, T. L. and Villalba van Dijk, Performance Audit Handbook: Routes to

effective evaluation, Santa Monica, Calif.: RAND Corporation, TR-788-RE. As of August 2012:

http://www.rand.org/pubs/technical_reports/TR788

Jillson IA (2002) “The national drug-abuse policy Delphi. In The Delphi Method: Techniques and Applications”

Linstone H, Turoff M. (eds), pp. 119-154

Jones, J. (1995) “Qualitative Research: Consensus methods for medical and health services research” BMJ, 311:376

Keeney S, F. Hasson, H. McKenna (2001) “A critical review of the Delphi technique as a research methodology for

nursing.” International Journal of Nursing Studies, 38, pp. 195-200.

Khodyakov, D. S. Hempel, L. Rubenstein, P. Shekelle, R. Foy, S. Salem-Schatz, S. O'Neill, M. Danz and S. Dalal

(2011) Conducting Online Expert panels: a feasibility and experimental replicability study. BMC Medical Research

Methodology, 11, pp. 174

Kuwahara, T. (1999) Technology forecasting activities in Japan. Technol. Forecast. Soc. Change, 60, p. 12



- 14 -

Linstone, H. (1978) “The Delphi Technique”, R.B. Fowles (Ed.), Handbook of Futures Research, Greenwood Press,

Westport, CT (1978), pp. 271–300

Murphy MK, NA, Black, DL. Lamping, CM. McKee, CFB. Sanderson, J. Askham (1998) “Consensus development

methods, and their use in clinical guideline development.” Health Technology Assessment, 2

Passannante MR, Restifo RA, Reichman LB. (1993) Preventive therapy for the patient with both universal indication

and contraindication for isoniazid. Chest, 103, pp. 825–31

Paul, C., (2011), Counterinsurgency Scorecard: Afghanistan in Early 2011 Relative to the Insurgencies of the Past 30

Years, Santa Monica, Calif.: RAND Corporation, OP-337-OSD. As of October 2014:

http://www.rand.org/pubs/occasional_papers/OP337

Rowe, G. and G. Wright (2001) “Expert Opinions in Forecasting: The Role of the Delphi Technique.” J.S. Armstrong

(Ed.), Principles of Forecasting: A handbook for Researchers and Practitioners, Springer, pp. 125–144

Rowe, G. and G. Wright (1999) “The Delphi technique as a forecasting tool: issues and analysis.” Int. J. Forecasting,

15 (4), pp. 353–375

Snyder-Halpern R, C. Thompson, J. Schaffer (2000) “Comparison of mailed vs. Internet applications of the Delphi

technique in clinical informatics research.” Proceedings of the AMIA Symposium, pp. 809-813.

Shekelle P, J. Kahan, S. Bernstein, L. Leape, C. Kamberg, R. Park (1998) “The reproducibility of a method to identify

the overuse and underuse of medical procedures.” New England Journal of Medicine, 338, pp. 1888-1895

Turoff M, and Hiltz SR: Computer-based Delphi processes. In Gazing into the oracle: the Delphi method and its

application to social policy and public health. Edited by Adler M, Ziglio E. Jessica Kingsley Publishers, pp. 56-89

Van de Ven, A.H., and Delbecq, A.L. (1974), The effectiveness of nominal and Delphi, and interacting group decision

making processes, Acad.Manage. J. 17 (4), pp. 605–621.

Wainfan L and Davis PK (2004) Challenges in virtual collaboration: Videoconferencing, audioconferencing, and

computer-mediated communications. Santa Monica: RAND Corporation.

Benoit Guerin, Stijn Hoorens, Ohid Yaqub, Dmitry Khodyakov* · 2016. 6. 3. · 5th...

Documents

Transcript of Benoit Guerin, Stijn Hoorens, Ohid Yaqub, Dmitry Khodyakov* · 2016. 6. 3. · 5th...

Benoit Guerin*, Stijn Hoorens*, Ohid Yaqub**, Dmitry Khodyakov*** · 2016. 6. 3. · 5th...

Documents

Transcript of Benoit Guerin*, Stijn Hoorens*, Ohid Yaqub**, Dmitry Khodyakov*** · 2016. 6. 3. · 5th...

Benoit Guerin, Stijn Hoorens, Ohid Yaqub, Dmitry Khodyakov* · 2016. 6. 3. · 5th...

Transcript of Benoit Guerin, Stijn Hoorens, Ohid Yaqub, Dmitry Khodyakov* · 2016. 6. 3. · 5th...