Applying relevance-assessing methodologies to …...Applying relevance-assessing methodologies to...

Expert Group on Evaluation Methodologies for the Interim and Ex-post Evaluations of Horizon 2020 2017

Applying relevance-assessing methodologies to Horizon

2020

FINAL REPORT

Applying relevance-assessing methodologies to Horizon 2020 European Commission Directorate-General for Research and Innovation Directorate A — Policy Development and Coordination Unit A.5 — Better Regulation Contact Arnold Weiszenbacher E-mail [email protected] ; [email protected] European Commission B-1049 Brussels

Manuscript completed in 2017.

This document has been prepared for the European Commission however it reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

More information on the European Union is available on the internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2017

© European Union, 2017. Reuse is authorised provided the source is acknowledged. The reuse policy of European Commission documents is regulated by Decision 2011/833/EU (OJ L 330, 14.12.2011, p. 39).

For any use or reproduction of photos or other material that is not under the EU copyright, permission must be sought directly from the copyright holders.

EUROPEAN COMMISSION

Applying relevance-assessing methodologies to Horizon 2020

| P a g e EXPERT GROUP ON EVALUATION METHODOLOGIES FOR THE INTERIM AND EX-POST EVALUATIONS OF HORIZON 2020

Identif y


Prepared by the Expert Group on Evaluation Methodologies for the

Interim and Ex-post Evaluations of Horizon 2020

February, 2017


i | P a g e EXPERT GROUP ON EVALUATION METHODOLOGIES FOR THE INTERIM AND EX-POST EVALUATIONS OF HORIZON 2020

Identif y

Authors

This publication has been coordinated by Katerina Valta (Rapporteur), Mirjana Pejić Bach (Co-chair for Relevance), Irena Kondratenko, and Ramon Flecha (Chair), with the collaboration of Anna Maria Augustyn, Sorin Cheval, Michele De Gennaro, Toivo Maimets, Christine Mohler, Milan Palat, Krassimira Paskaleva, Angela Cristina Plescan, Alfred Radauer (Rapporteur), David Southwood, Karen Hanghoj Sorensen, Peter Van den Besselaar and Barbara Widera.

Acknowledgements

The authors would like to acknowledge the support by the European Commission services and in particularly Daniel Deybe, Liviu Stirbat, Nelly Bruno and Arnold Weiszenbacher from the DG Research and Innovation.

Linguistic version

Original: EN

Disclaimer

This publication is based on the work produced by the Expert Group on Evaluation Methodologies for the Interim and Ex-post Evaluations of Horizon 2020 (EG METHO). The content of this publication does not reflect the official opinion of the European Union. Responsibility for the information and views expressed in the ‘Development of new methodologies to assess the relevance of European Research and Innovation Framework Programmes’ lies entirely with the authors.


ii | P a g e EXPERT GROUP ON EVALUATION METHODOLOGIES FOR THE INTERIM AND EX-POST EVALUATIONS OF HORIZON 2020

Identif y

Table of Contents

Acronyms ..................................................................................................................... v

Executive summary ...................................................................................................... 6

CHAPTER 1: Introduction ........................................................................................... 32

CHAPTER 2: The notion of relevance from the European Commission perspective ..... 36

CHAPTER 3: General framework and methods for the assessment of the relevance of European research and innovation framework programmes ...................................... 38

CHAPTER 4: Process for the assessment of the relevance of European research and innovation framework programmes ........................................................................... 41

4.1. General description ............................................................................................. 41

4.2. Methodologies for the identification of the: current priorities based on EU and international key policy documents (Q1.1); needs of society to EU citizens (Q1.2); and the subsequent technological and scientific advances (Q1.3) .................................... 43

4.2.1. Methodologies for the identification of the current priorities from EU and international key policy documents (Q1.1) ................................................................. 43

4.2.1.1. Exploratory text analysis by experts for the identification of the current priorities from EU and international key policy documents ..................................................... 43

4.2.1.1.1. Short description of the methodology ................................................................ 43 4.2.1.1.2. Step-by-step description of the methodology...................................................... 44 4.2.1.1.3. Advantages and disadvantages of the methodology ............................................. 47 4.2.1.1.4. Strategies for overcoming disadvantages ........................................................... 48

4.2.1.2. Computer content analysis for the identification of the current priorities from EU and international key policy documents .................................................................. 50


4.2.2. Methodologies for the identification of the needs of society to EU citizens (Q2.1) .. 55

4.2.2.1. Experts’ exploratory methodology for the identification of the needs of society to EU citizens ......................................................................................................... 55


4.2.2.2. Experts’ exploratory approach for the identification of the needs of society to EU citizens from Eurobarometer surveys and EC public consultations reports ................... 61



iii | P a g e EXPERT GROUP ON EVALUATION METHODOLOGIES FOR THE INTERIM AND EX-POST EVALUATIONS OF HORIZON 2020

Identif y

4.2.2.3. Experts’ selection of keywords for the identification of the needs of society to EU citizens .............................................................................................................. 68


4.2.2.4. Computer content analysis for the identification of the needs of society to EU citizens from Eurobarometer surveys ..................................................................... 75


4.2.2.5. Social media analysis for the identification of the needs of society to EU citizens......................................................................................................................... 79


4.2.3. Methodologies for the identification of the subsequent technological and scientific advances (Q3.1) ..................................................................................................... 89

4.2.3.1. Experts’ exploratory approach for the identification of subsequent technological and scientific advances ........................................................................................ 89


4.2.3.2. Bibliometric analysis for the identification of subsequent technological and scientific advances .............................................................................................. 96


4.2.3.3. Social media analysis for the identification of subsequent technological and scientific advances ............................................................................................ 100

4.2.3.3.1. Short description of the methodology .............................................................. 100 4.2.3.3.2. Step-by-step description of the methodology.................................................... 101 4.2.3.3.3. Advantages and disadvantages of the methodology ........................................... 107 4.2.3.3.4. Strategies for overcoming disadvantages ......................................................... 108

4.2.3.4. Patent analysis for the identification of subsequent technological or scientific advances.......................................................................................................... 110

4.2.3.4.1. Short Description of the methodology .............................................................. 110 4.2.3.4.2. Step-by-step description of the methodology.................................................... 112 4.2.3.4.3. Advantages and disadvantages of the method .................................................. 113 4.2.3.4.4. Strategies for overcoming disadvantages ......................................................... 114


iv | P a g e EXPERT GROUP ON EVALUATION METHODOLOGIES FOR THE INTERIM AND EX-POST EVALUATIONS OF HORIZON 2020

Identif y

4.3. Methodologies for the assessment of relevance of European research and innovation framework programmes with: the current priorities based on EU and international strategic policy documents (Q1.2); the needs of society to EU citizens (Q2.2); and the subsequent technological and scientific advances (Q3.2) ............... 115

4.3.1. Experts’ exploratory approach using human content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2) .............................................................. 115

4.3.1.1. Short description of the methodology ...................................................... 115

4.3.1.2. Step-by-step description of the methodology ............................................ 115

4.3.1.3. Advantages and disadvantages of the methodology ................................... 121

4.3.1.4. Strategies for overcoming disadvantages ................................................. 121

4.3.2. Experts’ exploratory approach using computer content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2) .......................................................... 122



4.3.2.3. Advantages and disadvantages of the methodology .......................... 124

4.3.2.4. Strategies for overcoming disadvantages .......................................... 125

4.3.3. Text mining approach for the assessment of relevance of H2020 to the identified EU and international priorities (Q1.2) ........................................................................... 125

4.3.3.1. Short Description of the methodology ...................................................... 126




4.3.4. Social media analysis for the assessment of relevance of H2020 (Q1.2, Q2.2, Q3.2) .......................................................................................................................... 130





CHAPTER 5: Application of the methodologies to the relevance questions and key findings in different sections of Horizon 2020 .......................................................... 135

CHAPTER 6: Conclusions .......................................................................................... 154

References ............................................................................................................... 172

List of Tables ............................................................................................................ 173

List of Figures .......................................................................................................... 174

Appendix 1. Summary of the overall methodological approach ................................ 176

Appendix 2. Summary of used methodologies and assessment of their advantages and disadvantages .......................................................................................................... 180

Appendix 3. Integrated keywords ............................................................................ 196


v | P a g e EXPERT GROUP ON EVALUATION METHODOLOGIES FOR THE INTERIM AND EX-POST EVALUATIONS OF HORIZON 2020

Identif y

Acronyms

EC European Commission

ECO Economic

ENV Environmental

ERC European Research Council

EU European Union

FET Future and Emerging Technologies

FNR Fundamental Rights

FP Framework Programme

FPs Framework Programmes

GS Google Scholar

H2020 Horizon 2020

IPRs Intellectual Property Rights

MSCA Marie Skłodowska-Curie Actions

n.a. not available

NSEUCs Needs of society to EU citizens

OECD Organisation for Economic Co-operation and Development

PDAEs Policy Documents Appointed Experts

PEINPDs Priorities from EU and international key policy documents

REA Research Executive Agency

RTs ReTweets

SM Social Media

SMEs Small Medium Enterprises

SOC Social

SSH Social sciences and humanities

TAAEs Text Analysis Appointed Experts

TRLs Technology Readiness Levels

TT Trending topics

UN United Nations

WoS Web of Science

WHO World Health Organization

WP Work Programme

WPs Work Programmes


6 | P a g e EXPERT GROUP ON EVALUATION METHODOLOGIES FOR THE INTERIM AND EX-POST EVALUATIONS OF HORIZON 2020

Identif y

Executive summary

INTRODUCTION Background of assessment

One of the objectives of the interim evaluation of Horizon 2020 as stated in Article 32(3) of the Regulation (EU) No 1291/2013 of the European Parliament and of the Council is to assess the progress of the different parts of Horizon 2020 against ‘the achievements (at the level of results and progress towards achieving an impact, based, where applicable, on the indicators outlined in Annex II of the specific programme) of the objectives of Horizon 2020 and continued relevance of all related measures’.

The EC expert group on Methodologies developed and applied several methodologies for assessing the relevance of Horizon 2020. This was done by comparing the degree of matching1 between keywords extracted from the establishing act and the work programmes against keywords extracted from international and the EU policy documents, social media, and patents and publications.

Methodologies used for the assessment of relevance

According to the European Commission Better Regulation "Toolbox"2 the notion of relevance is the following: ‘Relevance looks at the relationship between the needs and problems in society and the objectives of the intervention. As things may change over time, certain objectives may be met or superseded; needs and problems change and new ones arise. The assessment of relevance could be defined as the process determining whether the original objectives of the Framework Programme are still relevant and how well they still correspond to present needs and challenges. The assessment was structured around three basic questions concerning the institutional perspective, the citizens’ perspective, and the scientific and technological perspective: Q1: Is the programme in line with the EU and international priorities? Q2: Is Horizon 2020 in line with the needs of the EU citizens? Q3: How well adapted is Horizon 2020 to the subsequent technological or scientific advances? These questions guided the development of four methodologies for the assessment of the relevance:

an expert exploratory approach using computer-based content which entails using computer-aided text mining as a quick tool for detecting the key topics in the legal documents of the framework programme for the purpose of the human assessment of relevance of the framework programme.

an expert exploratory approach using human content analysis which entails using a pre-selected group of independent experts who have high professional thematic knowledge and experience, in order to extract concepts and

1 If the keyword was present, explicitly or implicitly it was concluded that the keyword was ‘matched’. Implicit presence refers to the situation when the keyword is not directly mentioned, but the intervention allows its inclusion in the broad line of activities. 2 http://ec.europa.eu/smart-regulation/guidelines/tool_42_en.htm



Identif y

keywords reflecting challenges for the society based on the capabilities of the human mind.

a text mining approach which entails automatic extraction of topics: (1) from the legal documents of the framework programme, and (2) from the set of Strategic Policy documents published by the EU and other international institutions. Topics that are extracted by an automatic procedure are compared for the purpose of assessment of relevance of the framework programme.

a social media approach – which entails using social media (e.g. Facebook, Twitter, Wikipedia, YouTube) as a source to identify the most relevant social needs and technological and scientific advances as expressed by the citizens. This is done by the means of searching pre-selected key words (top down approach) or examining trending topics (bottom up approach).

When applied to Horizon 2020, the following texts were considered as a basis for keyword matching: (i) Council Decision of 3 December 2013 establishing the specific programme implementing Horizon 2020 - (2013/743/EU) (ii) the 2014 – 2015 Work Programme; (iii) the 2016 – 2017 Work Programme. The following texts were used to identify relevant keywords for the EU and international priorities: European Commission (2010). Europe 2020, A strategy for smart, sustainable and

inclusive growth, Communication from the Commission, Brussels, COM (2010) 2020 final.

Juncker, J.-C. (2014). A New Start for Europe: My Agenda for Jobs, Growth, Fairness and Democratic Change. Political Guidelines for the next European Commission. Opening Statement in the European Parliament Plenary Session. Strasbourg, 15 July 2014.

European Commission (2015). Communication from the Commission to the European Parliament and the Council, The Paris Protocol – A blueprint for tackling global climate change beyond 2020. /COM/2015/081 final/2. Brussels, 4.3.2015.

Gurría, A. (2015). “21-21”: A Proposal for Consolidation and Further Transformation of the OECD.

United Nations, General Assembly, Seventieth session. (2015). Transforming Our World: The 2030 Agenda for Sustainable Development (A/RES/70/1).

World Health Organisation (2015). Health in 2015: from MDGs, Millennium Development Goals to SDGs, Sustainable Development Goals. ISBN 978 92 4 156511 0. World Health Organization 2015.

G7 Leaders (2016). G7 Ise-Shima Leaders’ Declaration, G7 Ise-Shima Summit, 26-27 May 2016.

Several sources were used to identify the needs of the EU citizens: e.g. websites specialized for expressing opinions of European citizens, Eurobarometer documents, and social media, such as Twitter, Facebook, Wikipedia and YouTube. In order to assess keywords relevant for subsequent technological and scientific advances, the analysis used several specialized websites that track the development of disruptive technologies, as well as the bibliometric analysis using Google Scholar and social media, such as Twitter, Facebook, Wikipedia and YouTube



Identif y

APPLICATION OF THE METHODOLOGIES TO THE RELEVANCE QUESTIONS

Q1: Is Horizon 2020 in line with the EU and international priorities? Step 1. Identification of the EU and international priorities

By using an exploratory analysis by experts, as well as a computer-based content analysis (text mining), the relevant documents were used as knowledge sources and analysed. As a result keywords and phrases which correspond to indicators of the EU and international priorities were extracted from the Strategic Policy documents published by the EU and other international institutions.

Step 2. Assessment of whether the framework programme is relevant to the identified the EU and international priorities

Firstly, the expert exploratory approach using human content analysis for the assessment of relevance was used in order to investigate whether the extracted keywords and phrases used as indicators of the EU and international priorities are present in the Council Decision 2013/743/EU and the 2014-2015 and the 2016-2017 work programmes. The comparison was conducted while considering the different pillars and components of Horizon 2020. As far as the extent of the matching of the Council Decision 2013/743/EU and the work programmes with the extracted keywords and phrases is concerned, the following degrees of relevance were identified: (none, very low (matching up to 25% of the keywords), low (from 25% up to 50%), medium (from 50% up to 75%), and high (from 75% to 100%); N.A. – not applicable).

Secondly, the text mining approach for the assessment of relevance was used in order to compare the coverage of the topics and keywords identified with the results of the text mining of the Strategic Policy documents and the Council Decision 2013/743/EU.

Q2: Is Horizon 2020 in line with the needs of the EU citizens? Step 1. Identification of societal needs

The experts used an exploratory analysis which investigates the most important channels/ways that the European citizens use for making their voices heard in the EU, and then produced related keywords and phrases. As a result, key needs of the EU citizens were identified and prioritised in accordance with the four categories of societal needs, i.e.: social, economic, environmental and fundamental rights. Additionally, the experts also used the exploratory analysis along with the computer-based content analysis (text mining) to investigate the recent EU Barometer surveys and EC consultation reports, and in doing so, they identified, based on societal trends and concerns, key emerging needs of the EU citizens and recorded the most important keywords and phrases. Then, the experts, using their thematic knowledge, identified an extensive set of keywords organised according to the different pillars and components of Horizon 2020, which was based on the understanding of the expert group’s members of the relevant documents.

Secondly, the socialmedia content analysis approach was used for the identification of the needs of the EU citizens expressed by the latter through social media, resulting in the identification of appropriate keywords and phrases.

Step 2. Assessment of relevance

Firstly, the expert exploratory approach using human content analysis for the assessment of relevance was used in order to investigate whether the extracted keywords and phrases used as indicators for the needs of the EU citizens are present in the relevant documents.



Identif y

The comparison was conducted according to different pillars of Horizon 2020 and cross-cutting issues. These indicated the following levels of relevance related to the degrees of matching of Horizon 2020 and work programmes with the extracted keywords and phrases: none, very low (matching up to 25% of the keywords), low (from 25% up to 50%), medium (from 50% up to 75%), and high (from 75% to 100%); N.A. – not applicable.

Secondly, the social media approach for the assessment of relevance was deployed for comparing the extracted keywords and phrases used as indicators of the needs of the EU citizens with Horizon 2020 official Twitter accounts and funded projects’ websites.

Q3: How well adapted is Horizon 2020 to the subsequent technological or scientific advances?

Step 1. Identification of subsequent technological or scientific advances

Through the experts’ exploratory approach, the knowledge of experts in specific thematic areas was used to collect scientific and technological advances from various relevant sources. Keywords and phrases were subsequently delineated.

Additionally, a bibliometric analysis was conducted using Google Scholar to investigate the degree to which subsequent technological and scientific advances are reflected in the academic literature. Based on these results, keywords, phrases and topics were formulated.

Finally, a social media content analysis was performed for the identification of subsequent technological and scientific advances, as expressed by citizens through social media (Twitter, Facebook, Wikipedia and YouTube), and keywords and phrases were further delineated.


The expert exploratory approach using human content analysis for the assessment of relevance was used in order to investigate whether the extracted keywords and phrases used as indicators of the subsequent technological or scientific advances are present in the Council Decision 2013/743/EU and the 2014-2015 and the 2016-2017 work programmes. The comparison was conducted according to the different pillars of Horizon 2020 and cross-cutting issues. These indicated the following levels of relevance related to the degrees of matching of Horizon 2020 and work programmes with the extracted keywords and phrases: none, very low (matching up to 25% of the keywords), low (from 25% up to 50%), medium (from 50% up to 75%), and high (from 75% to 100%); N.A. – not applicable. Table - Tabulation of methodology use per question

Q1 Q2 Q3 Step 1. Identification of the EU and international priorities


Step 1. Identification of societal needs

Step 2. Assessment of

relevance

Step 1. Identification of subsequent technological or scientific advances


relevance

Expert exploratory human-content

Text mining approach

(only for the Horizon 2020 establishment act)

NOT APPLICABLE

NOT APPLICABLE NOT APPLICABLE

Social media NOT APPLICABLE

NOT APPLICABLE



Identif y

KEY FINDINGS – in different sections of Horizon 2020 In this part of the summary, the key findings are presented across different sections of Horizon 2020. Since text mining and social media were applied only partially, this part of the document is focusing to the experts’ exploratory approach using human content analysis: (1) examples of the keywords that are covered in legal documents are presented – in most of the cases for Horizon 2020 establishment act, and (2) detailed report on the keywords that are not covered in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes are presented. Social media and text mining results are elaborated in the sections: Key findings – by sections, and Conclusions on methodologies. In their work, experts identified if the keywords reflecting the EU and International priorities, the needs of EU citizens, and the subsequent technological and scientific advances are directly or indirectly present in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes. The relevance methodologies were not applicable in the case of Pillar 1-Excellent Science components (i.e. ERC, FET, MSCA, research infrastructures) because of its bottom-up nature. The key finding of the following sections of Horizon 2020 is presented in the following section:

II.1.1. Information and Communication Technologies II.1.2. Nanotechnologies, Advanced Materials, Advanced Manufacturing and

Processing, and Biotechnology II.1.3. Space II.2. Access to Risk Finance II.3. Innovation in SMEs III.1. Health, Demographic Change and Wellbeing (SC1) III.2. Food Security, Sustainable Agriculture and Forestry, Marine, Maritime and

Inland Water Research and the Bioeconomy (SC2) III.3. Secure, Clean and Efficient Energy (SC3) III.4. Smart, Green and Integrated Transport (SC4) III.5. Climate Action, Environment, Resource Efficiency and Raw Materials (SC5) III.6. Europe in a Changing World - Inclusive, Innovative and Reflective Societies

(SC6) III.7. Protecting Freedom and Security of Europe and its Citizens (SC7) IV. - Spreading Excellence and Widening Participation V. – Science with and for Society VI.1. Cross-cutting Issues: International Cooperation VI.2. Cross-cutting Issues: Social Sciences and Humanities VI.3. Cross-cutting Issues: Gender

II.1.1. Information and Communication Technologies

In the case of II.1.1, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a medium degree covered in the Horizon 2020 establishment act, 2014-2015, and in the 2016-2017 work programmes. The needs of EU citizens are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. The subsequent



Identif y

technological and scientific advances are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: boost investment; digital single market; digital technologies; digital technologies and online services as a horizontal policy; enable innovative business models; enhance enabling technology; growth and jobs; high-speed internet; increase competitiveness; innovation 2.0 and entrepreneurship; smart grids using ICTs; strengthen innovation chain; and improve research and innovation. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes:: 5G; adaptive design; age appropriate design; Big data; Bluetooth; business; clicks versus information; cloud computing; computer; data; data anonymization; data quality; data storage; data technology; datification; digital; Domotica; encryption; home apps; ICT; information; innovation; internet; jobs; media; mobile; network; photonics; privacy; programming; quantum; robotics; security; semantic; services; SMEs; social media; technology; web; wireless. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: Big Data; Speech technology; App for Syrian refugees; Advanced machine learning; Quantum computing; App bipolar disorder; Reliable ultrasonic testing device; Software alternative to pesticides; Alert4all system; Detector lethal asbestos fibers; environmental impact online calculator. However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: bridge digital divides; cybercrime; develop an accessible, open, interoperable, reliable and secure cyberspace; digital skills; enhance digital literacy; personal data protection; and promote digital adoption for improved quality of life. All of the relevant keywords are found in both 2014-2015 and 2016-2017 work programmes.

II.1.2. Nanotechnologies, Advanced Materials, Advanced Manufacturing and Processing, and Biotechnology

In the case of H2020 section ‘Nanotechnologies, Advanced Materials, Advanced Manufacturing and Processing, and Biotechnology’, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities, the needs of EU citizens and the keywords related to the subsequent technological and scientific advances are to a high degree covered in the Horizon 2020 documentation, both in the establishment act as well as the 2014-2015 and the 2016-2017 work programmes.

For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: improve research and innovation; strengthen innovation chain; increase competitiveness; enhance enabling technology; stimulate investment in new technologies. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act: smart insulation; vacuum insulation; aerogel insulation; biological building materials; phase change materials; nanotechnologies; graphene; molecular; micro sensors; environment;



Identif y

nanopharmaceutical; efficiency; electronics; photonics; Nano devices; nanomaterials; nano systems; Nano risk; coating; laser; fabrication composites; metallurgy; hybrid; insulating; low carbon; carbine; innovation; biomaterials; biotechnology; bioengineering; biodegradable; bio manufacturing; architecture; KET; cell; bioprocessing; sustainability; genomics; molecular; Biosystems; biocatalysts; bioinformatics; meta-genomics; LCA; SME; efficient; lightweight; 4.0; circularity; industry; pharmaceutical. Additionally, the following keywords were found in 2014-2015 and 2016-2017 work programmes: nature-based solutions; eco-innovation; miniaturisation; GMO; robotics. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: organic semiconductor; black phosphorus; grapheme structures; 2D materials; nanosensors and the internet of nanothings; phenolic carbon; tissue engineering; biofibre-based pyjamas.

However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: nature-based solutions; eco-innovation; miniaturisation; GMO and robotics. However, as it was above mentioned, these keywords were found in in 2014-2015 and 2016-2017 work programmes.

II.1.3. Space

In the case of II.1.3, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act, 2014-2015, and in the 2016-2017 work programmes. The needs of EU citizens are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords. For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: research and innovation; increase competitiveness; enabling technology. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: navigation; geoinformation; climate; satellite; Copernicus; Galileo; surveillance; tracking; weather; safety; telecommunications; environment. The methodology did not reveal any keywords related to space and subsequent technological and scientific advances. All of the relevant keywords are found in Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes.

II.2. Access to Risk Finance In the case of II.2, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act, 2014-2015, and in the 2016-2017 work programmes. The needs of EU citizens are to



Identif y

a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords. For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: boost investment; increase competitiveness; fund absorption; effective financial instruments; loans or guarantees with greater risk capacity; creation of innovative start-ups. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: SME; finance; small enterprise; business; banking; competitiveness; agreements; stakeholders; transparency.

However, the following keywords from the key strategic documents and needs of EU citizens were not covered in Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: micro-enterprise; crisis; tax; transparency.

II.3. Innovation in SMEs

In the case of II.2, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act, 2014-2015, and in the 2016-2017 work programmes. The needs of EU citizens are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords. For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: improve research and innovation; strengthen innovation chain; boost investment; increase competitiveness; improve business environment; entrepreneurship; creation of innovative start-ups. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: innovation; new enterprises; low unemployment; skills; digital economy; industry 4.0; entrepreneurship; education knowledge transfer; competitiveness; SMEs.

All of the relevant keywords are found in Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes.

III.1. Health, Demographic Change and Wellbeing (SC1)

In the case of SC1, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a low degree covered in both the Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes. The needs of EU citizens are to a high degree covered in the Horizon 2020 establishment act – 2013 and the 2016 -2017 work programme and to a medium degree to the 2014-2015



Identif y

work programme. When looking at the subsequent technological and scientific advances, the identified keywords are to a low degree covered in the Horizon 2020 establishment act while to a very low degree covered in the 2014-2015 and the 2016-2017 work programmes.

For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: hunger. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act: neighbourhood; health literacy; poverty; health service; health literacy; health; population; ageing; living standard; sustainable lifestyle; eco-innovation; climate change; biodiversity; urban green; urban; environment; demographic; change; medicine; sustainable; human; patient; disease; cancer; cardiovascular; prevention; care; migration; employment; illness; vaccines; public; diversity. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: vaccines; communicable diseases.

However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: access to safe, nutritious and sufficient food by all people including infants; promote Universal Health Coverage; reinforcing response to public health emergencies and antimicrobial resistance; end epidemics of AIDS, tuberculosis, malaria and neglected tropical diseases; combat hepatitis, water-borne diseases and other communicable diseases; end preventable deaths of newborns and children; reduce neonatal mortality; adequate and equitable sanitation and hygiene; democratic change; attract talent to cope with the demographic challenges; ensure access to sexual and reproductive health-care services for family planning; information and education; integration of reproductive health into national strategies and programmes; healthy lives; promote well-being at all ages; support research and development of vaccines and medicines; affordable essential medicines and vaccines; universal health coverage; access to essential health-care services; access to safe, effective, quality, affordable essential medicines; vaccines for all; reduce maternal mortality ratio; reduce premature mortality from non-communicable diseases; prevention, treatment and promote mental health and well-being; reduce deaths and injuries from road traffic accidents; sustainable development and lifestyles in harmony with nature; provide information and awareness for sustainable development and lifestyles in harmony with nature; urban regeneration; deprived; therapy access; sustainable health; lifestyle quality; biophilia; poor indoor environment; health impact; indoor pollution; green housing; sustainable living; indoor environment; nature-based solutions; greening cities; sustainable health; sustainable neighbourhood; eco-districts; ecological sites; ecological architecture; biological architecture; equilibrium of natural and built environment; biological building materials; hybrid ventilation; high performance ventilation; insulation; phase change materials; passive house; vertical gardens; city farming; green spaces; environment quality; green infrastructure; blue infrastructure; age appropriate design; multi generation living; sustainable urban environment; temperature increase; wellbeing; dignity; Optogenetics; Systems Metabolic Engineering; Molecular methods; human capital; reproductive health.

The following keywords were not covered in the 2014-2015 work programme: hunger; access to safe, nutritious and sufficient food by all people including infants; promote Universal Health Coverage; reinforcing response to public health emergencies and



Identif y

antimicrobial resistance; end epidemics of AIDS, tuberculosis, malaria and neglected tropical diseases; combat hepatitis, water-borne diseases and other communicable diseases; end preventable deaths of new-borns and children; reduce neonatal mortality; adequate and equitable sanitation and hygiene; democratic change; attract talent to cope with the demographic challenges; ensure access to sexual and reproductive health-care services for family planning, information and education; integration of reproductive health into national strategies and programmes; healthy lives; promote well-being at all ages; support research and development of vaccines and medicines; affordable essential medicines and vaccines; universal health coverage; access to essential health-care services; access to safe, effective, quality, affordable essential medicines; vaccines for all; reduce maternal mortality ratio; reduce premature mortality from non-communicable diseases; prevention, treatment and promote mental health and well-being; reduce deaths and injuries from road traffic accidents; sustainable development and lifestyles in harmony with nature; provide information and awareness for sustainable development and lifestyles in harmony with nature; urban regeneration; deprived; neighbourhood; therapy access; sustainable health; lifestyle quality; biophilia; living standard; poor indoor environment; indoor pollution; green housing; sustainable living; indoor environment; nature-based solutions; greening cities; sustainable health; sustainable neighbourhood; eco-districts; ecological sites; sustainable lifestyle; ecological architecture; eco-innovation; biological architecture; biodiversity; equilibrium of natural and built environment; biological building materials; hybrid ventilation; high performance ventilation; insulation; phase change materials; passive house; urban green; vertical gardens; city farming; green spaces; environment quality; green infrastructure; blue infrastructure; age appropriate design; multi generation living; sustainable urban environment; Optogenetics; Systems Metabolic Engineering; Molecular methods; human capital; employment; reproductive health; communicable diseases.

Finally, the following keywords were not covered in the 2016-2017 work programme: hunger; access to safe, nutritious and sufficient food by all people including infants; promote Universal Health Coverage; reinforcing response to public health emergencies and antimicrobial resistance; end epidemics of AIDS, tuberculosis, malaria and neglected tropical diseases; combat hepatitis, water-borne diseases and other communicable diseases; end preventable deaths of new-borns and children; reduce neonatal mortality; adequate and equitable sanitation and hygiene; democratic change; attract talent to cope with the demographic challenges; ensure access to sexual and reproductive health-care services for family planning, information and education; integration of reproductive health into national strategies and programmes; promote well-being at all ages; support research and development of vaccines and medicines; affordable essential medicines and vaccines; universal health coverage; access to essential health-care services; access to safe, effective, quality, affordable essential medicines; vaccines for all; reduce maternal mortality ratio; reduce premature mortality from non-communicable diseases; prevention, treatment and promote mental health and well-being; reduce deaths and injuries from road traffic accidents; sustainable development and lifestyles in harmony with nature; provide information and awareness for sustainable development and lifestyles in harmony with nature; urban regeneration; deprived; neighbourhood; therapy access; lifestyle quality; biophilia; living standard, poor indoor environment; indoor pollution; green housing; sustainable living; indoor environment; nature-based solutions; greening cities; sustainable neighbourhood; eco-districts; ecological sites; sustainable lifestyle; ecological architecture; eco-innovation; climate change; biological



Identif y

architecture; biodiversity; equilibrium of natural and built environment; biological building materials; hybrid ventilation; high performance ventilation; insulation; phase change materials; passive house; urban green; vertical gardens; city farming; green spaces; environment quality; green infrastructure; blue infrastructure; age appropriate design; multi generation living; sustainable urban environment; urban; temperature increase; dignity; Optogenetics; Systems Metabolic Engineering; Molecular methods; human capital; employment; reproductive health; communicable diseases.

III.2. Food Security, Sustainable Agriculture and Forestry, Marine, Maritime and

Inland Water Research and the Bioeconomy (SC2)

In the case of SC2, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities, the needs of the EU citizens and the subsequent technological and scientific advances are to a high degree covered in both the Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes.

For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: food safety; agricultural productivity; sustainable food production systems; hunger; access to safe, nutritious and sufficient food by all people including infants; investment in rural infrastructure; investment in agricultural research and extension services; investment in plant and livestock gene banks; enhance agricultural productive capacity in developing countries; enhance agricultural productive capacity in least developed countries; enhance conservation and sustainable use of oceans and their resources by implementing international law; address impacts of ocean acidification; small-scale food producers (including women, indigenous peoples, family farmers, pastoralists and fishers); ensure sustainable consumption and production patterns; resilient agricultural practices; increase productivity and production; maintain ecosystems that strengthen capacity for adaptation to climate change; maintain ecosystems that progressively improve land and soil quality; regulate harvesting; end overfishing, illegal, unreported and unregulated fishing and destructive fishing practices; implement science-based management plans to restore fish stocks; sustainable use of marine resources; sustainable management of fisheries, aquaculture and tourism; proper functioning of food commodity markets; limit extreme food price volatility; maintain genetic diversity of seeds; maintain cultivated plants; maintain farmed and domesticated animals; maintain wild species. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act: sustainability; trade; agriculture; biodiversity; food; security; sustainable; forestry; inland; water; bio economy; rural; hunger; land; development; adaptation; fisheries; sea; production; consumption; impact; biological; ecological; biotechnology; healthy; consumers; acidification; degradation; ocean; biomass; fresh; drinking; waste; ecosystem; aquatic; overfishing; circular economy; desalination. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: ocean acidification; rural development; biodiversity; sustainable intensification; high nature value farming; agroecology; agricultural innovation; agroforestry.

The following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act, the 2014-2015 work programme and the 2016-2017 work programme: poverty; industrialisation; GMO.



Identif y

III.3. Secure, Clean and Efficient Energy (SC3) In the case of SC3, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 documentation, both in the establishment act as well as the 2014-2015 and the 2016-2017 work programmes. The needs of EU citizens are to a medium and low degree covered in the Horizon 2020 documentation. When looking at the subsequent technological and scientific advances, the identified keywords are to a medium degree covered in the Horizon 2020 establishment act while to a high degree covered in the 2014-2015 and the 2016-2017 work programmes. For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: improve research and innovation; invest in innovation in energy technologies; strengthen innovation chain; boost investment; decarbonising the economy; reduce greenhouse gas emissions; increase renewable sources use; increase energy efficiency; smart grids using ICTs; competitiveness; facilitate energy investments; clean energy; diversify the energy sources; European energy market open; reform energy policy; enhance energy efficiency of buildings. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act: energy; transport; growth; competitiveness; market; buildings; urban; innovation; building; renovation; consumers; energy; efficiency; energy efficiency; refurbishment; performance; end-user; emissions; industry; urban; renewable; sustainable development; renewable energy; engagement; renewables; climate change; biodiversity; circular economy; batteries; awareness; recycling; ageing; risk; adaptation; mitigation; vulnerability; industry; heating; pollution; cradle to cradle; resources; sustainable; sustainability; forest; biofuel; nuclear; fusion; carbon; electricity; minerals; resilience; storage; wind; bioenergy; geothermal; CO2; fossil; consumption; alternative; solar; decarbonisation; hydro; implicit: sustainable living; eco-districts; building integrated renewables; on site renewables; renewable; smart grid; environment quality; eco-innovation; green infrastructure; eco design; health; sustainable urban environment; resource use inefficiency; energy generation; building energy. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: pollution; emissions; low-carbon; smart buildings; sustainable; renewable; electricity; local; sustainability AND energy. The following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: reduce energy dependency; European Energy Union; Poor indoor environment; indoor environment; indoor pollution; health impact; green housing; nature-based solutions; greening cities; sustainable health; sustainable neighbourhood; smart appliances; robotics; smart insulation; vacuum insulation; aerogel insulation; high performance ventilation; insulation; climate policies; biological building materials; hybrid ventilation; phase change materials; passive house; home apps; artificial intelligence; adaptive design; buildings recyclability; district energy efficiency; urban green; vertical gardens; city farming; green spaces; blue infrastructure; age appropriate design; lifestyle quality; green transport; overconsumption; temperature increase; LCA; hydrocarbons; sustainable lifestyle; ecological architecture; equilibrium of natural and



Identif y

built environment; NZEB; prosument; smart cities; local AND energy; pollution AND energy; ecological architecture; green housing. The following keywords were not covered in the 2014-2015 work programme: sustainable living; nature-based solutions; greening cities; sustainable neighbourhood; eco-districts; sustainable lifestyle; ecological architecture; biological architecture; biodiversity; building integrated renewables; district energy efficiency; NZEB; building energy; circular economy; energy flexibility; batteries; climate policies; smart insulation; vacuum insulation; aerogel insulation; biological building materials; hybrid ventilation; high performance ventilation; phase change materials; passive house; Domotica; home apps; smart appliances; robotics; artificial intelligence; technology; adaptive design; buildings recyclability; source separation; urban green; vertical gardens; city farming; green spaces; environment quality; green infrastructure; blue infrastructure; age appropriate design; multi generation living; lifestyle quality; ageing; temperature increase; vulnerability; green transport; cradle to cradle; eco design; resource use inefficiency; overconsumption; deforestation; fusion; hydrocarbons; minerals; pollution AND energy; ecological architecture; green housing. Finally, the following keywords were not covered in the 2016-2017 work programme: sustainable building; carbondioxide; health impact; green housing; sustainable living; nature-based solutions; greening cities; sustainable health; sustainable neighbourhood; eco-districts; ecological sites; sustainable lifestyle; ecological architecture; eco-innovation; biological architecture; district energy efficiency; circular economy; energy flexibility; on site renewables; climate policies; smart insulation; vacuum insulation; aerogel insulation; biological building materials; hybrid ventilation; high performance ventilation; insulation; phase change materials; passive house; Domotica; home apps; robotics; artificial intelligence; adaptive design; buildings recyclability; pretreatment; source separation; urban green; vertical gardens; city farming; green spaces; environment quality; green infrastructure; blue infrastructure; age appropriate design; multi generation living; population; lifestyle quality; environment; ageing; temperature increase; mitigation; vulnerability; green transport; energy generation; cradle to cradle; resource use inefficiency; overconsumption; local AND energy; pollution AND energy; ecological architecture; green housing.

III.4. Smart, Green and Integrated Transport (SC4) In the case of SC4, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities, needs of EU citizens and the subsequent technological and scientific advances are to a high degree covered in the Horizon 2020 documentation, both in the establishment act as well as the 2014-2015 and the 2016-2017 work programmes. For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: improve research and innovation; strengthen innovation chain; boost investment; decarbonising the economy; reduce greenhouse gas emissions; increase renewable sources use; modernise transport sector; develop green technologies; smart grids using ICTs; increase competitiveness. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act: transport; travel; oil;



Identif y

decarbonisation; infrastructure; mobility; trade; network; greenhouse; emissions; climate; electric cars; green transport; electric vehicles; e-mobility; low carbon; environment; carbon footprint; emissions; GHGs; CO2; decarbonisation; green transport; smart; green; integrated; transport; electric ; vehicle; road; emissions; greenhouse; aviation; mobility; congestion; efficiency; electrical; logistics; noise; traffic; pollution; climate; urban; regional; car; clean air. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: low carbon transport; Urban mobility; Electric vehicles; air AND transport; pollution AND transport; climate AND transport. However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: battery. The following keywords were not covered in the 2014-2015 work programme: smart grids using ICTs. Finally, the following keywords were not covered in the 2016-2017 work programme: none.

III.5. Climate Action, Environment, Resource Efficiency and Raw Materials (SC5) In the case of SC5, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities and the needs of EU citizens are to a high degree covered in the Horizon 2020 documentation, both in the establishment act as well as the 2014-2015 and the 2016-2017 work programmes, while to a medium degree covered when looking at the subsequent technological and scientific advances. For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: improve research and innovation; address water scarcity; combat poaching and trafficking of protected species; conservation of mountain ecosystems; conserve and sustainably use biodiversity and ecosystems; conserve coastal and marine areas; decarbonising the economy; develop green technologies; develop policies to decrease vulnerability; develop policies to improve countries' capacity to adapt to the impacts of climate change adaptation; eliminate dumping; ensure conservation, restoration and sustainable use of terrestrial and inland freshwater ecosystems; fight climate change; fight global warming; improve resource efficiency in consumption and production; improve water quality; increase energy efficiency; increase recycling; increase water-use efficiency; limit temperature increase; low-carbon economy; minimize release of hazardous chemicals and materials; prevent environmental degradation; prevention, reduction, recycling and reuse of waste; protect biodiversity loss; reduce greenhouse gas emissions; reduce pollution; reduce untreated wastewater; sustainable use of resources; tackle resource scarcity; waste safe reuse; wastewater treatment; water efficiency; water harvesting; water recycling and reuse technologies. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act: adaptation; biodiversity; biological architecture; circularity; climate action; climate change; climate policies; deforestation; eco-innovation; ecosystems; energy efficiency; environment quality; equilibrium of natural and built environment; footprint; green infrastructure; greenhouse; high performance ventilation; hybrid ventilation; material loops; mineral resources; mitigation; overconsumption; pollution; raw material; recycling; renewable energy; resilience; resource; resource assessment; reusable; risks; scarcity; sustainability; sustainable lifestyle; sustainable living; sustainable urban



Identif y

environment; tele detection; temperature increase; vulnerability; waste; circular economy; waste management. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: smart city; alternative electricity; circular economy; climate change adaptation; climate change mitigation; climate services; fossil fuel; fuel cell vehicles; raw material; resource efficiency; solar energy; sustainable lifestyles; waste management. However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: adequate and equitable sanitation and hygiene; combat desertification; Desalination; integrate ecosystem and biodiversity values into national and local planning; reduce food waste at the retail and consumer levels; sustainable withdrawals and supply of freshwater; aerogel insulation; city farming; desertification; eco-districts; footprint; greening cities; home apps; indoor environment; indoor pollution; insulation; micropollutants; passive house; rural; smart appliances; smart insulation; sustainable neighbourhood; vertical gardens; artificial photosynthesis; data base on climate change; energy storage; green light-emitting diodes; nuclear fusion; plasma pre-treatment detecting deformations and movements landslides; recyclable thermoset plastics. The following keywords were not covered in the 2014-2015 work programme: sustainable withdrawals and supply of freshwater; integrate ecosystem and biodiversity values into national and local planning; combat desertification; aerogel insulation; city farming; desertificstion; eco-districts; footprint; greening cities; home apps; indoor environment; indoor pollution; insulation; micropollutants; passive house; recycling; reusable; rural; smart appliances; smart insulation; sustainable neighbourhood; vertical gardens; artificial photosynthesis; data base on climate change; energy storage; green light-emitting diodes; nuclear fusion; plasma pre-treatment detecting deformations and movements landslides; recyclable thermoset plastics. Finally, the following keywords were not covered in the 2016-2017 work programme: aerogel insulation; buildings recyclability; city farming; eco-districts; food waste; greening cities; home apps; indoor environment; indoor pollution; insulation; micropollutants; passive house; recycling; reusable; rural; smart appliances; smart insulation; sustainable neighbourhood; teledetection; vertical gardens; artificial photosynthesis; data base on climate change; energy storage; geen light-emitting diodes; nuclear fusion; plasma pre-treatment detecting deformations and movements landslides; recyclable thermoset plastics.

III.6. Europe in a Changing World - Inclusive, Innovative and Reflective Societies (SC6)

In the case of SC6, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a low degree covered in the Horizon 2020 establishment act – 2013 and the 2016-2017 work programmes and to a medium degree covered in the and the 2014-2015 work programme. The needs of EU citizens are to a medium degree covered in the Horizon 2020 establishment act – 2013 and to a high degree covered to the 2014-2015 work programme and the 2016-2017 work programme. When looking at the subsequent technological and scientific advances, the identified



Identif y

keywords are to a high degree covered in the Horizon 2020 and the 2016-2017 work programme and to a medium degree covered in the 2014-2015 work programme.

For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: enhance performance of education systems; boost for new jobs; create economic opportunities for workers, consumers and firms; creation of innovative start-ups; decent jobs; digital skills; effective, accountable and inclusive institutions; gender equality; global leadership in strategic sectors with high-value jobs; growth and jobs; increase development of new skills, poverty; live in dignity; promote peaceful and inclusive societies; promote women empowerment; well-being; youth employment. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act: open and innovative government; innovation talents and skills; innovative citizens; green innovation; public sector innovation; health innovation; inclusive innovation; co-operation; access and participation; citizens empowerment; collaboration; partnerships with citizens. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: Digital Archive of Intangible Cultures; social inclusion; migration management; migration; natural heritage; demography.

However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: adopt social protection policies; combat tax fraud; deeper and fairer economic and monetary union; deeper and fairer internal market; develop and integrate capital markets; digital technologies; elimination of market distorting measures; enhance transparency with stakeholders and lobbyists; enhancement of market function; Europe a Global Actor to foreign policy; Europe a Global Actor to security and defence; extreme poverty; fight corruption; fixing the banking sector; improve anti-corruption capacity building; improve financing of the economy; increase competitiveness; increase employment; personal data protection; policies to promote local culture and products; poverty; provide access to justice; put debt on a sustainable path; recover public trust, confidence and hope; reinforce higher education; simplify regulations; stability of single currency; strengthen law enforcement cooperation; younger generation back to work; knowledge creation and diffusion; innovation; public sector reform; cosmopolitan Prioritarianism (Ethical Guideline); income inequality; poverty.

The following keywords were not covered in the 2014-2015 work programme: adopt social protection policies; create economic opportunities for workers, consumers and firms; creation of innovative start-ups; deeper and fairer economic and monetary union; develop and integrate capital markets; effective, accountable and inclusive institutions; elimination of market distorting measures; enhance transparency with stakeholders and lobbyists; enhancement of market function; fixing the banking sector; global leadership in strategic sectors with high-value jobs; improve financing of the economy; increase development of new skills; live in dignity boost for new jobs; personal data protection; policies to promote local culture and products; promote peaceful and inclusive societies; provide access to justice; put debt on a sustainable path; recover public trust, confidence and hope; stability of single currency; younger generation back to work; Cosmopolitan Prioritarianism (Ethical Guideline); demography.

Finally, the following keywords were not covered in the 2016-2017 work programme: enhance performance of education systems; boost for new jobs; create economic



Identif y

opportunities for workers, consumers and firms; creation of innovative start-ups; decent jobs; deeper and fairer economic and monetary union; digital skills; digital technologies; elimination of market distorting measures; enhance transparency with stakeholders and lobbyists; enhancement of market function; Europe a Global Actor to foreign policy; Europe a Global Actor to security and defence; extreme poverty; fixing the banking sector; gender equality; global leadership in strategic sectors with high-value jobs; growth and jobs; improve anti-corruption capacity building; improve financing of the economy; increase competitiveness; increase employment; policies to promote local culture and products; poverty; put debt on a sustainable path; reinforce higher education; simplify regulations; stability of single currency; strengthen law enforcement cooperation; well-being of societies; open and innovative government; innovative citizens; green innovation; Digital Archive of Intangible Cultures; Cosmopolitan Prioritarianism (Ethical Guideline).

III.7. Protecting Freedom and Security of Europe and its Citizens (SC7)

In the case of SC7, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a low degree covered in the Horizon 2020 establishment act – 2013 and in the 2016-2017 work programme and to a medium degree covered in the 2014-2015 work programme. The needs of EU citizens are to a medium degree covered in both the Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes. When looking at the subsequent technological and scientific advances, the identified keywords are to a high degree covered in both the Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes.

For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: digital single market; anticipate events early; combat all forms of organized crime; cooperation in defence procurement; Cybercrime; develop an accessible, open, interoperable, reliable and secure cyberspace; effective, accountable and inclusive institutions; ensure equal opportunity; eradicate human trafficking; fundamental rights; human trafficking; Justice; manage migration; mutual trust; organised crime; promote appropriate legislation, policies and action against discrimination; promote peaceful and inclusive societies; promote policy on legal migration; provide access to justice; Radicalisation; recover public trust; reduce exposure and vulnerability to climate-related extreme events; reduce exposure and vulnerability to other economic, social and environmental shocks and disasters; reduce trafficking; reinforce the work of FRONTEX; resilience of the poor and those in vulnerable situations; strengthen law enforcement cooperation; stronger security and defence; Terrorism; well-being of societies. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act: aid; borders; citizenship; cooperation; crime; crisis; cybercrime; defence; democracy; diplomacy; energy; environment; equality; extremism; fairness; freedom; human rights; humanitarian; inclusion; inequality; infection; jobs; migration; mobility; peace; police; policy; poverty; privacy; protection; radical; radicalization; refugees; rights; rights; risks; security; social exclusion; surveillance; terrorism; threat; trafficking; transparency; workforce. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: surveillance; privacy AND secure; extremism; military equipment.



Identif y

However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: a strong common asylum policy; age discrimination; belief discrimination; cross-border crime; deal irregular migration; discrimination for disability; discrimination for people belonging to a minority; eliminate discriminatory laws; eradicate forced labour; eradicate modern slavery; ethnic discrimination; fight abuse; fight all forms of violence; fight corruption; fight torture of children; improve anti-corruption capacity building; nationality discrimination; penalise human traffickers; personal data protection; prevent and treat of substance abuse, narcotic drug abuse and alcohol; prohibit and eliminate of child labour; prohibit and eliminate recruitment and use of child soldiers; put European Border Guard Teams into action; racial discrimination; rapid border interventions; reduce exploitation; reduce illicit financial and arms flows; reduce inequalities of outcome; religion discrimination; sex discrimination; sexual orientation discrimination; Smuggling; strengthen the recovery and return of stolen assets; anti-islam; asylum; bioweapon; bullying; countermeasures; discrimination; encryption; extreme right; immigration; liberties; preparedness; refugee Schengen area; violence; violence against women; xenophobia; flexicurity; violence.

The following keywords were not covered in the 2014-2015 work programme: a strong common asylum policy; cooperation in defence procurement; effective, accountable and inclusive institutions; eliminate discriminatory laws, policies and practices; ensure equal opportunity; eradicate forced labour; eradicate human trafficking; eradicate modern slavery; fight abuse; fight all forms of violence; fight torture of children; improve anti-corruption capacity building; penalise human traffickers; prevent and treat of substance abuse, narcotic drug abuse and alcohol; prohibit and eliminate of child labour; prohibit and eliminate recruitment and use of child soldiers; promote appropriate legislation, policies and action against discrimination; promote policy on legal migration; provide access to justice; recover public trust, confidence and hope; reduce exploitation; reduce illicit financial and arms flows; reduce inequalities of outcome; strengthen the recovery and return of stolen assets; aid; anti-islam; asylum; bioweapon; citizenship; countermeasures; democracy; diplomacy; extreme right; human rights; infection; poverty; preparedness; refugee rights; refugees; violence against women; workforce; xenophobia; flexicurity.

Finally, the following keywords were not covered in the 2016-2017 work programme: ): a strong common asylum policy; age discrimination; anticipate events early; belief discrimination; discrimination for disability; discrimination for people belonging to a minority; effective, accountable and inclusive institutions; eliminate discriminatory laws, policies and practices; ensure equal opportunity; eradicate human trafficking; eradicate modern slavery; ethnic discrimination; fight abuse; fight all forms of violence; fight torture of children; mutual trust; nationality discrimination; penalise human traffickers; prohibit and eliminate of child labour; prohibit and eliminate recruitment and use of child soldiers; promote appropriate legislation, policies and action against discrimination; promote peaceful and inclusive societies; put European Border Guard Teams into action; racial discrimination; recover public trust, confidence and hope; reduce exploitation; reduce inequalities of outcome; reinforce the work of FRONTEX; religion discrimination; sex discrimination; sexual orientation discrimination; stronger security and defence; anti-islam; asylum; bioweapon; bullying; democracy; diplomacy; equality; extreme right; fairness; humanitarian; inclusion; inequality; infection; jobs; liberties; migration; preparedness; refugee rights; refugees; social exclusion; violence against women; xenophobia; flexicurity; military equipment.



Identif y

IV. - Spreading Excellence and Widening Participation

In the case of IV, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act, to a medium degree covered in the 2014-2015 work programme and to a low degree covered in the 2016-2017 work programme. The needs of EU citizens are to a high degree covered in the H2020 establishment act while to a medium degree covered in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advances were not applicable for analysis to the section IV of Horizon 2020. For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: improve research and innovation; strengthen innovation chain; enhance performance of education systems; reinforce higher education; better match labour supply and demand; boost for new jobs. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act: excellence; participation; mobility; access; career; education; talent; business; job; twinning; teaming; collaboration; leverage; opportunity; qualification; skills; capacity; research; innovation; citizen; gender. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: No keywords identified by the methodology.

However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: scholarships; qualification. The following keywords were not covered in the 2014-2015 work programme: reinforce higher education; better match labour supply and demand; boost for new jobs; career; talent; opportunity; scholarships; qualification; skills; citizen. Finally, the following keywords were not covered in the 2016-2017 work programme: strengthen innovation chain; enhance performance of education systems; reinforce higher education; better match labour supply and demand; boost for new jobs; career; job; scholarships; qualification; skills.

V. – Science with and for Society In the case of V, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act and in the 2016-2017 work programme, while to a medium degree covered in the 2014-2015 work programme. The needs of EU citizens are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advances were not applicable for analysis to the section V of Horizon 2020. For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: boost for new jobs; digital technologies and online services as a horizontal policy; enhance performance of education systems; provide information and awareness for sustainable development and lifestyles in harmony with nature; reinforce higher education; strengthen innovation chain; improve research and innovation. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon



Identif y

2020 establishment act: academic; access; accessible; awareness; careers; citizen; education; employment; engagement; ethics; gender; inspiration; integrity; investment; knowledge; layman; observatory; open; preparedness; public; schools; science; social; society; transdisciplinary. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act: No keywords identified by the methodology. However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: complete tertiary education. The following keywords were not covered in the 2014-2015 work programme: complete tertiary education; digital technologies and online services as a horizontal policy; provide information and awareness for sustainable development and lifestyles in harmony with nature; observatory; preparedness. Finally, the following keywords were not covered in the 2016-2017 work programme: complete tertiary education; digital technologies and online services as a horizontal policy; observatory; preparedness; investment; inspiration.

VI.1. Cross-cutting Issues: International Cooperation In the case of the H2020 cross-cutting issue related to international cooperation, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU an International priorities are to a high degree covered in the Horizon 2020 establishment act – 2013 and the 2016-2017 work programme, while they are to a medium degree covered in the 2014-2015 work programme. The needs of EU citizens are to a medium degree covered in the Horizon 2020 establishment act – 2013, while they are to a high degree covered in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords. For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: simplify regulations; strengthen capacity for early warning, risk reduction and management of national and global health risks in developing countries; keep markets open; encourage climate-resilient sustainable development by promoting international cooperation; ease access to markets; better cooperation with third countries; enhance international support for implementing effective and targeted capacity-building in developing countries to support national plans to implement all the Sustainable Development Goals; recruitment, development, training and retention of the health workforce in developing countries, least developed countries and small island developing States; transboundary cooperation in water resources management; international support to developing countries; international cooperation on and access to science, technology and innovation. The following keywords related to the needs of EU citizens were found in the Horizon 2020 establishment act: open markets; EU enlargement; public-private partnerships (PPPs); international relations; EU enlargement.

The following words emerging from key strategic documents worldwide and social needs of EU citizens were not found in the Horizon 2020 establishment act: free trade



Identif y

Agreement with the USA; encourage trade liberalization efforts; fight protectionism; trust in transatlantic relations; assist developing countries in attaining long-term debt sustainability through coordinated policies; duty-free and quota-free market access on a lasting basis for all least developed countries; development, transfer, dissemination and diffusion of environmentally sound technologies to developing countries; international cooperation and capacity-building support to developing countries in water- and sanitation-related activities and programmes; international trade; production-consumption systems; NGOs. The following words emerging from key strategic documents worldwide and social needs of EU citizens were not found in the 2014-2015 work programme: free trade Agreement with the USA; encourage trade liberalization efforts; fight protectionism; ease access to markets; assist developing countries in attaining long-term debt sustainability through coordinated policies; duty-free and quota-free market access on a lasting basis for all least developed countries; recruitment, development, training and retention of the health workforce in developing countries, least developed countries and small island developing States. The following words emerging from key strategic documents worldwide and social needs of EU citizens were not found in the 2016-2017 work programme: free trade Agreement with the USA; encourage trade liberalization efforts; fight protectionism; duty-free and quota-free market access on a lasting basis for all least developed countries; recruitment, development, training and retention of the health workforce in developing countries, least developed countries and small island developing States.

VI.2. Cross-cutting Issues: Social Sciences and Humanities In the case of the H2020 cross-cutting issue related to social sciences and humanities, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU an International priorities are to a high degree covered in the Horizon 2020 establishment act – 2013, 2014-2015 and the 2016-2017 work programme. The needs of EU citizens are to a medium degree covered in the Horizon 2020 establishment act – 2013, 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords. The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: improve research and innovation; strengthen innovation chain; poverty; justice; fundamental rights; mutual trust; fight abuse; reduce trafficking; hunger; access to safe, nutritious and sufficient food by all people including infants; access to essential health-care services; resilience of the poor and those in vulnerable situations. The following keywords related to the needs of EU citizens were found in the Horizon 2020 establishment act: employment; unemployment; inclusion; wellbeing; poverty; sharing; education; social inclusion.

The following keywords emerging from key strategic documents worldwide were found in the 2014-2015 and 2016-2017 work programmes: improve research and innovation; access to essential health-care services; access to safe, nutritious and sufficient food by all people including infants; mutual trust; fundamental rights; personal data protection; strengthen innovation chain; poverty; justice. The following keywords related to the needs of EU citizens were found in the 2014-2015 and 2016-2017 work programmes: sharing; education; employment; social inclusion; inclusion; wellbeing; unemployment; poverty.



Identif y

The following keywords emerging from key strategic documents worldwide were not found in the Horizon 2020 establishment act: personal data protection; reduce exploitation; fight all forms of violence; fight torture of children; extreme poverty; nationality discrimination; sex discrimination; racial discrimination; ethnic discrimination; religion discrimination; belief discrimination; discrimination for disability; age discrimination; sexual orientation discrimination; discrimination for people belonging to a minority; adopt social protection policies. The following keywords emerging from key strategic documents worldwide were not found in the in the 2014-2015 and 2016-2017 work programmes: fight abuse; reduce exploitation; reduce trafficking; fight all forms of violence; fight torture of children; extreme poverty; hunger; nationality discrimination; sex discrimination; racial discrimination; ethnic discrimination; religion discrimination; belief discrimination; discrimination for disability; age discrimination; sexual orientation discrimination; discrimination for people belonging to a minority. The following keywords related to the needs of EU citizens were not found in the 2014-2015 and 2016-2017 work programmes: cultural integration.

VI.3. Cross-cutting Issues: Gender

In the case of the H2020 cross-cutting issue related to gender, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU an International priorities are to a medium degree covered in the Horizon 2020 establishment act – 2013, and high degree in 2014-2015 and the 2016-2017 work programme. The needs of EU citizens are to a high degree covered in the Horizon 2020 establishment act – 2013, 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords. The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: gender equality; give women equal rights to economic resources; give women equal rights to financial services, inheritance and natural resources; women’s equal opportunities for leadership; capacity-building such as education and training of women in Science, Technology, Engineering and Mathematics (STEM) fields; sex discrimination; promote and protect women’s rights; support policy for the promotion of gender equality and the empowerment of all women and girls; promote women empowerment; protect women migrants; recognize the role of women as key actors to ensure peace and security; give women equal rights to ownership; give women equal rights control over land and property; eliminate violence against women and girls. The following keywords related to the needs of EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: gender; societal; vulnerability; education; awareness.

The following keywords emerging from key strategic documents worldwide were not found in the Horizon 2020 establishment act: recognize the role of women as key actors to ensure peace and security; give women equal rights to ownership; give women equal rights control over land and property; eliminate violence against women and girls; protect women migrants, adolescent girls, maternal mortality, sexual and gender based violence. All of the keywords related to the needs of EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes.



Identif y

KEY FINDINGS – by Question Q1 - The application of different methodologies for analysing the relevance of Horizon 2020 compared to the EU and international priorities revealed a high degree of matching at all the three levels of assessment (Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes) for the following sections of H2020: II.1.2. Nanotechnologies, Advanced Materials, Advanced Manufacturing and Processing, and Biotechnology, II.1.3. Space, II.2. Access to Risk Finance, II.3. Innovation in SMEs, III.2 Food Security, Sustainable Agriculture and Forestry, Marine, Maritime and Inland Water Research and the Bioeconomy, III.3 Secure, Clean and Efficient Energy, III.4 Smart, Green and Integrated Transport, III.5 Climate Action, Environment, Resource Efficiency and Raw Materials, and V. Science with and for Society. A medium degree of matching was found for the sections: II.1.1. Information and Communication Technologies, VI.2 Cross-cutting Issues: Social Sciences and Humanities. Moreover, a low to a very low degree of matching was found for SC1 i.e. III.1 Health, Demographic Change and Wellbeing. Due to different degrees of matching between the three levels of assessment in the case of both III.7 Secure Societies – Protecting Freedom and Security of Europe and its Citizens and III.6 Europe in a Changing World - Inclusive, Innovative and Reflective Societies a low to a medium degree of matching was noticed. For both VI.1 Cross-cutting Issues: International Cooperation and VI.3. Cross-cutting Issues: Gender the degree of matching was from a medium to a high one. Finally, for section IV. Spreading Excellence and Widening Participation a high differentiation ranging from a low to a high degree of matching was found. Throughout the Societal Challenges, the application of different methodologies for analysing the relevance of Horizon 2020 compared to the EU and international priorities showed a high degree of matching at all the three levels of assessment in the case of SC2, SC3, SC4 and SC5, while it showed a medium match for the SC6 and a low to a medium match for SC7. A very low to a low match was only found for SC1. The section of H2020 concerning science with and for society showed a high degree of match at all the three levels of assessment. For section IV. Spreading Excellence and Widening Participation a high differentiation ranging from a low to a high degree of matching was found. In the cross-cutting issues, a high degree of relevance was found for VI.3. Cross-cutting Issues: Gender while for VI.2 Cross-cutting Issues: Social Sciences and Humanities and for VI.1 Cross-cutting Issues: International Cooperation the degree of matching was from a medium to a high one. Q2 -The application of different methodologies for analysing the relevance of Horizon 2020 compared to the needs of the EU citizens showed a high degree of matching at all the three levels of assessment (Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes) in all but one section of the LEITS (except Access to Risk Finance section where it was found to be medium). Throughout the Societal Challenges, the application of different methodologies for analysing the relevance of Horizon 2020 compared to the needs of the EU citizens showed a high degree of matching at all the three levels of assessment in the case of SC2, SC4 and SC5), and a medium match for the SC7. A high degree of match with the Horizon 2020 establishment act – 2013 and a medium match with the 2014-2015 and the 2016-



Identif y

2017 work programmes was found in SC 1. Interestingly, in two SCs (SC3 and SC6), it was found that the match with the Horizon 2020 establishment act – 2013 (Medium) was lower than the match with the 2014-2015 and the 2016-2017 work programmes A low match with the 2014-2015 and the 2016-2017 work programmes was only found in SC3. The section of H2020 concerning science with and for society showed a high degree of match at all the three levels of assessment, while a high and medium degree of match in was found in the spreading excellence and widening participation section. In the cross-cutting issues, a high degree of match at all the three levels of assessment was found in the social sciences and humanities and gender, while a medium and high degree of match was found in the international cooperation. Q3 – Different methodologies for analysing the relevance of Horizon 2020 compared to the needs of the EU citizens were not applicable to several sections: II.2. Access to Risk Finance, II.3 Innovation in SMEs, IV. Spreading Excellence and Widening Participation, V. Science with and for Society, VI.1 Cross-cutting Issues. 1.International Cooperation, VI.2 Cross-cutting Issues. Social Sciences and Humanities (SSH). The methodology used for the extraction of the keywords indicating the subsequent technological or scientific advances did not generate the related keywords. For example, the keywords identified as the indicators of the subsequent technological or scientific advances were not related to social science and humanities. The following sections of the LETIS showed a high degree of matching at all the three levels of assessment (Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes): II.1.1. Information and Communication Technologies (ICT); II.1.2. Nanotechnologies, Advanced Materials, Advanced Manufacturing and Processing, and Biotechnology (NANO); and II.1.3. Space. Throughout the Societal Challenges, the application of different methodologies for analysing the relevance of Horizon 2020 compared to the needs of the EU citizens showed a high degree of matching at all the three levels of assessment in the case of SC2, SC4, and SC7, while at SC5 a medium degree of matching was identified. Only in SC6, a high degree of matching was found in the Horizon 2020 establishment act – 2013, and the 2016-2017 work programme, while a medium degree of matching was found in the 2014-2015 work programme. In the cross-cutting issues, a high degree of matching at all the three levels of assessment was found in the Cross-cutting Issue: Gender.

CONCLUSIONS ON METHODOLOGIES The relevance methodologies were not applicable in the case of Pillar 1-Excellent Science components (i.e. ERC, FET, MSCA, research infrastructures.) because of its bottom-up nature etc. The use of an exploratory approach using human content analysis was very successful because despite being quite resource intensive (using a pre-selected group of, in this case 17, independent experts who have high professional thematic knowledge and



Identif y

experience), showed highest applicability throughout the analysis of keywords in all basic questions (Q1, Q2 and Q3), whereby other methods had their applicability limitations. The use of an exploratory approach using computer-based content was conducted experimentally on the legal documents related to one of the societal challenges. The method has several limitations. The most important limitation is that the topic extraction from the legal documents missed several important topics, which was not the case with the exploratory approach using human content analysis. At this stage this method should be refined in the future, and its effectiveness heavily depends on the quality of the algorithm used for the topic extraction. The use of text mining was quite successful in the case of Q1, since in both cases the topics are not present in the Horizon 2020 establishment act, and the same topics were also identified by the use of exploratory approach using human content. However, the text mining is useful if two pairs of strategic documents are compared, in this case (1) legal documents of the framework programme, and (2) Strategic Policy documents published by the EU and other international institutions. The use of social media – The use of social media was quite successful in the case Q2 and successful in Q3, as in both questions it has revealed the importance of the vivid discussion on the social media as indicator of the social needs or the values posed into particular subsequent scientific and technological advances. It has allowed to capture the voices of lay citizens who might not be represented by the experts or stakeholders usually consulted.

Degree of matching of H2020 keywords and phrases

H2020 sections

Q1: Is Horizon 2020 in line with the EU and international

priorities?

Q2: Is Horizon 2020 in line with the needs of the EU

citizens?

Q3: How well adapted is Horizon 2020 to the

subsequent technological or scientific advances?

II.1.1. Information and Communication Technologies (ICT)

Medium (A; B; C) High (A; B; C) High (A; B; C)

II.1.2. Nanotechnologies, Advanced Materials, Advanced Manufacturing and Processing, and Biotechnology (NANO)

High (A; B; C) High (A; B; C) High (A; B; C)

II.1.3. Space High (A; B; C) High (A; B; C) N.A. II.2. Access to risk finance High (A; B; C) Medium (A; B; C) N.A. II.3 Innovation in SMEs High (A; B; C) High (A; B; C) N.A. III.1 Health, Demographic Change and Wellbeing

Low (A, B) Very low (C) High (A; C); Medium (B) Low (C); Very low (A; B)

III.2 Food Security, Sustainable Agriculture and Forestry, Marine, Maritime and Inland Water Research and the Bioeconomy


III.3 Secure, Clean and Efficient Energy High (A; B; C) Medium (A)

Low (B; C) Medium (A) High (B; C)

III.4 Smart, Green and Integrated Transport

High (A; B; C) High (A; B; C) High (A; B; C) III.5 Climate Action, Environment, Resource Efficiency and Raw Materials

High (A; B; C) High (A; B; C) Medium (A; B; C)

III.6 Europe in a Changing World - Inclusive, Innovative and Reflective Societies

Low (A; C); Medium (B) Medium (A) High (B; C) High (A; C); Medium (B)

III.7 Secure Societies – Protecting Freedom and Security of Europe and its Citizens

Low (A; C); Medium (B) Medium (A; B; C) High (A; B; C)



Identif y

Degree of matching of H2020 keywords and phrases

H2020 sections

Q1: Is Horizon 2020 in line with the EU and international

priorities?

Q2: Is Horizon 2020 in line with the needs of the EU

citizens?


subsequent technological or scientific advances?

IV. Spreading Excellence and Widening Participation

High A Medium B

Low C

High (A) Medium (B; C) N.A.

V. Science with and for Society High (A; C) Medium (B) High (A; B; C) N.A.

VI.1 Cross-cutting Issues. 1.International cooperation

High (A; C) Medium (B)

Medium (A) High (B; C) N.A.

VI.2 Cross-cutting Issues. Social Sciences and Humanities (SSH) Medium (A; B; C) High (A; B; C) N.A.

VI.3. Cross-cutting Issues. Gender Medium (A) High (B; C) High (A; B; C) N.A.

A- Horizon 2020 establishment act – 2013 B – Horizon 2020 Work Programme 2014-15 C - Horizon 2020 Work Programme 2016-17 Notes: (i) The approach was not applicable to the following sections of Horizon 2020: I.1. European Research Council I.2. Future and Emerging Technologies; I.3. Marie Skłodowska-Curie Actions; I.4. Research Infrastructures, including e-Infrastructures; and for Q3. for several other sections; (ii) The comparison was conducted according Horizon 2020 sections, indicating the following levels of relevance related to the degrees of matching of Horizon 2020 and work programmes with the extracted keywords and phrases: none, very low (matching up to 25% of the keywords), low (from 25% up to 50%), medium (from 50% up to 75%), and high (from 75% to 100%); (iii) N.A. – not applicable

Examples of keywords/phrases

H2020 sections Successful matching Unsuccessful matching II.1.1. Information and Communication Technologies (ICT)

Big Data; Speech technology; Quantum computing Cybercrime; Digital literacy; Personal data protection


Nanotechnologies; Graphene; Nanosafety Nature-based solutions; Eco-innovation; GMO

II.1.3. Space Navigation; Geoinformation; Safety -

II.2. Access to Risk Finance Effective financial instruments; Loans or

guarantees with greater risk capacity; Creation of innovative start-ups

Micro-enterprise; Crisis; Transparency

II.3 Innovation in SMEs Industry 4.0; Entrepreneurship; Education & knowledge transfer -

III.1 Health, Demographic Change and Wellbeing Antimicrobial resistance; Ageing; Migration Health impact ;Reproductive health; Biodiversity


Sustainable agriculture; Biodiversity; Rural development Poverty; GMO; everyday products

III.3 Secure, Clean and Efficient Energy Energy efficiency; Renewable energy; Smart grids Sustainable neighbourhood; District energy

efficiency; Health impact III.4 Smart, Green and Integrated Transport Electric vehicles; Aviation; Carbon Footprint Smart grids using ICTs

III.5 Climate Action, Environment, Resource Efficiency and Raw Materials

Excellent science; Raw materials; Climate change adaptation

Green light-emitting diodes; Desertification & Deforestation; Artisanal mining


Employment; Innovation; Education Stability of single currency; Transparency with stakeholders; Debt on a sustainable path


Fundamental rights; Human trafficking; Manage migration

Age discrimination; Personal data protection; Bioweapon

IV. Spreading Excellence and Widening Participation Twinning; Teaming; Collaboration; Performance of education systems; Better match

labour supply and demand; Boost for new jobs

V. Science with and for Society Improve research and innovation; Boost for new jobs; Strengthen innovation chain

Complete tertiary education; Digital technologies and online services as a horizontal policy; provide

information for sustainable development

VI.1 Cross-cutting Issues. 1.International Cooperation

International support to developing countries; International cooperation on and access to science,

technology and innovation; Transboundary cooperation in water resources management;

Free trade Agreement with the USA; Encourage trade liberalization efforts; Fight protectionism

VI.2 Cross-cutting Issues. Social Sciences and Humanities (SSH) Unemployment; Inclusion; Wellbeing Extreme poverty; Nationality discrimination

VI.3. Cross-cutting Issues. Gender Gender equality; Equal opportunities for leadership; STEM equality

Role of women to ensure peace; Violence against women and girls; Protect women migrants

Notes: (i) The approach was not applicable to the following sections of Horizon 2020: I.1. European Research Council I.2. Future and Emerging Technologies; I.3. Marie Skłodowska-Curie Actions ; I.4. Research Infrastructures, including e-Infrastructures; and for Q3. for several other sections



Identif y

CHAPTER 1: Introduction

Horizon 2020 was established for the period from 1 January 2014 to 31 December 2020 through the Regulation (EU) No 1291/2013 of the European Parliament and of the Council. Horizon 2020 is helping to achieve the Union objectives of smart, sustainable and socially inclusive Europe of the Europe 2020 Strategy by focusing on three mutually reinforcing priorities of excellent science, industrial leadership and societal challenges.

In accordance with Article 32(3) and 32(4) of the Regulation (EU) No 1291/2013 of the European Parliament and of the Council, ‘the European Commission (EC) shall carry out an interim and an ex-post evaluation of Horizon 2020, assisted by independent experts selected on the basis of a transparent process’. In addition, it specifies that the progress of the different parts of Horizon 2020 should be assessed against “the achievements (at the level of the results and progress towards achieving an impact) of the objectives of Horizon 2020 and continued relevance of all related measures”.

Historically, the main purpose of FPs evaluations as requested in the legal basis has been to assess management, implementation and efficiency with regard to their research output. However, there has been limited data collection on the relevance of these FPs in relation to policy goals such as the Europe 2020 Strategy as well as current needs of citizens and society at large. In particular, the need to assess the social impacts of Horizon 2020 and its relevance in relation to policy goals such as the Europe 2020 Strategy, raises new challenges in terms of data, methodologies and analysis. This requires a major step-up in the evaluation approach. Advances in the digital economy may help with this. Big Data, open access to these data, the linking and visualisation of these data, altmetrics, the increasing importance of social media as means of expressions of citizen opinions and other new trends expand the traditional limits of monitoring and evaluation and offer new opportunities to assess the relevance of programmes in relation to the needs of society.

This report is focused on the presentation of the new methodologies that were developed by the expert group in order to assess the relevance of European research and innovation framework programmes, and particularly the contribution of Horizon 2020 to tackling key challenges for society and to the Europe 2020 Strategy. Therefore, the largest part of this publication develops this methodological contribution, including detailed description of how to implement each methodology, to end with a summary of how these methodologies were applied by the expert group for assessing the relevance of Horizon 2020 with some key findings.

Chapter 2 presents the notion of relevance. According to the European Commission Better Regulation "Toolbox"3, “relevance looks at the relationship between the needs and problems in society and the objectives of the intervention. As things may change over time, certain objectives may be met or superseded; needs and problems change and new ones arise”. Therefore, we define the assessment of relevance as the process to determine whether the original objectives of the Framework Programme are still relevant and how well they still correspond to present needs and challenges.

3 http://ec.europa.eu/smart-regulation/guidelines/tool_42_en.htm



Identif y

To this end, the assessment of relevance of the framework programmes has been structured upon three basic questions, related to the institutional perspective (Q1), the citizens’ perspective (Q2), and the science and technological perspective (Q3). Additionally, each question has been addressed through two sub-questions, related to 1) identifying policy priorities, citizens’ needs and scientific and technological advances, and 2) assessing if the framework programme is relevant to what has been previously identified. This general methodological framework is presented in Chapter 3.

Chapter 4 elaborates in depth on the process for the assessment of relevance of the European framework programmes and the plurality of methodologies required. Four methodologies have been developed:

an expert exploratory approach using computer-based content which entails using computer-aided text mining as a quick tool for detecting the key topics in the legal documents of the framework programme for the purpose of the human assessment of relevance of the framework programme.

an expert exploratory approach using human content analysis which entails using a pre-selected group of independent experts who have high professional thematic knowledge and experience, in order to extract concepts and keywords reflecting challenges for the society based on the capabilities of the human mind.

a text mining approach which entails automatic extraction of topics: (1) from the legal documents of the framework programme, and (2) from the set of Strategic Policy documents published by the EU and other international institutions. Topics that are extracted by an automatic procedure are compared for the purpose of assessment of relevance of the framework programme.

a social media approach – which entails using social media (e.g. Facebook, Twitter, Wikipedia, YouTube) as a source to identify the most relevant social needs and technological and scientific advances as expressed by the citizens. This is done by the means of searching pre-selected key words (top down approach) or examining trending topics (bottom up approach).

These methodologies have been developed to answer the three basic questions (Q1, Q2 and Q3) and sub-questions, as summarised below. Chapter 4 also includes a step-by-step description of its application as well as a summary of main advantages, disadvantages and strategies for the mitigation of the weaknesses.

Question 1: Is the framework programme in line with the EU and international priorities?

Sub-question 1.1: How to identify the EU and international priorities?

Methodologies developed to respond to Q1.1: experts’ exploratory approach and computer content analysis (text mining).

Result: keywords and phrases reflecting the EU and international priorities.

Sub-question Q1.2: How to asses if the framework programme is relevant to the identified EU and international priorities?

Methodologies developed to respond to Q.1.2: experts’ exploratory approach and computer content analysis (text mining).



Identif y

Result: conclusion regarding relevance of the framework programme to the EU and international priorities.

Question 2: Is the framework programme in line with the needs of society to EU citizens?

Sub-question Q2.1: How to identify the needs of society to EU citizens?

Methodologies developed to respond to Q2.1: experts’ exploratory approach; on-line content analysis (Eurobarometer surveys and EC consultation reports; and social media analysis); keywords experts’ exploratory approach; and computer content analysis (text mining).

Result: keywords and phrases reflecting the needs of society to EU citizens.

Sub-question Q2.2: How to asses if the framework programme is relevant to the identified needs of society to EU citizens?

Methodologies developed to respond to Q2.2: experts’ exploratory approach.

Result: conclusion regarding relevance of the framework programme to the identified needs of society to EU citizens.

Question 3: How well adapted is the framework programme to the subsequent technological or scientific advances?

Sub-question Q3.1 How to identify the subsequent technological or scientific advances?

Methodologies developed to respond to Q3.1: experts’ exploratory approach; bibliometric analysis; social media content analysis; and patent analysis.

Result: keywords and phrases reflecting the subsequent technological or scientific advances.

Sub-question Q3.2 How to asses if the framework programme is relevant to the identified subsequent technological or scientific advances?

Methodologies developed to respond to Q3.2: experts’ exploratory approach.

Result: conclusion regarding relevance of the framework programme to the identified subsequent technological or scientific advances.

The process of assessing the relevance of European research and innovation framework programmes is based mostly on a keyword approach (also containing phrases and topics), by comparing the degree of matching between keywords extracted from the Horizon 2020 establishing act (Council Decision 2013/743/EU) and work programmes (2014-15 and 2016-17) against keywords extracted from international and EU policy documents, social media, and patents and publications.

Several texts were used to identify relevant keywords for the EU and international priorities, including Europe 2020 Strategy, Junker pillars, UN Sustainable Development Goals, WHO Millennium Development Goals, among other. To identify the needs of the EU citizens, the sources used included websites specialized for expressing opinions of European citizens, Eurobarometer documents, and social media, such as Twitter, Facebook, Wikipedia and



Identif y

YouTube. For selecting the keywords relevant for subsequent technological and scientific advances, the analysis used several specialized websites that track the development of disruptive technologies, as well as the bibliometric analysis using Google Scholar and social media, such as Twitter, Facebook, Wikipedia and YouTube. These methodological steps can be found in the Chapter 4.

Following this detailed methodological development, Chapter 5 provides a synthesis of the results of the assessment of the relevance of Horizon 2020 when applying these methodologies. A summary of these key findings is presented in relation to some sections of Horizon 2020 (i.e. societal challenges, ICT, Innovation in SMEs, SWAFs, and crosscutting issues, among other). While this assessment has been exhaustive in length, the present publication contributes a glimpse of these results, as the main purpose is to thoroughly contribute solid methodological guidelines for current and future assessments of EU framework programmes.

Finally, the concluding chapter summarizes the methodological contribution for the assessment of relevance and an overview of the key findings.

The development of the methodologies was based on strong and effective teamwork approach. Open discussion between experts was encouraged while expression of comments and constructive criticism were also welcomed; thus, allowing for a clear insight into the specific margins and particular strengths of each developed methodology. Moreover, difficulties related to the heterogeneity of the specific composition of the expert group and others related to the time limitations were viewed as a strategic enabler for producing interdisciplinary teamwork. As a result, a great synergy occurred in the expert group as a consequence of the different expertise, complementary backgrounds and approaches which confirms that the whole is greater than the sum of its parts. In the same manner, the combination of the presented methodologies can provide the most valuable results by ensuring that limitations of one methodology are balanced by strengths of another. Methodologies that were developed are complementary in terms of collecting information by different groups (experts, academic community, and citizens); covering different scopes; achieving different depth of analysis; and applying different methods i.e. human expertise or automated processes like data mining software. Considering the above and depending on the resources to be committed (time, budget, personnel, software) as well as the required level of assessment one can identify and apply the most appropriate methodology or set of methodologies from the plethora of the developed methodologies for the assessment of relevance of European research and innovation framework programmes.



Identif y

CHAPTER 2: The notion of relevance from the European Commission perspective

From the European Commission’s perspective, RELEVANCE constitutes one of the five criteria that has to be assessed for the evaluation of interventions already in place. In particular, evaluation is defined as ‘an evidence-based judgment of the extent to which an intervention: has been effective and efficient, has been relevant given the needs and its objectives, has been coherent both internally and with other EU policy interventions, and has achieved EU added-value (European Commission, 2015a). All evaluations must assess the criteria of effectiveness, efficiency, coherence, relevance as well as EU added-value of an intervention. In Figure 1 and Figure 2 the connection of the criteria with the intervention logic are illustrated (European Commission, 2015b).

Figure 1: Conceptual representation of evaluation criteria and the intervention logic by European Commission (European Commission, 2015b)

According to European Commission (2015b): ‘relevance looks at the relationship between the needs and problems in society and the objectives of the intervention. As things may change over time, certain objectives may be met or superseded; needs and problems change and new ones arise. The wrong problem-drivers’ may have been identified during the impact assessment; incorrect assumptions may have been made about the cause and effect relationships; circumstances may have changed and the current needs are not the same as the ones considered when the intervention was designed’.

In order to assess relevance of an intervention, typical examples of relevance questions are:

To what extent is the intervention still relevant?

To what extent have the (original) objectives proven to have been appropriate for the intervention in question?

How well do the (original) objectives (still) correspond to the needs within the EU?



Identif y

How well adapted is the intervention to subsequent technological or scientific advances?

How relevant is the EU intervention to EU citizens? (European Commission, 2015b).

Relevance analysis is very important – because if an intervention does not help to address present needs or problems then it does not matter how effective, efficient or coherent it is – it is no longer appropriate. This is why there is a strong link between relevance analysis and the criteria of EU added value – which assesses whether action continues to be justified at the EU level (European Commission, 2015a).

How effective has the EU intervention been?

The evaluation analyses the progress made towards achieving the objectives of the intervention, looking for evidence of why, whether or how these changes are linked to the EU intervention.

The answer to this question goes further than showing if the intervention is on track. It seeks to identify the factors driving or hindering progress and how they are linked (or not) to the EU intervention.

How efficient has the EU intervention been?

The evaluation looks at both the costs and benefits of the EU intervention as they accrue to different stakeholders, identifying what factors are driving these costs/benefits & how these factors relate to intervention.

The answer to this question provides evidence on the actual costs and benefits, making it clear what can and what cannot be linked to the EU intervention. Efficiency analysis helps policy makers & stakeholders to draw conclusions if the costs of the intervention are proportionate to the benefits.

How relevant is the EU intervention?

The evaluation looks at the objectives of the EU intervention being evaluated in order to see how well they (still) match the (current) needs and problems.

The answer to this question aim to identify if there is any mismatch between the objectives of the intervention and the (current) needs or problems. This is key information which assists policy makers in deciding whether to continue, change or stop an intervention.

How coherent is the EU intervention internally & with other (EU) actions? The evaluation should look at how well the intervention works: i) internally and ii) with other EU interventions.

The answer to this question provides evidence of where and how EU interventions are working well together or point to areas where there are tensions.

What is the EU added-value of the intervention?

The evaluation considers arguments about the value resulting from EU interventions that is additional to the value that would have resulted from interventions initiated at regional or national levels by both public authorities and private sector.

The answer to this question responds to the subsidiarity analysis conducted in any related impact assessment. The analysis of EU added value is often limited to the qualitative, given the stated difficulties to identify a counter-factual.

Figure 2: Overview of the five evaluation criteria related to the evaluation process of European Commission (European Commission, 2015a)



Identif y

CHAPTER 3: General framework and methods for the assessment of the relevance of European research and innovation framework programmes

European funding for research activities was envisaged in the first Community Treaties, related to coal and steel and nuclear energy, and was extended in the early 1980s with the establishment of a European framework programme for research. Research policy was subsequently progressively integrated in the Treaties, to become a shared competence between the EU and its Member States. European funds for research and innovation activities are distributed between several interlinked EU programmes (Figure 3). For the current period (2014-2020), the main programme Horizon 2020 (the 8th FP for research and innovation) is fully dedicated to funding such activities across all policy fields (Reillon, 2015).

Figure 3: The continuity of the eight EU framework programmes for the research and innovation

From the first framework programme, established by the Council resolution of 25 July 1983, to Horizon 2020 the aim was to achieve scientific and technical objectives to be pursued at Community level in line with the current priorities that the society had to tackle.

In most of the cases, it is expected that the scientific research has a positive impact to the challenges of society. However, there is a time discrepancy present between the goals of the framework programmes and the challenges of society, since the goals of the framework programmes are set usually at the beginning of the programmes, while the challenges of society are changing over time. This process is presented in Figure 4, with the presumed number of social challenges (14 social challenges) and presumed time span (10 years). It is presumed that the framework programme is completely in line with the challenges in society in Year 1. However, some new challenges could occur in the societies that are highly important, but it is questionable if these challenges could be met with the specific scientific program, if they are not presumed to be part of it at the



Identif y

beginning. Considering the above, the assessment of relevance could be defined as the process to determine whether the original objectives of the framework programme are still relevant and to assess how well these initial challenges still match the present and future needs.

Challenges in society

Objectives of the FPs related to challenges of

society

Challenges of society in

Year 1


Year 3


Year 7

Challenges of society in Year 10

SOC-CHAL-1

SOC-CHAL-2

SOC-CHAL-3

SOC-CHAL-4

SOC-CHAL-5

SOC-CHAL-6

SOC-CHAL-7

SOC-CHAL-8

SOC-CHAL-9

SOC-CHAL-10

SOC-CHAL-11

SOC-CHAL-12

SOC-CHAL-13

SOC-CHAL-14

Figure 4: Relation of challenges in society and goals of the research and innovation framework programmes (hypothetical example)

The assessment of relevance of FPs can be structured upon three basic questions (Q1, Q2, and Q3) related to the institutional perspective, the citizens’ perspective, and the science and technological perspective accordingly. These questions further analysed in sub-questions are:

Q1: Is the framework programme in line with the EU and international priorities?

Q1.1 How to identify the EU and international priorities?

Q1.2 How to assess if the FP is relevant to the identified EU and international priorities?

Q2: Is the framework programme in line with the needs of society to EU citizens?

Q2.1 How to identify the needs of society to EU citizens?

Q2.2 How to assess if the FP is relevant to the identified needs of society to EU citizens?

Q3: How well adapted is the framework programme to subsequent technological or scientific advances?

Q3.1 How to identify the subsequent technological or scientific advances?



Identif y

Q3.2 How to assess if the FP is relevant to the identified subsequent technological or scientific advances?

These questions are presented in Figure 5.

Figure 5: The three basic questions and sub-questions which determine the framework for the assessment of relevance of European research and innovation framework

programmes



Identif y

CHAPTER 4: Process for the assessment of the relevance of European research and innovation framework programmes

4.1. General description

The process for the assessment of relevance of European research and innovation framework programmes is based on the keyword approach. The keyword approach is the basis for the assessment of relevance in the information science (McCrudden & Schraw, 2007). The ‘keyword’ in this text refers not only to the words, but also to the phrases.

In the proposed process of the assessment of relevance of European research and innovation framework programmes, the keywords were used as the indicators of: (i) the EU and international priorities based on key policy documents; (ii) the needs of society to EU citizens; (iii) the subsequent technological or scientific advances; and (iv) the topics in the Council Decision of 3 December 2013 (4); the 2014–2015 Work Programme adopted in the framework of the specific programme implementing Horizon 2020; and the 2016 – 2017 Work Programme also adopted in the framework of the specific programme implementing Horizon 2020.

The following process was used for the assessment of relevance of European research and innovation framework programmes:

STEP 1.1: Identification of the EU and international priorities (Q1.1)

o OBJECTIVE: to provide the answer to the question Q1.1: How to identify the EU and international priorities?

o METHODS USED: exploratory text analysis by experts and computer content analysis (text mining).

o RESULT: keywords and phrases as the indicators of the EU and international priorities.

STEP 1.2: Assessment if the framework programme is relevant to the identified EU and international priorities (Q1.2)

o OBJECTIVE: to provide the answer to the question Q1.2: How to assess if the Framework Programme is relevant to the identified EU and international priorities?

o METHODS USED: expert exploratory approach and computer content analysis (text mining).

(4) Council Decision of 3 December 2013 establishing the specific programme implementing Horizon 2020 - the Framework Programme for Research and Innovation (2014-2020) and repealing Decisions 2006/971/EC, 2006/972/EC, 2006/973/EC, 2006/974/EC and 2006/975/EC (2013/743/EU) (OJ L 347, 20.12.2013, p. 965).



Identif y

o RESULT: conclusion regarding relevance of the framework programme to the EU and international priorities.

STEP 2.1: Identification of the needs of society to EU citizens (Q1.2)

o OBJECTIVE: to provide the answer to the question Q2.1: How to identify the needs of society to EU citizens?

o METHODS USED: experts’ exploratory approach; on-line content analysis (Eurobarometer surveys and EC consultation reports; and social media analysis); keywords experts’ exploratory approach; and computer content analysis (text mining).

o RESULT: Keywords and phrases as the indicators of the needs of society to EU citizens.

STEP 2.2: Assessment if the Framework Programme is relevant to the identified needs of society to EU citizens (Q2.2)

o OBJECTIVE: to provide the answer to the question Q2.2: How to assess if the Framework Programme is relevant to the identified needs of society to EU citizens?

o METHOD USED: experts’ exploratory approach.

o RESULT: conclusion regarding relevance of the framework programme to the identified needs of society to EU citizens.

STEP 3.1: Identification of the subsequent technological or scientific advances (Q3.1)

o OBJECTIVE: to provide the answer to the question Q3.1: How to identify the subsequent technological or scientific advances?

o METHODS USED: expert’s exploratory approach; bibliometric analysis; on-line content analysis (social media); and patent analysis.

o RESULT: keywords and phrases as the indicators of the subsequent technological or scientific advances.

STEP 3.2: Assessment if the framework programme is relevant to the identified technological or scientific advances (Q3.2)

o OBJECTIVE: to provide the answer to the question Q3.2: How to assess if the framework programme is relevant to the identified subsequent technological or scientific advances?

o METHOD USED: expert’s exploratory approach.

o RESULT: conclusion regarding relevance of the framework programme to the identified subsequent technological or scientific advances.



Identif y

4.2. Methodologies for the identification of the: current priorities based on EU and international key policy documents (Q1.1); needs of society to EU citizens (Q1.2); and the subsequent technological and scientific advances (Q1.3)

4.2.1. Methodologies for the identification of the current priorities from EU and international key policy documents (Q1.1)

4.2.1.1. Exploratory text analysis by experts for the identification of the current priorities from EU and international key policy documents

Methodology: Exploratory text analysis by experts for the identification of the current priorities from EU and international key policy documents

Related Question: Q 1.1

General objective of the methodology:

The general objective of the proposed methodology is the identification of the priorities of society as explicitly or implicitly given in key EU and international policy documents by exploratory text analysis conducted by experts – in the form of keywords and phrases (concepts).

Specific objectives of the methodology:

The specific objectives for the development of the proposed methodology are to:

identify explicitly expressed key priorities of society from key EU and international policy documents;

identify implicitly expressed key priorities of society from key EU and international policy documents;

extract words, phrases and the like from EU and international policy documents by exploratory text analysis conducted by experts that depict the content of these documents and can be used for the assessment of relevance;

evaluate exploratory text analysis conducted by experts for classifying concepts, words, phrases and the like from EU and international policy documents; and

evaluate the value of extracted concepts, words, phrases and the like from EU and international policy documents by exploratory text analysis conducted by experts.

Why this methodology was selected to be developed?

This methodology was developed in order to examine and validate the strong hypothesis that the human understanding is important in analysing texts.

4.2.1.1.1. Short description of the methodology

The exploratory text analysis of key policy documents by experts is based on extracting concepts reflecting challenges for the society from the texts of the policy



Identif y

documents based on the capabilities of humans’ mind. The methodology can be applied by a well-organised group of experts with high level of professional experience and appropriate backgrounds with the issues raised under the policy documents. As the specific methodology constitutes a question-driven text analysis, the appointed experts while studying the policy documents shall have in mind question Q1.1 i.e. the identification of EU and international priorities. Having this in mind, they shall proceed to the extraction of ‘concepts’ reflecting the priorities. After appropriate coding, they will finally come up with a list of identified priorities for the society according to key policy documents. For the needs of this methodology concepts can reflect a single idea, or ideational kernel, regardless of whether it is represented by single word or a phrase.

4.2.1.1.2. Step-by-step description of the methodology

The methodology that is proposed is structured upon the following steps:

STEP 1: Selection of the strategic policy documents

The first step is to select the strategic policy documents that set up EU and global policy agendas. The selection of reliable, representative and valuable in content documents is of most importance. In order to ensure these requirements, the key policy documents must have been officially published by prominent international and European organizations and institutions. Indicatively, such international organizations and institutions are the United Nations (UN) and specialised agencies of UN e.g. World Health Organisation (WHO); the Organisation for Economic Co-operation and Development (OECD); and the European Commission (EC). The documents shall reflect the positions of these organisations and institutions on challenges that the society has to face directly or in the near future (5-15 years) in the light of sustainable development. The selection of strategic policy documents can be done by the working group under assistance of relevant institution i.e. European Commission.

STEP 2: Organisation of the working group

The second step is related to the organisation of the working group. The working group shall consist of text analysis appointed experts (TAAEs) who will work on text analysis tasks as well as policy documents appointed experts (PDAEs) who will be responsible for key policy documents. More particularly, TAAEs must be responsible for the preparation of summary texts of policy documents, the extraction of concepts based on human capabilities and the coding of concepts. PDAEs will be responsible for the validation of created summaries, the validation of extracted concepts, and the coding of concepts and the validation of the final list. For each policy document a PDAE will be defined e.g. PDAE No. 1 responsible for policy document No. 1 etc. The size of the working group must be proportionate to the length and content of the key policy documents to be analysed. Members of the working groups shall have the necessary cognitive abilities as well as the appropriate backgrounds with the issues raised under the policy documents and reflecting the priorities. When the volume of documents exceeds individuals’ abilities and as a result teamwork becomes essential, particular emphasis should be placed to coordination of work and application of same rules for the codification and/or extraction of concepts as well as for the validation purposes. To this end, clear and exhaustively



Identif y

formulated written instructions should be prepared and given to the members of the working group.

STEP 3: Study, re-presentation in tables of the strategic policy documents and validation of the selected texts

During third step all members of the working group start by familiarising themselves with the chosen documents. TAAEs start to decompose the documents with the focus on identifying the key challenges for society at European and international level. Then, TAAEs will present in a systematic way, e.g. in table formats (summary table 1 for key policy document 1 etc.), the text of each policy document that shall be considered for the subsequent extraction of key concepts (under step 4). Summary table of each policy document must reveal in a systematic way the challenges explicitly or implicitly included in the text representing each policy document. Completed summary tables of each document will be then circulated by the TAAEs to the corresponding PDAEs for his validation. Each PDAE responsible for a policy document will revise the summary table related to his appointed key policy document and send his comments back to the TAAEs. When validation process will be finished then the step 4 can follow.

STEP 4: Extraction of concepts related to the identification of the challenges for the society, re-presentation in the form of a consolidated table and validation of the consolidated table

During the fourth step, the TAAEs shall extract concepts reflecting priorities from the summary table based on their perceptual and cognitive skills. All extracted concepts related to key priorities for the society from the different policy documents shall be included in an integrated table prepared by the TAAEs. The layout of the template used is presented in Figure 6.

(a) (b) (c) (d) (e) (f) (g)

Sou

rce

Tim

e sp

an

Cha

lleng

e id

entif

ied

Det

ails

Con

cept

s Key

wor

ds/

key

phra

ses Main Dimension

(only one cross) Secondary Dimension

(more than one crosses)

ECO

N

SO

C

ENV

FNR

ECO

N

SO

C

ENV

FNR

…… …… …… …… …… + + + +

…… …… …… …… …… + + + +

…… …… …… …… …… + +

…… …… …… …… …… +

…… …… …… …… …… + + + +

…… …… …… …… …… + + +

…… …… …… …… …… + +

Figure 6: Layout of the template used for the extraction and representation of concepts reflecting EU and international priorities from strategic policy documents in the context

of STEP 4 of the exploratory text analysis conducted by experts



Identif y

As illustrated in Figure 6, this table can include the following information:

(a) Source: the strategic policy document that the specific concept (challenge) was extracted can be defined here.

(b) Time span: the time period that the document covers can be defined here.

(c) Challenge identified: the challenge identified in brief can be defined here.

(d) Details: more details on the specific challenge as given in the strategic policy document can be defined here.

(e) Concepts: for the needs of this methodology concepts can reflect a single idea, or ideational kernel, regardless of whether it is represented by single word or a phrase. This explanation on the concept has been given by (Carley, 1993) according to who ‘Examples of concepts are: friends, textual analysis, and likes to play golf’. The selected concepts can be also used for comparative analysis with the results from other methodologies e.g. the text mining analysis.

(f) Basic dimension: in order to facilitate the coding of the concepts the TAAEs shall provide information that mostly characterizes each identified concept. More particular, in this column indication of only one (the basic) dimension will be provided by TAAEs i.e. economic or social or environmental or related to fundamental rights shall be determined with a cross, and;

(g) Secondary dimension: due to ‘multidimensional character’ of challenges in this column further characterisation can be provided by TAAEs. Contrary to the previous selection, here TAAEs can provide with more than one selection by putting crosses in the related cells.

TAAEs will be initially responsible for extracting the concepts and providing their indication about the basic and secondary character of each concept in the consolidated table. Then the completed table shall be circulated by the TAAEs to the corresponding PDAE for his validation. Each PDAE responsible for a policy document will revise the entire consolidated table and send his comments back to the TAAEs. After the validation process is finished, then step 5 can follow.

STEP 5: Coding of extracted concepts, extraction of the final list of priorities and validation of the list

During the fifth step, TAAEs will code the validated concepts based on common instructions and will finally prepare the list of PEINPDs. Then the final list shall be circulated by the TAAEs to the PDAEs for their validation. PDAEs will validate the list based on their previously conducted task as well as their expertise.

After the validation procedure the final list of identified challenges for the society from strategic policy documents will be delivered. The exploratory text analysis conducted by experts for identifying EU and international priorities from strategic policy documents is presented in Figure 7.



Identif y

Step 1 Selection of reliable, representative and

valuable strategic policy documents

Step 2 Organisation of the working group

Step 3 Study, re-presentation in tables of the

strategic policy documents and validation of the selected texts

Study of policy documents by

TAAEs & PDAEs

Representation in summary

tables by TAAEs

Validation of the summary tables

by PDAEs

Step 4 Extraction of concepts, re-presentation in

the form of a consolidated table and validation of the consolidated table

Extraction of concepts by

TAAEs

Representation in a consolidated

table by TAAEs

Validation of the consolidated

tables by PDAEs

Step 5 Coding of extracted concepts, extraction of the final list of priorities and validation of

the list

Coded of concepts by

TAAEs

Extraction of the final list of

priorities by TAAEs

Validation of the final list by

PDAes LIST

Figure 7: Flowchart of the exploratory text analysis for the identification of current priorities as explicitly and implicitly given in key EU and international policy documents

4.2.1.1.3. Advantages and disadvantages of the methodology

Advantages:

The methodology takes advantage of the opinion / expertise of the thematic experts.

By this methodology context based insights and implicit ideas can be extracted.



Identif y

There is no need for any specific software and/or equipment.

It provides with an in-depth and holistic approach.

It is a highly interactive methodology.

The methodology can consider the opinion of stakeholder representatives.

Disadvantages:

The methodology is subjective and characterised by personal bias.

Interpretation of the key documents by the expert is subjective.

Loss of information can occur since researchers can get carried away by conceptions that make sense to them but in reality, have little to do with the contexts they are analysing.

The methodology strongly depends on experts’ participation. If experts do not participate in all stages, then the results might be wrong and subjective.

Data preparation (process of extracting concepts and do the coding) is time-consuming.

Implementation of the methodology is time-consuming.

The execution of the methodology demands from the expert to spend much time and being really concentrated on his task.

The systematisation of the process of extracting concepts and do the coding is difficult.

The methodology lacks of a strong theoretical base based on well-established principles.

The lack of systematisation is reflected to the results which are difficult to replicate and validate.

Results may not be quantified.

Results may not be replicated in the same manner with a different group of experts or at a different time.

Context is not recognised automatically.

Expert interpretation is needed for the results.

Depending on the number of participating experts of the working group, the methodology might be expensive.

Wrong selection of keywords would lead to wrong results.

4.2.1.1.4. Strategies for overcoming disadvantages

Table 1: Mitigation measures to overcome weaknesses of the exploratory text analysis for the identification of priorities as explicitly and implicitly given in key EU and

international policy documents

Weakness Issue Mitigation measure

The methodology is subjective and characterised by personal bias. Interpretation of the key documents by the expert is subjective.

This is embedded in the nature of any exploratory method and can potential lead

The objectivity of the method can be increased if at least two control groups of experts participate in the described



Identif y

Weakness Issue Mitigation measure to misleading results. tasks.

Clear instructions to the participants would contribute to the prevention of personal bias.

Loss of information can occur. Wrong selection of keywords.

Both loss of information and wrong selection of keywords can lead to wrong results.

Selection of keywords, key-phrases and concepts shall be done by experts relevant to text analysis and must be then confirmed also by thematic experts i.e. related to the content of the specific document that the content was extracted. Extraction of keywords must be conducted in adequate time period, long enough to provide with high quality work.

The methodology strongly depends on experts’ participation.

If experts do not participate in all stages, then the results might be wrong and subjective.

By highlighting to the experts group the importance of their participation, higher involvement can be achieved. Preparation of a contingency plan for worst case scenario which could be implemented in case of low expert participation in the methodology. Allocation of work in pairs (leader and buddy concept) can minimise the risk of low participation.

Results may not be quantified. Results may not be replicated in the same manner with a different group of experts or at a different time.

Production of unreliable, unrepresentative and non-replicable results.

Provision of clear and common instructions to each expert involved would increase the chances that experts who apply this methodology, now and in the future, follow as much as possible the same methodology of work; and thus indirectly increasing the reliability and replicability of results.

Data preparation (process of extracting concepts and do the coding) is time-consuming. Implementation of the methodology is time-consuming. Large number of experts is needed. The execution of the methodology demands from the expert to spend much time and being really concentrated on his task.

Increasing the resources to be committed may increase costs and time to collect and analyse input

Optimise the size and time-span when using the experts’ exploratory approach method, to suit the intended task can mitigate such disadvantages.

The systematisation is difficult. It lacks of a strong theoretical base. The lack of systematisation is reflected

The lack of systematisation and automation decreases

Clear instructions of the extraction and coding process can increase the understanding



Identif y

Weakness Issue Mitigation measure to the results which are difficult to replicate and validate. Results may not be quantified. Results may not be replicated in the same manner with a different group of experts or at a different time. Context is not recognised automatically. Expert interpretation is needed for the results.

the replicability and makes validation process a difficult task.

and common sense of the process, thus optimising the replicability of results.

4.2.1.2. Computer content analysis for the identification of the current priorities from EU and international key policy documents


The general objective of the proposed methodology is the identification of the key priorities of society as explicitly given in key EU and international policy documents by computer content analysis in the form of the keywords and phrases.



identify explicitly expressed key priorities of society from key EU and international policy documents;

evaluate human content analysis for classifying words, phrases and the like from EU and international policy documents; and

evaluate the value of extracted words, phrases and the like from EU and international policy documents by human content analysis.


This methodology was developed in order to validate the assumption that the computer aided text mining is a quick tool which produces reliable results, for the purpose of the detecting key priorities of society as explicitly given in key EU and international policy documents.


Text mining is a method used in order to provide automated insight in large corpus of text, by extracting the most often used keywords, phrases and topics. It can be also used to provide objective outlook to the human content analysis.

This method is used in order to mine the key policy documents provided for the purpose of the assessment of the relevance of framework programmes and in particular of H2020. The key challenges for the society and policy objectives related to these challenges could

Methodology: Computer content analysis for the identification of the current priorities from EU and international key policy documents




Identif y

be extracted using documents produced by prominent EU and international policy organisations and institutions such as EC, UN, OECD etc.


Provalis QDAMiner 4 and Wordstat 7 softwares were used in order to extract keywords, phrases and topics from the key policy documents. The text mining of key policy documents was conducted in the following steps.

STEP 1: Selection of the strategic policy documents

In the first step of the analysis the key documents produced by the key EU and international policy institutions and organisations should be selected that could be used for the extraction of the keywords, phrases and topics as indicators of the key priorities of society. Possible list of such documents is:

(1) Europe 2020 – Issue date: Brussels, 3.3.2010; Time span: 10 years (2010-2020).

(2) G7 Ise-Shima Leaders' Declaration - Issue date: G7 Ise-Shima Summit, 26-27 May 2016; Time span: 15 years (2016-2030).

(3) United Nations Sustainable Development Goals - Issue date: 12 August 2015; Time span: 15 years (2016-2030).

(4) World Health Organisation priorities - Issue date: 12 September 2015; Time span: 15 years (2016-2030).

(5) Juncker priorities - Issue date: Strasbourg, 15 July 2014; Time span: 10 years (2015-2025).

(6) Paris Protocol under United Nations Climate Change Conference (21st Conference of the Parties also called COP 21) - Issue date: Brussels, 4.3.2015; Time span: 10 years (2015-2030).

(7) Document “21 to 21” of Organisation for Economic Co-operation and Development - Issue date: 05 May 2015; Time span: 5 years (2016-2021).

STEP 2: Pre-processing of the key policy documents The key policy documents used for the extraction of the priorities of society from key EU and international bodies were prepared in the txt format with the removing of all of the pictures and text formatting. One text database with the seven documents was created, and with the one classifying variable indicating the document used (from 1 to 7).

STEP 3: Extraction of the most often 20 key words in the key documents Initial text mining was conducted in order to track the most often key words in the key policy documents. Initially, the general keywords, like principal or Europe were excluded from the analysis. The goal was to detect the 20 most often keywords in the key documents in order to get general insight into the relevance of specific social impacts.

STEP 4: Extraction of the full list of the keywords Full list of the keywords was selected using the dictionary settings for the key words extraction. Keywords were extracted using the full list of keywords for the documents with the exclusion of the words that are common in English language, e.g. they, we, and, other etc. Exclusion dictionary ‘English.exc’ is provided as part of the Provalis Wordstat software. The software was set to generate the maximum number of 1 000 keywords.

STEP 5: Extraction of the full list of phrases



Identif y

Full list of the phrases was selected using the following settings: (1) Minimum number of words in a phrase: 2; (2) Maximum number of words in a phrase: 5; (3) Minimum frequency of a phrase: 2.

STEP 6: Extraction of the topics Full list of the most occurring 100 topics was generated with the following settings: (1) Minimum combination of keyword and phrases in a topic: 2; (2) Minimum combination of keyword and phrases in a topic: 5; (3) Minimum frequency of a topic: 2.

STEP 7: Validation of the keywords, phrases and topics Keywords, phrases and topics were evaluated in the context of their usage for the purpose of the relevance of H2020 and WPs assessment. The conclusion is that the method is; (1) the most useful when a large number of documents needed to be scanned; (2) it requires a limited amount of human interaction; (3) it is fast and not expensive; and (4) its objectivity can be easily documented. An overall summary representation is illustrated in Figure 8.



Identif y

Step 3 Extraction of the most often 20 key words

in the key documents

Step 4 Extraction of the full list

of the keywords

Step 5 Extraction of the full list of phrases

Step 6 Extraction of the topics

Step 2 Preprocessing of the key documents

Step 7 Validation of the keywords,

phrases and topics

Step 1 Selection of the key documents

Figure 8: Flowchart of the computer content analysis for the identification of current priorities as explicitly given in key EU and international policy documents


Advantages: • The methodology can be automated.

• The methodology can be done relatively rapidly.

• Outcome of the software is objective.

• The methodology can handle large amounts of texts.

• The methodology can allows quantification.

• The methodology can is straightforward to replicate.

• The expert’s knowledge and the automatic procedure are complementary.



Identif y

• The dictionary created can be used subsequently.

• The methodology can produce unexpected results that might have been easily excluded by human expert (i.e. not necessarily obvious).

Disadvantages:

• Interpretation by the expert could be subjective.

• It demands the use of software.

• Context is not recognised automatically.

• Expert interpretation is needed for the results.

• If dictionary is not adequate there are problems with synonyms and words used in a non-significant manner.

• Wrong selection of keywords would lead to wrong results.


Table 2: Mitigation measures to overcome weaknesses of the computer content analysis for the identification of current priorities as explicitly given in key EU and international

policy documents


Interpretation by the expert could be subjective.

Context is not recognised automatically. Expert interpretation is needed for the results.

If dictionary is not adequate there are problems with synonyms and words used in a non-significant manner.

Wrong selection of keywords and input documents would lead to wrong results.

The interpretation of the expert can both enhance and decrease the value of the results.

However, without the expert interpretation the text mining method cannot be fully exploited, since the software cannot recognize the context of the keywords.

The selection of inadequate keywords and/or input documents can decrease the value of results.

The text mining method can be performed with several human executed tasks used in order to assure the validity of the results: (1) human text mining expert who prepares the documents; (2) human text mining expert who selects the algorithms and other software features (e.g. minimum number of words in phrases); (3) human context expert who selects the documents to be extracted; (4) human context expert who has the context knowledge for the interpretation of the results.

The objectivity of the method can be increased if at least two control groups of experts participate in the described tasks, especially in the task related to the interpretation of the results.



Identif y

4.2.2. Methodologies for the identification of the needs of society to EU citizens (Q2.1)

4.2.2.1. Experts’ exploratory methodology for the identification of the needs of society to EU citizens

Methodology: Experts’ exploratory methodology for the identification of the needs of society to EU citizens



The general objective of the proposed methodology is the identification, by using an experts’ exploratory methodology, of the key needs of society to EU citizens (NSEUCs), from the most important channels/ways that citizens use on their voice to be heard in the EU in the form of keywords and phrases.



collect the key needs of society to EU citizens through reviewing of experts of the most important channels/ways that citizens use on their voice to be heard in the EU;

prioritise the key identified needs of society to EU citizens according to the primary and secondary attribution of the four categories of the social needs: social, economic, environmental and related to fundamental rights;

prepare the combined and the final list of collected NSEUCs; and

validate the collected NSEUCs.


This methodology was developed in order to validate the assumption that the human (expert’s) understanding is important for the identification of the needs of society to EU citizens.


The experts’ exploratory methodology is used for the purpose of achieving the main objective, that is to identify the ‘key needs of society (and their emerging challenges), to EU citizens’. This methodology, as indicated in its title (expert exploratory), is based on using a pre-selected group of independent experts who have high professional thematic knowledge and experience in order to explore the question on identifying the needs of society to EU citizens.

The thematic expertise of the experts is in one or more sectoral areas, is in line with the societal challenges and enabling/industrial technologies as identified under the Horizon 2020. This methodology consists of a group of experts (in this case 17 experts of the Expert Group on evaluation methodologies for the interim and ex-post evaluation of Horizon 2020), who make use their knowledge in order to achieve the set main objective as stated above.



Identif y

The methodology is based on collecting, in a step-by-step manner, the opinion of experts (independent scientists that speak for themselves), in order to identify key needs of society to EU citizens (taken from the perspective of finding their common emerging needs).

The terms ‘needs of society’ or ‘societal needs’ can be used freely with the same meaning as the term ‘challenges for society’. We recognize the difference between ‘challenges for society’ and ‘societal challenges’, whereby the latter is understood as the ‘societal challenges priorities’ identified under Horizon 2020(5).

In order to find out the needs of society to the European citizens, the Group’s independent experts explore, according to own thematic field(s) of expertise, the most important channels/ways that citizens use to make their ‘voice’ heard in the EU, for the four categories of societal needs, namely: social, economic, environmental and related to fundamental rights;.

The EU citizens in this context refers to various categories: including the individuals, businesses and trade organisations, associations and federations, networks, EU platforms, think-tank groups, for-profit and non-for-profit organisations, research and educational organisations.


STEP 1: Collecting independent experts’ option to identify key needs of society to EU citizens

At the outset of this task, thematic experts follow the instructions, tools and a given example on how to provide the required input.

The tools provided are:

Table Template (Excel) for collecting the independent thematic experts’ opinion on the key needs of society to EU citizens (NSEUCs) (Figure 9).

Guiding Document (Word) concisely explaining the instruction on what an Expert should do to identify NSEUCs.

5 Under the term ‘Societal challenges’ we understand the priority identified under ‘Horizon 2020’. Regulation (EU) No 1291/2013 establishing Horizon 2020 - the Framework Programme for Research and Innovation (2014-2020) and repealing Decision No 1982/2006/EC (OJ L 347 20.12.2013, p. 124): PRIORITY 'Societal challenges' This Part responds directly to the policy priorities and societal challenges that are identified in the Europe 2020 strategy and that aim to stimulate the critical mass of research and innovation efforts needed to achieve the Union's policy goals. Funding shall be focused on the following specific objectives: (a) Health, demographic change and well-being; (b) Food security, sustainable agriculture and forestry, marine, maritime and inland water research, and the bio-economy; (c) Secure, clean and efficient energy; (d) Smart, green and integrated transport; (e) Climate action, environment, resource efficiency and raw materials; (f) Europe in a changing world - Inclusive, innovative and reflective societies; (g) Secure societies - Protecting freedom and security of Europe and its citizens.



Identif y

Suggested (non-exhaustive) list of sources that experts may find relevant to use. Experts are encouraged to explore other sources of information that they consider to be relevant.

Each expert is asked to identify at least three needs of society to EU citizens within the category of societal needs i.e. social (SOC), economic (ECO), environmental (ENV) and fundamental rights (FNR), as per own expertise, and give feedback by:

(i) providing a short textual ‘narrative’ description of at least three societal needs identified, as well as by providing information on sources used;

(ii) providing five keywords associated with each the identified societal need; and

(iii) completing the table template (Figure 9) according to their own thematic expertise.

Category of

Societal Needs

Key identified needs of Society to EU Citizens (NSEUC - 'narrative')

Sources used to identify the needs

5 keywords from each identified Needs of Society for EU Citizens (NSEUC -

keyword)

NSEUC - 'narrative':

1


2


3

5 keywords - 1/NSEUC-narrative 1

5 keywords- 2/NSEUC-

narrative 2

5 keywords 3/NSEUC-

narrative 3

ECO

SO

C EN

V FN

R

Figure 9: Layout of the template used for the identification of the needs of society to EU citizens (STEP 1) in the context of experts’ exploratory methodology

The result is a summary table showing all identified needs of society to EU citizens (NSEUCs) as ‘narratives’, sources used and corresponding key words.

STEP 2: Compiling the list of sources used to identify needs of society to EU citizens

Once the experts provided the identified NSEUCs, the leader of this task on collecting needs of society to EU citizens through expert’s exploratory method, gathered all information that experts provided on sources they used, titles and web links, short descriptions and the relation of the source to the type of EU citizens, as well as the internet ranking result. The result is a summary table overview of all sources.

STEP 3: Multidimensional aspect of NSEUCs

Due to ‘multidimensional character’ of the NSEUCs experts were asked to identify the primary and/or other categories (social, economic, environmental, fundamental rights)



Identif y

that an NSEUC may be related to. To this end, an excel table as tool was provided for experts to complete for each of the NSEUC ’narratives’ they identified (Figure 10).

In the end, the leader of the task on collecting needs of society to EU citizens through experts’ exploratory approach gathered all information provided which resulted in an overview list of the main and other categories associated to each NSEUC.

Key identified Needs of Society to EU Citizens

Category of Societal Needs

Economic (ECO)

Social (SOC)

Environmental (ENV)

Fundamental Rights (FNR)

Main focus category √

Figure 10: Layout of the template used for the identification of the main category within the multi-dimensionality aspect of the identified needs of society to EU citizens (STEP 3)

in the context of the experts’ exploratory methodology

STEP 4: Presentation of the identified needs of society to EU citizens in the form of a consolidated table

In the 4th step the collected NSEUCs as ‘narratives’ identified by the experts, and their associated keywords, sources, as well as categorisation according to the primary ‘dimension’ towards the social, economic, environmental and fundamental rights character, was placed in a consolidated table in excel (using the table format as shown in Figure 9).

This represents the final list of NSEUCs identified using the experts’ exploratory method.

STEP 5: Validation of the final list of NSEUCs

During the fifth step, the completed list of NSEUCs (result in STEP 4) was circulated to the Expert Group and each member of the expert group validated through agreeing with the identified needs of society to EU citizens, their categorisation and related keywords, as well as on the way they were listed and categorised.

All steps of the experts’ exploratory methodology for identifying needs of society to EU citizens are presented in Figure 11.



Identif y

Step 2 Compiling the list of sources used to identify

needs of society to EU citizens

Step 3 Multidimensional aspect of NSEUCs

Step 4 Presentation of identified needs of society to EU citizens in the form of a consolidated table

and preparation of the final list of NSEUCs

Step 5 Validation of the final list of NSEUCs

Step 1 Collecting independent experts' option to identify key needs of society to EU citizens

Figure 11: Flowchart of the experts’ exploratory methodology for the identification of the needs of society to EU citizens


Advantages:

The methodology takes advantage of the opinion / expertise of the thematic experts (in this sense it is also subjective).

There is no need for any specific software and/or equipment.

It provides with an in-depth and holistic approach.

It is an interactive methodology.

By this methodology context based insights and implicit ideas can be extracted.

The methodology can consider the opinion of stakeholder representatives.

Disadvantages:



It depends on the diversity of the group.

Some topics may not be covered by the expertise covered within the specific expert group.



Identif y

The methodology might be expensive.

The methodology might be time consuming.

The systematisation of the process of collecting information is difficult.

The methodology lacks of a strong theoretical base based on well-established principles.

The lack of systematisation is reflected to the results which are difficult to replicate and validate.

Results may not be quantified.

Results may not be replicated in the same manner with a different group of experts or at a different time.


Table 3: Mitigation measures to overcome weaknesses of the experts’ exploratory methodology for the identification of the needs of society to EU citizens



This is embedded in the nature of the experts’ exploratory method.

The objectivity of the method can be increased if at least two control groups of experts participate in the described tasks. Clear instructions to the participants would contribute to the prevention of personal bias.

It depends on the diversity of the group. Results may not be replicated with a different group of experts or at a different time.

What is the reliability of the experts’ input in terms of variation of the method results should be repeated.

Careful selection of thematic expertise covered by experts; receiving timely input of all experts; using same steps of the method’s methodology when repeated.



By highlighting to the experts group the importance of their participation, higher involvement can be achieved. Preparation of a contingency plan for worst case scenario which could be implemented in case of low expert participation in the methodology. Allocation of work in pairs can minimise the risk of low participation.

The methodology might be expensive.


Increasing the number of experts involved may increase costs and time to collect and analyse input.

Optimise the size and time-span when using the experts’ exploratory approach method, to suit the intended task can mitigate such disadvantages.

Limitations on the ability to quantify results.

This is embedded in the nature of this method. Experts provide opinions for which it may not be straightforward to have quantification as result.

Provision of clear instructions to each expert can increase the understanding and common sense of the process. Although limited, some quantification of experts’ input is possible.



Identif y

4.2.2.2. Experts’ exploratory approach for the identification of the needs of society to EU citizens from Eurobarometer surveys and EC public consultations reports

Methodology: Experts’ exploratory approach for the identification of the needs of society to EU citizens from Eurobarometer surveys and EC public consultations reports



The general objective of the proposed methodology is to capture wider concerns and societal trends and consequently to identify new (emerging) challenges for EU society in the form of keywords and phrases.



identify wider societal concerns and trends (as potential new challenges for EU society) from Eurobarometer surveys and EC public consultation rapports available on-line;

indicate important topics that appear on-line (in abovementioned sources) and are related to potential emerging challenges for society;

present results in short form and prepare them for further verification by selecting keywords describing the potential new challenges for society;

validate results by checking if the same keywords appear in other on-line sources in similar context (i.e. if they are related to the same or similar topics); and

obtain final results and present them as list of identified emerging challenges for society.


This methodology was developed in order to identify and confirm existing and emerging actual key challenges for society from societal trends and concerns.


A novel methodological approach was proposed for this part of research in purpose to capture wider concerns and the most actual societal trends. Although experts’ exploratory approach is a common approach, in this case it was used in an innovative way as it involved analysis of sources available on-line such as results of Eurobarometer surveys and public consultations carried out by EC presented in the form of reports. The present methodological approach involved:

Qualitative analysis of selected reports available at Eurobarometer and other EC public consultations reports. In purpose to identify new (emerging) challenges, only sources published earlier than 2014 were considered.



Identif y

Exploratory analysis to examine data, evaluate validity of assumptions and understand the processes.

Interpretive techniques to interpret the data and report the observations in a structured form. Based on the results of exploratory analysis and their interpretation, the most important topics and issues were indicated for each group of topics (wider categories). Links between such groups were also distinguished and underlined. These topics and issues were recognised as potential emerging challenges for society.

Graphic techniques to visualise the results of the research.

Verification of results by thematic experts. The main purpose was to confirm or reject pre-selected topics and issues and decide whether they are potential emerging challenges for society.

Analysis of obtained results.

Selection of keywords. Based on results keywords related to selected topics were extracted. For validation purposes these keywords were checked as to whether or not they appear in other on-line contents e.g. social media or google.

Identification of emerging key challenges for society. The emerging key challenges for society were finally identified and presented for further consideration.


The group of experts (17 experts selected by EC for the Expert Group on Evaluation Methodologies for Interim and Ex-post Evaluation of Horizon 2020) carried out multistep analysis of various sources available on-line, such as Eurobarometer surveys and EC public consultations reports, in purpose to identify the emerging challenges for EU society. In order to achieve reliable results thematic experts from various disciplines need to be involved in the specific methodology and especially in STEPS 1, 2 and 3. At least five experts shall be involved; however, the number of involved experts need to cover all defined topics as well as to represent the expertise to cover all the main priorities under Horizon 2020.

STEP 1: Searching through on-line content in purpose to select the source documents and to pre-identify potential new challenges for society (based on area of expertise)

Firstly the experts selected for the task were searching through on-line content with the aim to select the adequate source documents and secondly, based on the qualitative analysis of the selected documents, to identify potential new needs of society to EU citizens. Each expert could propose adequate source documents, with a special focus on expert’s area of expertise. Source documents were selected from Eurobarometer reports, EC public consultations reports and related advice reports of EC Expert Groups. In purpose to identify new (emerging) challenges only the sources published after 2014 were taken into consideration. The selection of sources was confirmed by the expert group and the source documents were listed. The following source documents were selected:



Identif y

(1) Eurobarometer Qualitative study – ‘Public opinion on future innovations, science and technology’ – Aggregate Report, ISBN 978-92-79-47935-9, DOI 10.2777/691455 © European Union, 2015.

(2) Eurobarometer Qualitative study – ‘DG Competition Stakeholder Survey’ - Aggregate Report, ISBN 978-92-79-44101-1, DOI 10.2763/82233 © European Union, 2015.

(3) Eurobarometer Qualitative study – ‘Coping with the crisis’ - Aggregate Report, ISBN 978-92-79-43241-5, DOI 10.2767/55315 © European Union, 2014.

(4) Eurobarometer Qualitative study – ‘The Promise of the EU’ - Aggregate Report, ISBN 978-92-79-43632-1, DOI 10.2775/21479 © European Union, 2014.

(5) Flash Eurobarometer 388 ‘Attitudes of Europeans towards Waste Management and Resource Efficiency’ – ISBN 978-92-79-38487-5, DOI 10.2779/14825 © European Union, 2014.

(6) Special Eurobarometer 416 ‘Attitudes of European citizens towards the environment’ ISBN 978-92-79-39763-9, DOI 10.2779/25662 © European Union, 2014.

(7) Advice for 2018–2020 of the Horizon 2020 Advisory Group for Societal Challenge 1, "Health, Demographic Change and Well-being" June 2016 http://ec.europa.eu/research/health/pdf/ag_advice_report_2018-2020.pdf.

(8) Consultation of the Horizon 2020 Advisory Groups – Providing Advice on Potential Priorities for Research and Innovation in the 2018-2020 Work Programme - Scene Setter https://ec.europa.eu/research/consultations/swafs-wp2018-2020/scene_setter.pdf.

(9) Horizon 2020 Societal Challenge 5 'Climate Action, Environment, Resource Efficiency and Raw Materials' Advisory Group Recommendations Programming period 2018 – 2020, ISBN 978-92-79-58930-0, DOI: 10.2777/8980, © European Union, 2016.

In-depth reading of documents by all involved experts was followed to examine collected data and to evaluate validity of assumptions. Interpretive techniques were applied to interpret the data and report the observations in a structured form of the table. Exploratory analysis of sources that are available on-line mentioned above and listed as ‘Source documents’ revealed connections between some of the priorities and challenges. Based on that observation groups of topics were formed, e.g. ‘Environment, climate, energy’ or ‘Health, well-being, aging’. As a consequence of analysis and interpretation of the results the most important topics were indicated for each group of topics (wider categories). These topics were recognised as potential emerging challenges for society. In purpose to validate the results all pre-selected topics were placed in a ‘Table of potential new challenges for society’. This was done by the methodology coordinator (expert leading the task), who afterwards delivered this table to all experts from ‘Expert Group on evaluation methodologies for the interim and ex-post evaluations of Horizon 2020’ so that they could provide their feedback based on their thematic expertise in STEP 2. The main purpose was to confirm or reject pre-selected topics and to decide whether they are potential emerging challenges for society.

STEP 2: Validation of pre-identified topics related to potential emerging challenges for society (validation done by experts plus their additional comments and suggestions).



Identif y

Experts were asked to complete the ‘Table of potential new challenges for society’ (Figure 12) and answer two questions for each topic i.e.:

- Should this topic be reflected stronger in Horizon 2020 actions and future EU Programmes?

- Is this potential emerging challenge for EU society?

Group of topics: ‘Environment, Climate, Energy’

PRE-IDENTIFIED TOPICS:

Should this topic be reflected stronger in

H2020 actions and future EU Programmes?

Is this potential emerging challenge

for EU society?

If your answer is ‘yes’, please provide 1-3 keywords that in best describe this challenge according to your expertise

1.1 Topic 1 pre-identified by the methodology coordinator

Please select one answer: yes/neutral/no

Please select one: ‘yes’/’neutral’/’no’

If your answer is ‘no’ or ‘neutral’ please provide if the challenge is already covered in H2020 or is not relevant. Add any other comment.

1.X Additional topic proposed by expert

Group of topics: ‘Health, well-being, aging’





for EU society?


2.x ……………. Please select one answer: yes/neutral/no


If your answer is ‘no’ or ‘neutral’ please provide if the challenge…

Group of topics: ‘Immigration, terrorism, safety, freedom’





for EU society?





Group of topics: ‘Communication and transport’





for EU society?





Group of topics: ‘Economy, Investment, Finances’ plus ‘EU Leadership’





for EU society?





OTHER CHALLENGES: Please describe here

Figure 12: Layout of the template used for the validation of pre-identified topics related to potential emerging challenges for society (STEP 2) for the implementation of the

experts’ exploratory approach from Eurobarometer for the identification of the needs of society to EU citizens



Identif y

Each time the experts could select one answer i.e. yes or neutral or no. If the answer was ‘yes’ experts were asked to provide one to three keywords that best describe this challenge according to their expertise. If the answer was ‘no’ or ‘neutral’ the experts were asked to explain if in their opinion the challenge was already covered in H2020 or it was not relevant. Additionally, experts were invited to provide their comments where needed and suggest additional topics as ‘Other potential challenges for society’ in the place provided for this purpose at the bottom of the ‘Table of potential new challenges for society’. The experts were also instructed to focus mainly on their particular area of expertise and to provide ‘neutral’ answer in each place where they were not sure about an answer or believed they did have enough experience to decide.

Analysis of obtained results proved that while in general experts were focused on their area of expertise, some topics, located within different thematic groups, received evidently more votes from experts, regardless initially indicated area of experience. While the wider groups of topics were suggested, each topic was analysed independently. Topics that passed threshold of minimum three positive answers (‘yes’) were selected for further analysis. For some topics experts provided their comments. These were taken into consideration in the construction of ‘Table of Results’. Consequently, some topics have been worded in a slightly different way that in original sources. This decision was taken in order to make best use of experience and knowledge of experts. Some experts have also suggested other potential challenges for society that were not found in surveys available on-line nor reports which were taken into consideration in this research. Such topics were described together with appropriate keywords and redirected for further analysis.

STEP 3: Selection of keywords describing the potential new challenges for society (5-10 keywords per each challenge)

As mentioned above, topics that passed threshold of minimum three positive answers (‘yes’) were selected for further analysis. Keywords related to selected topics were extracted by the methodology coordinator and prepared for:

Social media analysis for checking if the same keywords appear in other on-line contents e.g. social media, in a similar context

Presentation in the ‘Table of Final Results: Identified and Confirmed Emerging Challenges for Society’.

STEP 4: Checking if the same keywords appear in social media in similar context

Checking if the same keywords appear in social media in similar context should be carried out by experts (which could be the members of the Expert Group) who must have enough experience to assess whether selected keywords actually appear in other on-line sources in relation to the same or similar topics. The samples for on-line content analysis should be extracted at least three times and at least three months apart to avoid taking into account temporary 'searching peaks'.



Identif y

STEP 5: Extraction and validation of final results

As a final result of the research the emerging key challenges for society were identified and presented for further consideration in the ‘Table of Final Results: Identified and Confirmed Emerging Challenges for Society’.

The results were discussed and confirmed by the expert group. These results will be used to assess relevance of Horizon 2020 to the identified key emerging needs of society to EU citizens.

Figure 13: Flowchart of the experts’ exploratory methodology for analysis of Eurobarometer and EC public consultation reports for the identification of the needs of

society to EU citizens


Advantages:

Analysis of Eurobarometer surveys, EC public consultations reports and related advice reports of EC Expert Groups (available on-line) allows to gain the most



Identif y

actual knowledge about current observations, concerns and expectations of EU citizens.

Eurobarometer is focused specifically on the emerging trends.

It provides both multidisciplinary expert opinions and the citizens voices which are essential to capture wider societal concerns and emerging trends.

Eurobarometer surveys, EC public consultations reports and related advice reports of EC Expert Groups are reliable source of information as they are carried out according to the highest level of the methodological principles, including even distribution of EU countries, as well as occupation groups, gender, age, stakeholders’ representatives etc.

The method is relatively fast and cheap and uses publicly accessible data sources.

Disadvantages:

Validation of results is necessary (checking if emerging challenges are recognised properly and if selected keywords are adequate to these challenges).


Single-source analysis does not guarantee complete results.

On-line content analysis is linked to the risk of temporary 'searching peaks' (or ‘temporary fears’).


Table 4: Mitigation measures to overcome weaknesses of the experts’ exploratory methodology for analysis of Eurobarometer and EC public consultation reports for the

identification of the needs of society to EU citizens


Risk of wrong results.

Potential emerging challenges can be wrongly recognised and/or the selection of keywords may be not fully adequate to these challenges.

Results have to be validated by checking if/how often selected keywords appear in other on-line sources in similar context (i.e. if they are related to the same or similar topics).

The methodology demands experts’ participation.


In order to obtain valid and exact results all expertise areas must be covered. All thematic experts must be given clear instructions, including the information that they have to participate in the task and provide their feedback in the timeline.

Information maybe not comprehensive

Single-source analysis does not guarantee complete results.

Multiple on-line sources must be taken into account and analysed in parallel.

Some potential challenges may be short-term.

Surveys and on-line contents may show some temporary societal fears or interests.

Samples for on-line content analysis should be extracted at least 3 times and at least three months apart to



Identif y


avoid taking into account temporary 'searching peaks'

4.2.2.3. Experts’ selection of keywords for the identification of the needs of society to EU citizens

Methodology: experts’ selection of keywords for the identification of the needs of society to EU citizens



The general objective of the proposed methodology is the identification of the suitable and extensive set of keywords organised according the H2020 structure based on the understanding of the expert group’s members of (1) the reference documents, (2) the H2020 work programmes and (3) the H2020 legal basis.

The methodology aims at assessing the cumulative human ability to identify and prioritise key topics and keywords of the society according to the H2020 structure as well as generate a keywords dataset based on the experts’ exploratory approach to be used for further purposes.


This methodology is selected to uncover experts’ opinion, in the format of keywords organised as per H2020 structure, on the current key priorities for the society taking into consideration their own knowledge as well as reference and H2020 documents.


The keyword experts’ exploratory approach aims at identifying a suitable and extensive set of keywords based on background knowledge and expertise from a group of experts, combined with their understanding of the reference documents (i.e. an arbitrary set of documents taken as reference by the coordinator in charge of executing the methodology) and the legal documents related to the establishment and implementation of the framework programme. For the case of Horizon 2020 such documents are the topics in the Council Decision of 3 December 2013 (6); the 2014–2015 Work Programme adopted in the framework of the specific programme implementing Horizon 2020; and the 2016 – 2017 Work Programme also adopted in the framework of the specific programme implementing Horizon 2020. The methodology needs to be executed by a coordinator which carries out a blind collection of the keywords among the experts




Identif y

(i.e. first expert round), then aggregates them and finally carries out their blind selection (i.e. second expert round). The keywords are organised according to the structure of H2020, which includes 5 areas (i.e. from Area-1 to Area-5, see Table 5) and a variable number of sub-areas. An extra-area (i.e. Area-X) is set to provide room for input keywords, which, according to the experts’ opinion, do not fit in any of the other H2020 areas. Note that each keyword input must consist of a single word (i.e. ‘carbon’ is accepted, ‘low carbon’ is not). Two-word contribution is accepted under the cross-area input field ‘Pairs’.

Table 5: Structure adopted for the “Keywords Expert Exploratory Approach” (as per H2020)

Area Sub-Area Identifier Name

A1: Excellent Science

ES1 ERC + MCSA

ES2 Future Emerging Technology

ES3 Research Infrastructure

A2: Industrial Leadership

IL1 Information and Communication Technologies

IL2 Nanotechnologies

IL3 Advanced Materials

IL4 Biotechnology

IL5 Advanced manufacturing and processing

IL6 Space

A3: Societal Challenges

SC1 Health, demographic change and wellbeing;

SC2 Food security, sustainable agriculture and forestry, marine and maritime and inland water research, and the Bioeconomy;

SC3 Secure, clean and efficient energy;

SC4 Smart, green and integrated transport;

SC5 Climate action, environment, resource efficiency and raw materials;

SC6 Europe in a changing world – inclusive, innovative and reflective societies;

SC7 Secure societies – protecting freedom and security of Europe and its citizens;

A4: Spreading Excellence and Widening Participation

n.a. n.a.

A5: Science with and for Society n.a. n.a.

AX: Uncovered Areas n.a. n.a.

Pairs n.a. n.a.



Identif y


The keywords expert exploratory approach consists of three steps, i.e. keywords collection, keywords aggregation and keywords selection. Hereunder a short description of each step:

STEP 1: Keywords Collection

The input template, structured in accordance with Table 5, is provided to the members of the expert group in excel format (Figure 14), together with the (1) the reference documents, (2) the H2020 work programmes and (3) the H2020 legal basis. Each expert is asked by the coordinator to input keywords in the corresponding cell of each sub-area, separated by a comma (,) (first expert round). Keywords are solely accepted as single words (e.g. carbon, vehicle). Two-word contributions (e.g. low carbon, green vehicles) are accepted under the cross-disciplinary input field pairs’, while keywords which do not fit in any sub-area (if any) are accepted under the input field “AX”. Furthermore, the request for input must limit the keywords to a small number (e.g. three to five per sub-area), and the collection phase is blind (i.e. each expert has no access to the info from the other experts till the collection phase is completed). However, if the expert does not feel confident to provide an input under a certain sub-area, then he is free to leave the corresponding cell blank. Additionally, keywords from the Area’s title (i.e. nouns and adjectives) are inserted in the keywords set by default. The excel template constitutes a simple tool for collecting input within a small group of experts, but, on a larger scale, more automated input interfaces (e.g. a web-interface based on html) are highly desirable. Such interface would allow conducting large-scale on-line input collection.



Identif y

Figure 14: Layout of the excel template used for collecting experts’ input (keywords) (STEP 1) for the implementation of the experts’ selection of keywords for the

identification of the needs of society to EU citizens

STEP 2: Keywords Aggregation

When the collection phase is concluded, the methodology coordinator aggregates the input. The aggregation consists of removing the experts’ labels (so that keywords are not associated anymore to a single expert) and remove duplicates (if any). At the level of trial only these two actions have been undertaken, however, in a more complex framework, also synonyms can be aggregated and categorised by using text processing tools which automates the word comparison and categorisation (e.g. thesaurus). The



Identif y

aggregation step is non-blind, in the sense that the same set of aggregated results is presented to all the experts to perform the selection (i.e. STEP 3).

STEP 3: Keywords Selection

Based on the aggregated set of words, the keywords are re-submitted to the same expert group for a further evaluation (second experts round). In particular each expert is required to indicate 3 keywords per sub-area that, according to his/her personal understanding, were the most representative of the area. Also here the input is kept on a voluntary basis, thus, if the expert does not feel confident of indicating a preference in a specific sub-area, he/she must not be forced to do so. Moreover, the selection phase is also performed blind, as the collection one. The preferences from the group are therefore collected, and those keywords which do not accumulate a certain number of preferences (e.g. the threshold was set at 3, except the keywords from the sub-area titles which are not subjected to the threshold) are discarded from the dataset. The selection aims at prioritising specific keywords among the large set of keywords, identifying a small sub-set (e.g. core-set) to conduct further text mining analysis at a later stage.

A graphic representation of the methodological steps is reported in Figure 15.

Figure 15: Flowchart of the experts’ keywords selection for the identification of the needs of society to EU citizens


Advantages:

The selection of the keywords is based on an expert reviews, thus it is based on the human understanding of the documents taken as reference. This, in principle, allows recognising the context as well as avoiding trivial mistakes, which might arise because of the complete automatisation of the data processing, e.g. brute force text mining.

The selection of the reference documents is arbitrary, thus allows steering the results to avoid trivial mistakes.



Identif y

The methodology relies on the framework programme structure (here H2020 structure), and thus provides results that can be comparable across different methodologies based on a common structure.

A simple excel template can suit for executing the methodology with a small group of experts, hence there is no need of specific custom software.

The methodology is, by definition, interactive, since the results of the (blind) collection are then presented in (non-blind) aggregated format to perform the (blind) selection. Thus, the alternation of blind / non-blind / blind steps allows experts to have an overview also on the input from the other experts. This guided cooperation and common overview of the results exploits the potential of collective intelligence in large groups of experts.

The methodology can be carried out with different groups of experts and representatives from stakeholders, thus allowing including the position of industrial clusters and/or Research & Technology Organisation organisation.

Disadvantages:

The selection of the keywords is based on an expert reviews and it involves human actions, which are, by definition, influenced by subjective opinions and human bias.

The selection of the reference documents itself is arbitrary, thus influencing the objectivity of the results.

The methodology is based on an expert reviews, thus it involves human work-force to read and understand reference source documents, which might be costly and time-consuming.

The methodology relies on the H2020 structure thus limiting and influencing the inputs from the experts, providing a constrained framework.

The use of the excel template for the collection and selection phase is slow and involves manual works, hence resulting costly and time-consuming as well as feasible and manageable only in a relatively small group of experts.

The aggregation phase is also performed manually, thus slowing down the input processing time.

Keywords inputs might be incomplete, since they rely on the expertise of a finite and limited number of experts.

The methodology results might depend on the selection of the experts, thus posing questions on the representativeness and repeatability of the results by changing the expert group members.

The qualitative nature of the results and of the keywords is a potential disadvantage.


Table 6 describes mitigation issues to the disadvantages listed in the section above.



Identif y

Table 6: Mitigation measures to overcome weaknesses of the experts’ keywords selection for the identification of the needs of society to EU citizens


The selection of the reference documents is subjective.

This limitation is embedded in the arbitrary selection of the background information for carrying out STEP 1.

A potential mitigation measure can be to conduct a preliminary consultation in order to get a wider expert opinion on which documents must be taken as reference.

The selection of the keywords are based on expert, and thus the selection might be subjective.

This limitation is embedded in the nature of the methodology.

It would be advisable to involve a statistically representative sample of experts, ensuring coverage of the expertise portfolio, as well as a weighted distribution of the know-how in the expert group.

The involvement of human workforce in revising the reference documents can be a potential disadvantage.


In order to minimise the effects of this limitation, the group of experts can be provided with a common training on how to revise the reference documents.

Influence of the H2020 input structure on the choices of the experts during the collection and selection phases.


Leave some ‘degree of freedom’ for expressing innovative ideas.

The use of the excel template for the collection and selection phase can be a potential limitation.

Use of excel template is slow and involves manual works which makes the methodology time consuming.

This task can be automated by means of more refined input interfaces (e.g. a web-interface based on html).

The aggregation phase is performed manually.

Manual execution of the aggregation phase makes the methodology time consuming.

This task can be automated by means of text processing tools (word comparison & categorisation per synonyms).

Keywords inputs might be incomplete, since they rely on the expertise of a finite and limited number of experts.



Possibility to replicate the results with different groups of experts.



Qualitative nature of the results and of the keywords.


The keywords are per-se qualitative; however text mining analysis that can be conducted using these keywords can provide sound quantitative results.



Identif y

4.2.2.4. Computer content analysis for the identification of the needs of society to EU citizens from Eurobarometer surveys

Methodology: Computer content analysis for the identification of the needs of society to EU citizens from Eurobarometer surveys



The general objective of the proposed methodology is the identification of the key needs of the society to EU citizens by using the text mining method to Eurobarometer selected documents – in the form of keywords and phrases.



identify explicitly expressed key needs of the society to EU citizens from the Eurobarometer selected surveys;

evaluate the value of extracted words, phrases and the like from the Eurobarometer selected surveys.


This methodology was developed in order to validate the assumption that the computer aided text mining is a quick tool which produces reliable results for the purpose of detecting key NSEUCs, from Eurobarometer selected surveys.


Text mining is a method used in order to provide automated insight in large corpus of text, by extracting the most often used keywords, phrases and topics. It can be also used to provide objective outlook to the human content analysis.

This method is used in order to mine the Eurobarometer selected surveys in order to produces reliable results for the purpose of the detecting key NSEUCs. The following Eurobarometer surveys were used for the text mining:

(1) Eurobarometer Qualitative study – ‘Public opinion on future innovations, science and technology’ – Aggregate Report, ISBN 978-92-79-47935-9, DOI 10.2777/691455 © European Union, 2015.

(2) Eurobarometer Qualitative study – ‘DG Competition Stakeholder Survey’ - Aggregate Report, ISBN 978-92-79-44101-1, DOI 10.2763/82233 © European Union, 2015.

(3) Eurobarometer Qualitative study – ‘Coping with the crisis’ - Aggregate Report, ISBN 978-92-79-43241-5, DOI 10.2767/55315 © European Union, 2014.

(4) Eurobarometer Qualitative study – ‘The Promise of the EU’ - Aggregate Report, ISBN 978-92-79-43632-1, DOI 10.2775/21479 © European Union, 2014.

(5) Flash Eurobarometer 388 ‘Attitudes of Europeans towards Waste Management and Resource Efficiency’ – ISBN 978-92-79-38487-5, DOI 10.2779/14825 © European Union, 2014.



Identif y

(6) Special Eurobarometer 416 ‘Attitudes of European citizens towards the environment’ ISBN 978-92-79-39763-9, DOI 10.2779/25662 © European Union, 2014.


The text mining of key documents was conducted in the following steps. Provalis QDAMiner 4 and Wordstat 7 softwares were used in order to extract keywords, phrases and topics from the Eurobarometer documents indicating the needs of society, expressed by the citizens.

STEP 1: Pre-processing of the Eurobarometer documents

The Eurobarometer documents used for the extraction of the needs of society were prepared in the txt format with the removing of all pictures and text formatting. One text database with the 6 documents was created, and with the one classifying variable indicating the Eurobarometer document used (from 1 to 7).

STEP 2: Extraction of the most often 20 key words in the key documents

Initial text mining was conducted in order to track the most often key words in the Eurobarometer documents. In the first phases, the general keywords, like ‘principal’ or ‘Europe’ were excluded from the analysis. The goal was to detect the 20 most often key words in the Eurobarometer documents in order to get general insight into the relevance of specific needs of society, expressed by the citizens.

STEP 3: Extraction of the full list of the keywords

Full list of the keywords was selected using the dictionary settings for the key words extraction. Keywords were extracted using the full list of keywords for the documents with the exclusion of the words that are common in English language, e.g. they, we, and, other, and so on. Exclusion dictionary English.exc is provided as part of the Provalis Wordstat software. The software was set to generate the maximum number of 1000 keywords.

STEP 4: Extraction of the full list of phrases


STEP 5: Extraction of the topics

Full list of the most occurring 100 topics was generated with the following settings: (1) Minimum combination of keyword and phrases in a topic: 2; (2) Minimum combination of keyword and phrases in a topic: 5; (3) Minimum frequency of a topic: 2.

STEP 6: Validation of the keywords, phrases and topics

Keywords, phrases and topics were evaluated in the context of their usage for the purpose of the relevance of H2020 and WPs assessment. The conclusion is that the method is; (1) the most useful when a large number of documents needed to be



Identif y

scanned; (2) it requires a limited amount of human interaction; (3) it is fast and not expensive; and (4) its objectivity can be easily documented.

Step 2

Extraction of the most often 20 key words in the key documents

Step 3

Extraction of the full list of the keywords

Step 4

Extraction of the full list of phrases

Step 5

Extraction of the topics

Step 1

Preprocessing of the key documents

Step 6

Validation of the keywords, phrases and topics

Figure 16: Flowchart of the computer content analysis for the identification of the needs of society to EU citizens from Eurobarometer surveys


Advantages:

The methodology can be automated.

Outcome of the software is objective.

The methodology can handle large amounts of texts.



Identif y

The methodology can allows quantification.

The methodology can is straightforward to replicate.

The expert's knowledge and the automatic procedure are complementary.

The dictionary created can be used subsequently.

The methodology can produce unexpected results that might have been easily excluded by human expert (i.e. not necessarily obvious).

Disadvantages:


It demands the use of software.

Context is not recognised automatically.

Expert interpretation is needed for the interpretation of the results.

If dictionary is not adequate there are problems with synonyms and words used in a non-significant manner.

Wrong selection of keywords would lead to wrong results.

The methodology might be expensive, and time consuming.


Table 7: Mitigation measures to overcome weaknesses of the Computer content analysis for the identification of the needs of society to EU citizens from Eurobarometer surveys



Context is not recognised automatically. Expert interpretation is needed for the results

If dictionary is not adequate there are problems with synonyms and words used in a non-significant manner

Wrong selection of keywords and input documents would lead to wrong results.

The interpretation of the expert can both enhance and decrease the value of the results.

However, without the expert interpretation the text mining method cannot be fully exploited, since the software cannot recognize the context of the keywords.

The selection of inadequate keywords and/or input documents can decrease the value of results.

The text mining method can be performed with the several human tasks used in order to assure the validity of the results: (1) human text mining expert who prepares the documents; (2) human text mining expert who selects the algorithms and other software features (e.g. minimum number of words in phrases); (3) human context expert who selects the documents to be extracted; (4) human context expert who has the context knowledge for the interpretation of the results.

The objectivity of the method can be increased if at least two control groups of experts participate in the described tasks, especially in the task related to the interpretation of the results

The methodology might be Increasing the number of Optimise the size and time-span



Identif y


expensive.


experts involved may increase costs and time to collect and analyse input.

can mitigate this limitation.

4.2.2.5. Social media analysis for the identification of the needs of society to EU citizens

Methodology: Social media analysis for the identification of the needs of society to EU citizens



The general objective of the proposed methodology is the identification of the key NSEUCs of society as explicitly or implicitly expressed by citizens through social media analysis in the form of keywords and phrases.



identify explicitly and implicit expressed key NSEUCs from citizens’ expressions collected through social media (Twitter and Facebook) and other relevant online sources (Wikipedia and YouTube);

develop a top down and bottom up approach in the social media analysis;

evaluate the use of R-programme as suitable free tool for monitoring data extracted from Twitter and Facebook that allows capturing and analysing citizens’ expressions in particular themes;

validate the list of NSEUCs developed from results obtained from the two approaches and from the different social media and other online sources.


Social media analysis is a suitable methodology for capturing the voice of citizens on the analysis of NSEUCs. Social media analysis is increasing recognised as a valuable mean to capture citizens’ expressions and opinions.


The increasing relevance of social media in citizens’ lives opens up new venues for research and policy attempting to capture citizens’ expressions and opinions. The research community and policy-makers are increasingly paying more attention to consider them in multiple different ways. According to recent Eurostat data (EUROSTAT, 2015 Year in Review. Types of social media used, 2015) (EUROSTAT, Individuals using the internet for posting messages to social media sites or instant messaging, 2009) social media (SM) is becoming an important instrument to show people’s opinions and social interests. In fact, the average of individuals and enterprises that are daily employing SM



Identif y

are increasing every year. For instance, in 2014 the European average (EU-28) of enterprises using social media was 33% and one year later was 36%. Additionally, in several countries this percentage reaches 61% as in Netherlands and Ireland. Similarly, the same applies to common people’s use in which the change is also remarkable. The European average (EU-28) was 32% in 2009 while in 2012 increased to 40%, with countries such as Netherlands and Iceland to account for 65% of involvement for 2012.

Criteria for selecting the type of SM to be investigated include the existence of metrics-gathering by Altmetrics analytics and not covered by other experts’ from the group. Considering the above, Twitter and Facebook were selected. Furthermore, it is relevant to take into account that Almetrics analytics also includes data obtained from other sources like Wikipedia (Internet free encyclopaedia) and YouTube (video sharing website) although not being strictly SM, both of them are extremely popular for searches among citizens. Taking into account that the interest of this work is to understand the extent to subsequent needs of society are relevant to EU citizens, these two sources have also been included. Therefore, the methodology described here includes analysis for Facebook, Twitter, Wikipedia and YouTube.

This methodology has been framed under two different but complementary strategies: the top-down and the bottom-up. Top down approach included the relevance of pre-selected keywords defined through social media analysis (Twitter and Facebook) and other sources (Wikipedia and YouTube). Bottom up approach was mainly based on the identification of NSEUCs from trending topics generated within each SM or online source analysis in order to obtain a list of NSEUCs, without the limitation of pre-selected keywords. The use of the two approaches allows overcoming the limitations brought of applying just only one of the two.


4.2.2.5.2.1. Top-down approach

The top-down approach refers to the use a pre-defined keywords list to be searched in the selected SM (Facebook and Twitter) and other sources (Wikipedia and YouTube).

STEP 1: Selection of the type of Social Media Social Media have been selected according to their inclusion in Altmetrics analytics: Twitter Facebook, Wikipedia and YouTube.

STEP 2: Selection of keywords This step is related to the definition of the list of keywords to be searched in the preselected social media. The identification of keywords has been based on those defining words that appear in the definition of social impact from the Better regulation Toolbox of the European Commission, TOOL #16: IDENTIFICATION / SCREENING OF IMPACTS7. A total of 52 keywords were searched in each of the selected SM and online sources.

STEP 3: Selection of tools for social media analysis




Identif y

For each of the selected SM and online source, a different tool to collect the data has been applied. In particular, for: (a) Facebook and Twitter: the R Project for Statistical Computing (from now on also called ‘R’)8 has been used to collect data; (b) Wikipedia: the statistics Wikipedia Page, that is its own tool as it collects statistics of page views and it is possible to download the report as CSV file; and (c) YouTube: similarly to the latter, the You tube’s search engine has been used as it provides information of searches and most viewed videos.

STEP 4: Data collection and analysis for Facebook, Twitter, Wikipedia and YouTube

Data collection is described below for each SM (Twitter and Facebook), and sources (Wikipedia and YouTube) selected. Data collection of Twitter and Facebook has been collected through the ‘R Project for Statistical Computing’, Wikipedia data collection has been used through consulting the ‘page view statistics’, and YouTube from ‘YouTube’s search engine’.

To collect Facebook data, the first step was to install the Rfacebook package in the ‘R Project for Statistical Computing’. Once installed, the objective was to extract information from the public pages of Facebook, and more particularly, all those that contain the keyword searched. R makes possible to define the ‘N’ and in this case N was set to 10 000 pages as it was considered a sufficiently large number to collect the maximum information available. A total of automatic 48 searches were conducted one per keyword. For each keyword, the following information was obtained:

List of Facebook pages: (a) page name and link; (b) category: based on Facebook own classification; (c) geolocalisation: city, country (when available); (d) description provided by the page creator; and, (e) talking about: is the number of visitors that have interacted with the page (e.g. someone who has ‘Liked’ the page, has commented on it or has shared a post).

Selection of the five most ‘Talking about’ rate pages (automatically obtained through a code development): this information reflects the number of users that have interacted more with the Facebook pages (clicked, shared, commented on, etc.).

For the data analysis, a review for the pages with the highest ‘talking abouts’ and geolocation information have been provided, as well as analysis of the five Facebook pages with the greatest interaction (the highest number of ‘talking abouts’).

In order to extract Twitter data, the first step was to install Twitter-R package in the ‘R Project for Statistical Computing’. Once installed, programming should be performed in order to extract the obtained results in an Excel workbook. ‘N’ value was again set to 10 000 and again 48 searches were performed among hashtags, each per keyword. The Twitter-R package need to be connected with the freely available Twitter API. Once it is connected there are two limitations that need to be considered. In particular, the first limitation is that Twitter Search API searches against a sampling of recent Tweets published in the past 7 days and the second is that if a keyword has more than 10 000 tweets, the system stops searching when the ‘N’ is reached. The analysis of

8 https://www.r-project.org/



Identif y

hashtags allows assessing the relevance of a particular keyword in the community, as their presence or absence indicates the level of interest.

There are currently different freely available programmes to draw information from Twitter contents. However, all of them have limitations mainly related to the access to the tweets. In the case of the ‘R Project for Statistical Computing’, despite limitations mentioned above, it is possible to conduct an approximate analysis of the presence and frequency of the keywords selected and to extract conclusions regarding this. Currently, hashtag analyses are common to measure the impact of a particular topic in a given community. For each keyword, the following information was extracted:

List of the tweets obtained through the selected hashtag (containing the keyword) including: (i) list of tweets; (ii) date of publication of the tweet; (iii) favourite count: tweets marked as favourites; (iv) screen name: name of the Twitter user; (v) retweet count; and, (vi) retweet: if the tweet is not original but a retweet.

Summary of numerical data including: (i) total tweets; (ii) original tweets: number of original tweets of the total tweets; (iii) replies: number of tweets which are replying to other tweets of the total tweets; (iv) RTs (Retweets): number of tweets which are a retweet of the total retweets; (v) total RT: number of retweets done from the total tweets; (vi) total favourites: number of favorited tweets; (vii) contributors: amount of people who have tweeted the searched keyword; and (viii) tweet/contributor: average number of tweets per contributor.

In terms of data analysis, besides revising those keywords that account with the largest number of tweets, the engagement measure has been also useful in order to see the sum of tweets in a selected hashtag that are Replies, Favourites, RTs and total RTs.

The Wikipedia Analysis has been conducted through the statistics data tool of Wikipedia, namely: ‘Page view Statistics’. The data collected in the Wikipedia per keyword are the following: (i) whether the keyword has its page in the Wikipedia; (ii) date of query: the date of the query done is written down; (iii) time period: from the available options (10, 20, 30, 60 or 90 days) 90 days were selected so as to obtain data over a long period of time; and (iv) total number of page views: the total number of visits in all the languages in which a specific page has been published.

Regarding the analysis, the option has been to review whether the keyword was present in Wikipedia, number of views, and the number of languages translated, as an indicator for its relevance.

YouTube data collection has been implemented through the use of YouTube search engine, which is the YouTube´s video own search engine. This has allowed selecting manually the necessary data for this analysis, in particular: (i) search of the keyword in order to obtain the videos related to it; (ii) period in which the videos appear, limited to 1 year; (iii) order the results per view counts of the videos; and (iv) selection of the five most viewed videos.

Regarding the data analysis, these are the items that have been reviewed: (i) number of videos resulting for the selected search parameters: it indicates the extent to what said need of society to EU citizens is present or not within the YouTube community; (ii) list of



Identif y

the five videos with the most view counts: it helps obtain information concerning the most visited contents regarding the selected keywords related to NSEUCs. Therefore, it allows analysing the contents which have the largest number of visits within the selected parameter.

STEP 5: Comparison of the relevance of the selected keywords across SM and online sources

Because each social media has its own logics for measuring the presence of each keyword in them, there are no common parameters across SM and online sources. To overcome this limitation, for each social media and other sources the most representative measure has been taken: Facebook interactions (talking about), Twitter interactions (Engagement), YouTube (view counts of five most videos last year), Wikipedia (number of page views). In each case, a specific range has been defined and has been translated into four levels (Low, Medium, Medium High and High) signed for each SM allowing to compare and to identify which keywords have more or less presence in each data collection source. The following ranges were defined per each social media and other sources included:

o Range on Facebook (measured by Number of Talking about (pages) 1-999 (Low), 1,000-99,999 (Medium), 100,000-999,999 (Medium-High), +1,000,000 (High).

o Range on Twitter (measured by Number of Engagement (replies+favorites+Rts+Total Rts), 1-9,999 (Low) 10,000-99,999 (Medium), 100,000-999,9999 (Medium-High), (+) 1,000,000(High).

o Range on YouTube (measured by number of view counts (from 5 most view videos last year), 1-99,000 (Low), 100,000-999,000 (Medium), 1,000,000-9,999,999 (Medium-High), (+) 10.000.000 (High).

o Range on Wikipedia (measured by number of page views, 1,000-99,999 (Low), 100,000-499,999 (Medium), 500,000-999,999 (Medium-High), (+) 1,000,000 (High).

For the purpose of comparison, the common categories of high, medium-high, medium and low were used in order to see in which sources were more present the searched keywords.



Identif y

Figure 17: Flowchart of the top-down approach for social media analysis for the identification of the needs of society to EU citizens

4.2.2.5.2.2. Bottom-up approach

The bottom-up approach to this analysis intends to explore the main needs and concerns of the EU citizens based on their direct expressions as they are showed in the following sources: Facebook, Twitter and Wikipedia. Contrary to the top-down approach, instead of having a predefined selected list of keywords, the goal is to search which topics are most referred and consulted by citizens. This bottom-up approach has been used to identify of NSEUCs from SM secondary literature, Twitter and Wikipedia.

STEP 1: Identifying NSEUCs from SM secondary literature Twitter and Facebook produce every year a list of the most relevant topics that have been more presence in their networks: 2015 Year in Review (Facebook) – 10 topics with most talking abouts,9 and 2015 Year in Review Twitter (top trends and most influential topics).10 These reports have served to identify a list of NSEUCs.

STEP 2: Identifying NSEUCs from Twitter Trending Topics of 15 European Countries In order to identify NSEUCs, the ‘R Project for Statistical Computing’ has been used to extract the information of the maximum 50 trending topics (TT) for 15 European countries (Austria, Belgium, Denmark, France, Germany, Greece, Ireland, Italy, Latvia, Netherlands, Poland, Portugal, Spain, Sweden, United Kingdom) which are those available on the ‘R Project for Statistical Computing’. Practically this is implemented through the installation of a specific code from TwitterR that allows obtaining trending topics from different countries in one day (in this case during the 19th of September 2016). Once the results are extracted a first manual screening is conducted in order to identify those topics that would correspond to NSEUCs. In this sense all Trending Topics related to TV Shows, Series, Films and Music (singers or groups), jokes are discarded

9 http://newsroom.fb.com/news/2015/12/2015-year-in-review/ 10 https://2015.twitter.com/



Identif y

from the analysis. To do so, two columns are included in the excel file i.e.: 1st column: Is this a NSEUC? - In this column the applicable answers are either ‘Yes’ or ‘No’); and 2nd column: possible keyword – This column has been completed identifying explicit and implicit keywords. For example, the keyword ‘economy’ is explicit in a trending topic but ‘Lesbos” is dealing with refugees’ issues.

Figure 18: Layout of the template used for Identifying NSEUCs from Twitter Trending Topics of 15 European Countries (STEP 2) of the bottom-up approach of social media

analysis

STEP 3: Identifying NSEUCs from Wikipedia most viewed pages

Wikipedia (Internet free encyclopaedia) is other relevant online source used mainly for citizens around the world for consulting different topics. Therefore, the most viewed pages per language constitute an indicator to identify those topics most valued by citizens. Wikipedia provides its own tool for consulting the most viewed pages11. This tool allows downloading a report of those pages with more visits in a concrete period to select (a day, month). In this case a monthly report has been selected (August 2016). Four reports have been downloaded with the most viewed pages in four languages (English, French, German and Spanish). Once reports have been downloaded a review of pages has been done in order to identify NSEUCs. A list of NSEUCs per selected language has been obtained.

STEP 4: List of NSEUCs

The final step is to create a list of NSEUCs obtained from different sources explored.

In Figure 19 step-by-step illustration of the methodology is given.

11 https://tools.wmflabs.org/topviews/?project=en.wikipedia.org&platform=all-access&date=last-month&excludes=



Identif y

Figure 19: Flowchart of the bottom-up approach for social media analysis for the identification of the needs of society to EU citizens


Advantages:

The proposed methodology allows both top-down and bottom up approach.

By the application of the bottom-up approach conduction of direct analysis of the expressions made by citizens (primary data) can be implemented.

This methodology can be complementary to other methodologies since it can identify new NSEUCs (e.g. stop evictions, same-sex marriage) that have not been identified through other methodologies (for example from bottom up approach).

Allows to assess the relevance of each of the predefined needs according to what citizens express in social media and other online sources (from top down approach)

Easy, fast and free to use by non-experts

Different languages can be taken into account

Using different social media allows to compare and to embrace greater diversity of publics

Disadvantages:

Social media is just the reflection of the opinions of the sub-group of citizens that participates in the social media, not covering the insights of the entire population.

The usage of synonyms that are hard to recognize without the context

The coverage of countries, age and education using social media is uneven.

The reliability of the provided content is low in many cases.

There is a lack of tracing back historical data from Twitter and Facebook due to API Limitations of both SM.

STEP 4. Integrated list of

NSEUCs



Identif y

Manual annotation of YouTube searches information can be a drawback.


Table 8: Mitigation measures to overcome weaknesses of the social media analysis for the identification of the needs of society to EU citizens


Social media is just the reflection of the opinions of the sub-group of citizens that participates in the social media.

Social media analysis collects only data from SM users, and thus results from social media analysis might be subjective.

Although data is limited to SM users, it contains more representative citizens’ voices than other sources. The methodology can be combined with other more objective methodologies.


Identification of synonyms to define valid keywords

Two ways for mitigating this weakness: a) To use R-programme to extract associated words, b) To qualitatively analyse conversation on Facebook page (through comments) or to explore interactions under particular hashtags in order to understand in context meaning

Uneven coverage of countries, age and education.

Age and education data is not available; countries data is limited.

No strategy to overcome age and education limitation has been identified. Related to the countries this information is available on Facebook Data collection Top down approach, with the limitation of only those geocalizated pages could be identified. Also countries information is available in the list of 15 European Countries Trending Topics.

Reliability of content.

The reliability of the content is many times questionable which can have negative effect on the quality of the obtained results.

Reliability is ensured with a specific design of validity checks conducted by external researchers.



Identif y


Limitation of historical data from Twitter and Facebook due to API Limitations of both SM.

Limitation of historical data (7-9 days).

R allows defining a monitoring process accounting for collecting data throughout time. For instance, R allows defining a strategy of monitoring of Trending Topics on 15 European Countries defining a period to extract reports. This monitoring process is useful to compare Trending Topics from the available European countries and to extract which are the common and which are specific.

Manual annotation of YouTube data collection

Technical limitation of YouTube data collection.

Technical limitation of YouTube data collection could be overcome with last advances of R-program. Exploring new forms of obtaining YouTube data collection in an automatic manner will be explored.



Identif y

4.2.3. Methodologies for the identification of the subsequent technological and scientific advances (Q3.1)

4.2.3.1. Experts’ exploratory approach for the identification of subsequent technological and scientific advances

Methodology: Experts’ exploratory approach for the identification of subsequent technological and scientific advances



The general objective of the proposed methodology is the collection of experts’ opinion on the latest technological and scientific advances in the form of keywords and phrases.



design a methodology for collecting experts’ opinion on subsequent technological and scientific advances;

define subsequent technological and scientific advances; identify the current subsequent technological and scientific advances.


This methodology was developed in order to examine the assumption that the experts’ opinion method can create added value as a methodological approach due to its capacity of detecting advances, opportunities and threats in technology at an early stage.


Collecting experts’ opinions represent a data gathering technique which utilises the perceptions and knowledge of experts in specific thematic areas. More particular, experts’ opinions is collected through a structured and systematic process with the use of a template specifically designed for this methodology. The process is mainly based on the technology scout method (i.e. research and analyse technical and scientific documentation regarding a particular technology of interest/domain of expertise), according to each experts’ expertise.

In order to collect the experts’ opinion in a structured and organised manner, a methodology coordinator was assigned. The coordinator has the role to guide the experts through the entire process, review and also to bring together the opinions in a common database. The coordinator is also responsible to provide experts with a common toolbox and well-defined guidelines related to the assignments that the experts have to complete. The first toolbox was related to the definition of the four processes of technological advances i.e.: discovery, invention/ innovation, dissemination and application. Proposed definitions were provided in accordance with the article: ‘Technological Advances: Discovery, Invention, Innovation, Diffusion, Research and Development’. The toolbox was designed to familiarise the experts with the stages of



Identif y

scientific and technological development. The definitions were reviewed by experts and their final form is presented in Table 9.

Table 9: The four processes of technological advances12

Discovery Invention/ Innovation Dissemination Application

A discovery involves the elucidation of the fundamental processes of nature through observations of nature, reasoning, and experimentation.

An invention or innovation is the development of a product or process by applying previous knowledge in new ways.

Dissemination is the spread of innovation to other firms with the scope to remain competitive.

It involves exploit discoveries or inventions to produce a useful product or process for a specific application.

The second toolbox included the two definitions of ‘scientific advance’ and ‘technological advance’, in order to help the experts in the scouting process as provided in Table 10.

Table 10: Definitions of ‘scientific advance’ and ‘technological advance’

Scientific advance

Any new observation in a specific field - based on existing technologies, processes/methods, products, facts, natural phenomenon, objects - which in turn allow us to build even more scientific knowledge and which could inspire new technologies, processes/methods, products, facts.

Technological advance

A technological advance is an innovation, which has at least one of the four attributes: new, intelligent, leading-performer and structure-changing and a Technology Readiness Level (TRL) between 1 and 9. These innovations could be also seen as first concrete signs of emerging or future technological trend movements.

TrenOne13 platform for the trend scouting process uses these attributes:

New: It is essential for technology to feature a previously unknown element, in the form of an innovation or a significant enhancement.

Intelligent: It must be possible to detect an intelligent concept behind the product.

Leading performer: An innovation must have the potential for mass implementation and be able to adapt to other industries

Structure-changing: The technology should be capable of making fundamental changes to the way we live. Look at technologies which

TRL scale according to H202014

TRL 1 basic principles observed

TRL 2 technology concept formulated

TRL 3 experimental proof of concept

TRL 4 technology validated in lab

TRL 5 technology validated in relevant environment (industrially relevant environment in the case of key enabling technologies)

TRL 6 technology demonstrated in relevant environment (industrially relevant environment in the case of key enabling technologies)

TRL 7 system prototype demonstration in operational environment

12 ‘Technological Advances: Discovery, Invention, Innovation, Diffusion, Research and Development’, available at http://thismatter.com/economics/technological-advances 13 http://www.trendone.com/en/trend-universe/micro-trends.html 14 https://ec.europa.eu/research/participants/data/ref/h2020/wp/2014_2015/annexes/h2020-wp1415-annex-g-trl_en.pdf



Identif y

have a chance of having a volume impact on many, many people, or large segments of the society.

TRL 8 system complete and qualified

TRL 9 actual system proven in operational environment (competitive manufacturing in the case of key enabling technologies; or in space).

Except of the toolboxes, the coordinator of the methodology was responsible to design a template (in excel format) to be used by each expert for the collection of experts’ opinion. Moreover, the coordinator was responsible to integrate the input from each expert, to review the database, to correct the errors and to extract keywords from each article.

The result of applying this methodology is a list of technological and scientific advances, namely keywords, including current and emerging technological trends, which can be subsequently used for the assessment of the H2020.


STEP 1: Identification of the sources of information on science and technology Based on the definitions provided to the toolboxes, the experts were asked to identify and propose reliable and representative of information on technological and scientific advances according to their area of expertise, from various sources (e.g. platforms, websites, patents databases, etc.) in which experts would, in a later phase under STEP 2, identify the emerging technological and scientific advances. Besides the formal and informal sources, the experts were also encouraged to explore the sourcing of technology: patents, joint research, licensing, buying intellectual property rights (IPRs), creating joint-ventures or outright acquisition of start-ups.

More particularly, sources of information include: formal - technology platforms, official websites on technology, and informal - blogs, social media pages, groups, technology forums, scientific publications, innovation and technical reports etc (Table 11).

Table 11: Examples of sources of information utilised for the experts’ exploratory approach for the identification of the subsequent technological and scientific advances

Source Publishing date

www.sciencedaily.com 05.10.2015

http://www.nanowerk.com 25.09.2015

www.electronicweekly.com 14.07.2015

http://www.dailymail.co.uk 06.05.2015

http://gadgets.ndtv.com/ 22.09.2015

http://www.sciencealert.com/ 26.06.2015

http://tech.firstpost.com 09.06.2015

http://www.independent.co.uk 14.08.2015

https://techcrunch.com 05.08.2015

http://neurosciencenews.com 06.10.2015

http://www.popsci.com 03.08.205

https://www.weforum.org 04.03.2015

http://www.theengineer.co.uk 27.10.2015



Identif y

http://scitechdaily.com/ 15.08.2015

http://www.technology.org 17.04.2015

STEP 2: Scouting for technological and scientific advances

During the second step, the experts were asked to scout for technological and scientific advances provided from the identified sources (under STEP 1). This can be achieved by: (a) directed: searching in specific technological fields (Technology Monitoring), or (b) undirected: searching for new technological advances in ‘white spaces’ (Technology Scanning) – both approaches were encouraged and applied. During scouting process, the experts also reviewed the ‘Top 10 emerging technologies’ for 2015 and 2016: List of 10 Breakthrough Technologies 2016 - MIT, Top Ten Emerging Technologies 2016 – World

Economic forum and ‘The last 15 scientific breakthroughs’ published by Research

Executive Agency (REA)15.

STEP 3: Collection, ranking and classification of the technological advancements After the experts have identified and selected max. 10 articles related to the technological and scientific advances, they were asked to include them in the template prepared by the coordinator presented in Figure 20.

Based on the common template, each expert filled in the following: names of the main analysed sources of information, the date at which each article was published, the title of each article, the full text of the identified articles, link to the entire article and a short assessment of the technologies based on defined criteria (four attributes and TRLs).

Expert No.

Name of the

platform or website

Date of publication

(2015, 2016)

Title (of article)

Full text of the article Link Rank New Intelligent Leading-

performer Structure-changing Notes

#1

#2

Figure 20: Layout of the excel template used for collecting experts’ input (keywords) (STEP 3) for the implementation of the Experts’ exploratory approach for the

identification of subsequent technological and scientific advances

15 http://ec.europa.eu/rea/breakthroughs/index_en.htm



Identif y

Then the experts were asked to grade the technological advances using the well-defined measurement scale of TRLs also applied in Horizon 2020. This step was applied only for the identified technological advances, and not for the scientific advances.

The experts were also asked to assess the technological advances based on the four attributes: new, intelligent, leading-performer and structure-changing.

The TRLs benchmark or the classification based on the four attributes was introduced in collecting expert’s opinions with the scope of having a clear idea about the latest technologies on the market and about their impact on our every-day life. The more attributes a technology checks or higher technology readiness levels a technology have, the more important a technology is and the more impact has on our lives (the classification was applied only for identified technologies, not for scientific advances).

STEP 4: Validation of the articles and extracting the keywords describing the technological and scientific advances

All the information collected from experts was gathered in a single database. The links were tested and the articles were reviewed in order to extract the specific keywords describing the technological and scientific advances. Based on the experts’ assessment of technologies, the coordinator analysed the data and classified the technologies based on their TRL levels and the number of attributes.

In Figure 21, the flowchart of the methodology is presented.

Figure 21: Flowchart of the experts’ exploratory approach for the identification of subsequent technological and scientific advances


Advantages:

• The methodology takes advantage of the opinion / expertise of the thematic experts.

• There is no need for any specific software and/or equipment.



Identif y

• It provides with an in-depth and holistic approach.

• It is a highly interactive methodology.

• By this methodology context based insights and implicit ideas can be extracted.

• The methodology can consider the opinion of stakeholder representatives.

• It is a relatively inexpensive and quick data collection technique.

Disadvantages:

• The outcomes might be biased by subjectivity.

• The outcomes might be biased by personal interest.

• The outcomes depend on the diversity group.

• Some topics may not be covered by the expertise.

• The implementation of the methodology is a time consuming.

• The process of collecting information is difficult to automate.

• Limitations for quantifying the results.

• Results may not be replicated in the same manner with the different group of experts or at a different point in time.

• There may be a problem in identifying a large enough group of qualified experts if the evaluator wishes to ensure statistical confidence in the results.

• Experts are unlikely to be equally knowledgeable about a subject area, and so weights should be assigned to the results.

• Like any subjective assessment, expert opinion also introduces a credibility problem.



Identif y


Table 12: Mitigation measures to overcome weaknesses of the experts’ exploratory approach for the identification of subsequent technological and scientific advances


Subjectivity Personal interest

It might be a problem of experts’ opinion credibility which might bias the outcomes.

Expert opinion’s credibility is enhanced if it is done as systematically as possible.

Expert opinion’s credibility is enhanced if the results are supported by other findings as well.

Group diversity

Some areas of interest might be left uncovered due to the limited number of experts.

Designing a well-defined methodology to select the experts (according to their thematic expertise).

Time consuming The time span of collecting experts’ opinion might be too long.

Optimise the group of experts by increasing the number of experts or reducing the time-span by selecting multidisciplinary experts.

Difficult to automate the process of collecting information.

A human error might occur in the final results.

The process of collecting information might be automatized by complex software

Limitations in quantifying the inputs (experts’ opinions).

The main result is a database with experts’ opinions which need further refinement.

Each experts should present its opinion in s structured way, according to a common template, allowing for future analyses and modelling

Results may not be replicated in the same manner with a different group of experts or at a different point in time.

Different groups of experts might point out different results.

The results must be confirmed by other findings as well (obtained with other methods).

There may be a problem in identifying a large enough group of qualified experts if the evaluator wishes to ensure statistical confidence in the results.

Lack of representativeness of the group of experts due to the small number of experts.

The methodology should clearly present the sampling method.

Experts are unlikely to be equally knowledgeable about a subject area.

The results might be biased by the different levels of knowledge.

Weights should be assigned to the results.

The methodology for selecting the experts can be problematic.

A poor methodology might bias the results.

The methodology should be adapted to activities’ objectives (should be very specific)



Identif y

4.2.3.2. Bibliometric analysis for the identification of subsequent technological and scientific advances

Methodology: Bibliometric analysis for the identification of subsequent technological and scientific advances



The general objective of the proposed methodology is the evaluation of the online content analysis for investigating how the subsequent technological and scientific advances are reflected by academic literature – in the form of keywords, phrases and topics.



design the methodology for online content analysis based on bibliometric analysis;

measure the occurrence of relevant key words, phrases and topics in academic publication.

use the specific keywords from the relevant lists of yearly scientific and technology achievements, e.g. Science journal (Science's Breakthrough of the Year) and MIT Technology Review (Breakthrough Technologies)


This methodology was developed in order to examine the assumption that by using an online content analysis based on bibliometric analysis and relevant lists of yearly scientific achievements, early stage technological and scientific advances could be detected.


The bibliometric analysis is an exploratory technique that can be used for investigating, both quantitatively and qualitatively, the coverage of topics or themes of interest across various academic literature formats, such as printed or online journals, books, reports, theses and dissertations and many others.

The recent development of the storage, communication and transfer capabilities triggered significant advances in disseminating the science and technology. The bibliometric analysis is typically used to inform the researchers of the state of the art in their field of interest. It is very likely that a direct and quasi-linear relationship may be claimed between the occurrence of a certain topic and its social impact. Therefore, the bibliometric analysis has been deployed as a valuable tool for investigating how the specific challenges proposed within the Horizon 2020 (H2020) Programme and their subsequent impact on the society are reflected in the academic literature.



Identif y

At present, several sources of evaluating bibliometrics are available, with a wide multi-disciplinary and international coverage of publications, and ISI Web of Science (WoS), Scopus and Google Scholar (GS) are the most used databases for bibliometric analysis. Based on its availability, friendly interface and reasonable performances, Google Scholar was tested as a bibliometric tool for identifying the occurrence of relevant key words, phrases and topics in relation with the Societal Challenges (SC) of the H2020.

Two relevant lists of yearly achievements were identified: Science journal (Science's Breakthrough of the Year) and MIT Technology Review (Breakthrough Technologies).


Exploring Google Scholar for bibliometric analysis is a fast and open procedure. The following methodological procedure was followed:

STEP 1: Formulate the assumption and define the investigation track

the first step was to define: (1) the exact phrases of the Horizon 2020 Societal Challenges (SC), e.g. ‘Health, Demographic Change and Wellbeing’; ‘Smart, Green and Integrated Transport’, etc.; and (2) relevant key words for each SC, as provided by experts in different fields .

STEP 2: Select and prepare the relevant key words, phrases and topics

The experts proposed relevant key words, phrases and topics aimed to reflect as accurate as possible the link between each SC and its actual impact on society, economy and environment. Generally, the words were used as provided, but in some cases they were rejected as considered not enough specific.

STEP 3: Run the Google Scholar search and aggregate the outputs

Google Scholar is available at https://scholar.google.com/, and it allows the advanced search based on different key word combinations (e.g. all the words, exact phrase) based on the given selection.

The occurrence of (1) and (2) selected key words across the academic publications in strict relation with the Horizon 2020 in the GS database was examined for each year between 2012 and 2016. The outputs were aggregated based on the year of occurrence. By using the tab Find articles with the exact phrase, the number of documents containing the SCs as they are formulated, in strict association with the ‘H2020’ topic, was returned for every year between 2012 and 2016. The same procedure was performed for relevant key words.

STEP 4: Add specific keywords from the relevant lists of yearly scientific achievements

Scientific and technology achievements are published in number of scientific and popular sources. Two sources were identified as the most reliable, and are published by the Science journal (Science's Breakthrough of the Year) and MIT Technology Review (Breakthrough Technologies). The keywords that describe specific scientific and technological achievements within three years before the relevance assessment were used for the analysis.



Identif y

STEP 5: Visualize and analyse the outputs

Visualization graphics were constructed based on the results, and they reveal the increasing occurrence of the SCs in the GS bibliometric database along the analysed period, in close relation with the launch and implementation of the H2020 Programme. The visualization allows capturing the trends and the comparison between different SCs. The proposed methodology is illustrated in Figure 22.

Figure 22: Steps of the Google Scholar Exploratory Approach Methodology on identifying subsequent technological and scientific advances


Advantages:

• The Google Scholar analysis may enable to discover the wider societal perspective that is available online.

• GS is an instrument which provide quantitative results expressed as the number of hits of certain key words or topics.

• It is the widest source available for investigating the contents of academic publications.

• One of the major advantages is its flexibility and straightforwardness use.

• Fast updating and capability to capture in near-real time the new and fast changing trends.

• The usage of the relevant lists of yearly scientific and technological achievements is a good benchmark of the globally recognized important discoveries that reflect the trend in the area of subsequent technological and scientific advances

Disadvantages:

• GS and other online bibliometric tools are limited to the public online sources and to online users.

• The online search engines use different algorithms and they may potentially bias the results.



Identif y

• Temporary trends determined by media interest or paid content may also bias the outputs.

• The keywords selection is subjective and their use is spelling sensitive.

• Web of Science and Scopus are both commercial databases and the access may be restricted by financial limitations.


Table 13: Mitigation measures to overcome weaknesses of the bibliometric analysis for the identification of subsequent technological and scientific advances

Weaknesses of the method Issue Mitigation

GS and other online bibliometric tools are limited to the public online sources and to online users.

The online search engines use different algorithms and they may potentially lead to bias in the results.

Temporary trends determined by media interest or paid content may also create bias in the outputs.

The keywords selection is subjective and their use is spelling sensitive.

Web of Science and Scopus are both commercial databases and the access may be restricted by financial limitations.

This access is limited to the public online sources and its users, and as such may provide limited representation of the obtained results. The results can be biased due to different algorithms in the different search engines.

The risk of influence of the trends in interest of the online source users and different paid content when performing online content analysis.

Search for most recent relevant documents that are available online.

Perform the online search by using at least two search engines.

The combination of using several bibliometric tools may be necessary as a validation technique to overcome the possibility of bias in the results.

Expert-assisted analysis could be used to address the bias related to paid content and other shortcomings.

If possible, a pool of expert should be used to address the issue of the inherent subjectivity with the keywords selection.

Acquisition of access from dedicated funds



Identif y

4.2.3.3. Social media analysis for the identification of subsequent technological and scientific advances

Methodology: Social media analysis for the identification of subsequent technological and scientific advances



To identify the key subsequent technological and scientific advances as explicitly or implicitly expressed by citizens through social media analysis – in the form of keywords and phrases



identify explicitly and implicit expressed key technological and scientific advances from citizens’ expressions collected through main social media (Twitter and Facebook) and other relevant online sources (Wikipedia and YouTube);

develop a top-down and bottom-up approach in the social media analysis;

evaluate the use of R-programme as suitable free tool for monitoring data extracted from Twitter and Facebook that allows capturing and analysing citizens’ expressions in particular themes;

validate the list of technological and scientific advances developed from results obtained from the two approaches and from the different social media and other online sources consulted.


Social media analysis is a suitable methodology for including the voice of citizens on the analysis of technological and scientific advances. Social media analysis is increasing recognised as a valuable mean to capture citizens’ expressions and opinions.


The increasing relevance of social media in citizens’ lives opens up new venues for research and policy attempting to capture citizens’ expressions and opinions. The research community and policy-makers are increasingly paying more attention to consider them in multiple different ways. According to recent Eurostat data (EUROSTAT, 2015 Year in Review. Types of social media used, 2015) (EUROSTAT, Individuals using the internet for posting messages to social media sites or instant messaging, 2009) social media (SM) is becoming an important instrument to show people’s opinions and social interests. In fact, the average of individuals and enterprises that are daily employing SM



Identif y

are increasing every year. For instance, in 2014 the European average (EU-28) of enterprises using social media was 33% and one year later was 36%. Additionally, in several countries this percentage reaches 61% as in Netherlands and Ireland. Similarly, the same applies to common people’s use in which the change is also remarkable. The European average (EU-28) was 32% in 2009 while in 2012 increased to 40%, with countries such as Netherlands and Iceland to account for 65% of involvement for 2012.

Criteria for selecting the type of SM to be investigated include the existence of metrics-gathering by Altmetrics analytics and not covered by other experts’ from the group. Considering the above, Twitter and Facebook were selected. Furthermore, it is relevant to take into account that Almetrics analytics also includes data obtained from other sources like Wikipedia (Internet free encyclopaedia) and YouTube (video sharing website) although not being strictly SM, both of them are extremely popular for searches among citizens. Taking into account that the interest of this work is to understand the extent to subsequent needs of society are relevant to EU citizens, these two sources have also been included. Therefore, the methodology described here includes analysis for Facebook, Twitter, Wikipedia and YouTube.

This methodology has been framed under two different but complementary strategies: the top-down and the bottom-up. Top down approach included the relevance of pre-selected keywords defined through social media analysis (Twitter and Facebook) and other sources (Wikipedia and YouTube). Bottom up approach was mainly based on the identification of technological and scientific advances from trending topics generated within each SM or online source analysis in order to obtain a list of technological and scientific advances, without the limitation of pre-selected keywords. The use of the two approaches allows overcoming the limitations brought of applying just only one of the two.


5.2.3.3.2.1 Top-down approach

The top-down approach refers to the use a pre-defined keywords list to be searched in the selected SM (Facebook and Twitter) and other sources (Wikipedia and YouTube).

STEP 1: Selection of the type of Social Media

Social Media have been selected according to their inclusion in Altmetrics analytics: Twitter Facebook, Wikipedia and YouTube.

STEP 2: Selection of keywords

This step is to define the list of keywords to be searched. In this case, a total of 18 keywords have been selected based on the compilation of technological and scientific advances coordinated by an appointed expert for the purpose of Q3.1, when possible, or adaptations of them to be searchable.

STEP 3: Selection of tools for social media analysis

For each of the selected SM and online source, a different tool to collect the data has been applied. In particular, for: (a) Facebook and Twitter: the R Project for Statistical



Identif y

Computing (from now on also called ‘R’)16 has been used to collect data; (b) Wikipedia: the statistics Wikipedia Page, that is its own tool as it collects statistics of page views and it is possible to download the report as CSV file; and (c) YouTube: similarly to the latter, the You tube’s search engine has been used as it provides information of searches and most viewed videos.

STEP 4: Data collection & Analysis for Facebook, Twitter, Wikipedia and YouTube

Data collection is described below for each SM (Twitter and Facebook), and sources (Wikipedia and YouTube) selected. Data collection of Twitter and Facebook has been collected through the ‘R Project for Statistical Computing’, Wikipedia data collection has been used through consulting the ‘page view statistics’, and YouTube from ‘YouTube’s search engine’.

To collect Facebook data, the first step was to install the Rfacebook package in the ‘R Project for Statistical Computing’. Once installed, the objective was to extract information from the public pages of Facebook, and more particularly, all those that contain the keyword searched. R makes possible to define the ‘N’ and in this case N was set to 10 000 pages as it was considered a sufficiently large number to collect the maximum information available. A total of automatic 48 searches were conducted one per keyword. For each keyword, the following information was obtained:

List of Facebook pages: (a) page name and link; (b) category: based on Facebook own classification; (c) geolocalisation: city, country (when available); (d) description provided by the page creator; and, (e) talking about: is the number of visitors that have interacted with the page (e.g. someone who has ‘Liked’ the page, has commented on it or has shared a post).

Selection of the five most ‘Talking about’ rate pages (automatically obtained through a code development): this information reflects the number of users that have interacted more with the Facebook pages (clicked, shared, commented on, etc.).

For the data analysis, a review for the pages with the highest ‘talking abouts’ and geolocation information have been provided, as well as analysis of the five Facebook pages with the greatest interaction (the highest number of ‘talking abouts’).

In order to extract Twitter data, the first step was to install Twitter-R package in the ‘R Project for Statistical Computing’. Once installed, programming should be performed in order to extract the obtained results in an Excel workbook. ‘N’ value was again set to 10 000 and again 48 searches were performed among hashtags, each per keyword. The Twitter-R package need to be connected with the freely available Twitter API. Once it is connected there are two limitations that need to be considered. In particular, the first limitation is that Twitter Search API searches against a sampling of recent Tweets published in the past 7 days and the second is that if a keyword has more than 10 000 tweets, the system stops searching when the ‘N’ is reached. The analysis of

16 https://www.r-project.org/



Identif y

hashtags allows assessing the relevance of a particular keyword in the community, as their presence or absence indicates the level of interest.

There are currently different freely available programmes to draw information from Twitter contents. However, all of them have limitations mainly related to the access to the tweets. In the case of the ‘R Project for Statistical Computing’, despite limitations mentioned above, it is possible to conduct an approximate analysis of the presence and frequency of the keywords selected and to extract conclusions regarding this. Currently, hashtag analyses are common to measure the impact of a particular topic in a given community. For each keyword, the following information was extracted:

List of the tweets obtained through the selected hashtag (containing the keyword) including: (i) list of tweets; (ii) date of publication of the tweet; (iii) favourite count: tweets marked as favourites; (iv) screen name: name of the Twitter user; (v) retweet count; and, (vi) retweet: if the tweet is not original but a retweet.

Summary of numerical data including: (i) total tweets; (ii) original tweets: number of original tweets of the total tweets; (iii) replies: number of tweets which are replying to other tweets of the total tweets; (iv) RTs (Retweets): number of tweets which are a retweet of the total retweets; (v) total RT: number of retweets done from the total tweets; (vi) total favourites: number of favorited tweets; (vii) contributors: amount of people who have tweeted the searched keyword; and (viii) tweet/contributor: average number of tweets per contributor.

In terms of data analysis, besides revising those keywords that account with the largest number of tweets, the engagement measure has been also useful in order to see the sum of tweets in a selected hashtag that are Replies, Favourites, RTs and total RTs.

The Wikipedia Analysis has been conducted through the statistics data tool of Wikipedia, namely: ‘Page view Statistics’. The data collected in the Wikipedia per keyword are the following: (i) whether the keyword has its page in the Wikipedia; (ii) date of query: the date of the query done is written down; (iii) time period: from the available options (10, 20, 30, 60 or 90 days) 90 days were selected so as to obtain data over a long period of time; and (iv) total number of page views: the total number of visits in all the languages in which a specific page has been published.

Regarding the analysis, the option has been to review whether the keyword was present in Wikipedia, number of views, and the number of languages translated, as an indicator for its relevance.

YouTube data collection has been implemented through the use of YouTube search engine, which is the YouTube´s video own search engine. This has allowed selecting manually the necessary data for this analysis, in particular: (i) search of the keyword in order to obtain the videos related to it; (ii) period in which the videos appear, limited to 1 year; (iii) order the results per view counts of the videos; and (iv) selection of the five most viewed videos.

Regarding the data analysis, these are the items that have been reviewed: (i) number of videos resulting for the selected search parameters: it indicates the extent to what said need of society to EU citizens is present or not within the YouTube community; (ii) list of



Identif y

the five videos with the most view counts: it helps obtain information concerning the most visited contents regarding the selected keywords related to technological and scientific advances. Therefore, it allows analysing the contents which have the largest number of visits within the selected parameter.

STEP 5. Comparison of the relevance of the selected keywords across SM and online sources

Because each social media has its own logics for measuring the presence of each keyword in them, there are no common parameters across SM and online sources. To overcome this limitation, for each social media and other sources the most representative measure has been taken: Facebook interactions (talking about), Twitter interactions (Engagement), YouTube (view counts of 5 most videos last year), Wikipedia (number of page views). In each case, a specific range has been defined and has been translated into four levels (Low, Medium, Medium High and High) signed for each SM allowing to compare and to identify which keywords have more or less presence in each data collection source. The following ranges were defined per each social media and other sources included:

• Range on Facebook (measured by Number of Talking about 1-100 (Low), 101-1.000 (Medium), 1001-10.000 (Medium-High), 10.001-80.000 (High).

• Range on Twitter (measured by Number of Engagement, 1-999 (Low) 1.000-9999 (Medium), 10.000-99.000 (Medium-High), (+) 100.000 (High).

• Range on YouTube (measured by number of view counts, 1-1.000 (Low), 1.001-100.000 (Medium), 100.001-999.999 (Medium-High), (+) 1.000.000 (High).

• Range on Wikipedia (measured by number of page views, 1-5.000 (Low), 5.001-100.000 (Medium), 100.001-550.000 (Medium-High), (+) 550.000 (High).



Identif y

Figure 23: Flowchart of the top-down approach for social media analysis for the identification of subsequent technological and scientific advances

4.2.3.3.2.2 Bottom-up approach

The bottom-up approach to this analysis intends to explore the main needs and concerns of the EU citizens based on their direct expressions as they are showed in the following sources: Facebook, Twitter and Wikipedia. Contrary to the top-down approach, instead of having a predefined selected list of keywords, the goal is to search which topics are most referred and consulted by citizens. This bottom up approach has been used to identify technological and scientific advances from SM secondary literature, Twitter and Wikipedia.

STEP 1: Identifying technological and scientific advances from SM secondary literature

Twitter and Facebook produce every year a list of the most relevant topics that have been more presence in their networks: 2015 Year in Review (Facebook) – 10 topics with most talking abouts 17 and 2015 Year in Review Twitter (top trends and most influential topics)18. These reports have served to identify a list of technological and scientific advances.

17 http://newsroom.fb.com/news/2015/12/2015-year-in-review/ 18 https://2015.twitter.com/



Identif y

STEP 2: Identifying technological and scientific advances from Twitter Trending Topics of 15 European Countries

In order to identify NSEUCs, the ‘R Project for Statistical Computing’ has been used to extract the information of the maximum 50 trending topics (TT) for 15 European countries (Austria, Belgium, Denmark, France, Germany, Greece, Ireland, Italy, Latvia, Netherlands, Poland, Portugal, Spain, Sweden, United Kingdom) which are those available on the ‘R Project for Statistical Computing’. Practically this is implemented through the installation of a specific code from TwitterR that allows obtaining trending topics from different countries in one day (in this case during the 19th of September 2016).

Once the results are extracted a first manual screening is conducted in order to identify those topics that would correspond to NSEUCs. In this sense all Trending Topics related to TV Shows, Series, Films and Music (singers or groups), jokes are discarded from the analysis. To do so, two columns are included in the excel file i.e.: 1st column: Is this a NSEUC? - In this column the applicable answers are either ‘Yes’ or ‘No’); and 2nd column: possible keyword – This column has been completed identifying explicit and implicit keywords. For example, the keyword ‘economy’ is explicit in a trending topic but ‘Lesbos” is dealing with refugees’ issues.

Figure 24: Layout of the template used for identifying technological and scientific advances from Twitter Trending Topics of 15 European Countries (Step 2) of the bottom-

up approach of social media analysis

STEP 3: Identifying technological and scientific advances from Wikipedia most viewed pages

Wikipedia (Internet free encyclopaedia) is other relevant online source used mainly for citizens around the world for consulting different topics. Therefore, the most viewed



Identif y

pages per language constitute an indicator to identify those topics most valued by citizens. Wikipedia provides its own tool for consulting the most viewed pages.19 This tool allows downloading a report of those pages with more visits in a concrete period to select (a day, month). In this case a monthly report has been selected (August 2016). Four reports have been downloaded with the most viewed pages in four languages (English, French, German and Spanish). Once reports have been downloaded a review of pages has been done in order to identify technological and scientific advances. A list of technological and scientific advances per selected language has been obtained.

STEP 4: List of technological and scientific advances keywords

The final step is to create a list of technological and scientific advances obtained from different sources explored.

Figure 25: Flowchart of the top-down approach for social media analysis for the identification of subsequent technological and scientific advances


Advantages

It allows both top-down and bottom up approach.

Allows to assess the relevance of each of the predefined technological and scientific advances according to what citizens express in social media and other online sources (top-down approach).

Easy, fast and free to use by non-experts.

Different languages can be taken into account.

Using different social media allows to compare and to embrace greater diversity of publics.

19 https://tools.wmflabs.org/topviews/?project=en.wikipedia.org&platform=all-access&date=last-month&excludes=

STEP 4. List of Keywords



Identif y

Disadvantages

Social media is just the reflection of the opinions of the sub-group of citizens that participates in the social media, not covering the whole population.

The usage of synonyms that are hard to recognize without the context.

Uneven coverage of countries, age and education.

The reliability of the provided content is low in many cases.


Manual annotation of YouTube searches information

Barriers in identifying technological and scientific advances as expressed by citizens from the bottom up approach


Table 14: Mitigation measures to overcome weaknesses of the social media analysis for the identification of subsequent technological and scientific advances


Social media is just the reflection of the opinions of the sub-group of citizens that participates in the social media

Social media analysis collects only data from SM users.

Although data is limited to SM users, it contains more representative citizens’ voices than other sources.


Identification of synonyms to define valid keywords

Two ways for mitigating this weakness; a) To use R-programme to extract associated words, b) To qualitatively analyse conversation on Facebook page (through comments) or to explore interactions under particular hashtags in order to understand in context meaning.

Uneven coverage of countries, age and education

Age and education data is not available; countries data is limited.

No strategy to overcome age and education limitation has been identified. Related to the countries this information is available on Facebook Data collection Top down approach, with the limitation of only those geocalizated pages could be identified. Also countries information is available in the list of 15 European Countries Trending Topics.



Identif y


Reliability of content To ensure a common analysis strategy for avoiding mistakes.

Reliability is ensured with a specific design of validity checks conducted by external researchers.


Limitation of historical data (7-9 days).

The R-programme allows defining a Monitoring process accounting for collecting data throughout time. For instance;

R-Programme allows defining a strategy of monitoring of Trending Topics on 15 European Countries defining a period to extract reports. This monitoring process could be useful to compare Trending Topics from the available European countries and to extract which are the common topics and which are specific. This process could be useful to update the identified technological and scientific advances expressed by citizens.

Manual annotation of YouTube data collection

Technical limitation of YouTube data collection

Technical limitation of YouTube data collection could be overcome with last advances of R-program. Exploring new forms of obtaining YouTube data collection in an automatic manner will be explored.

Barriers in identifying technological and scientific advances as expressed by citizens from the bottom up approach

Scarce technological and scientific advances identified among the trending topics obtained from bottom up analysis. It might be due to the limitation of collecting data within a limited period in time.

In order to mitigate this limitation, it would be useful to conduct a monitoring process that allows collecting more data in different occasions during a particular period of time in order to be able to contrast it chronologically.



Identif y

4.2.3.4. Patent analysis for the identification of subsequent technological or scientific advances

Methodology: patent analysis for the identification of subsequent technological or scientific advances



The general objective of the proposed methodology is to evaluate the value of patent analysis for analysing how the subsequent technological and scientific advances in H2020 and earlier FPs are reflected by patents – in the form of keywords, phrases and topics.



measure the occurrence of relevant key words, phrases and topics in patents and patent technology fields;

design the methodology for using patent analysis to analyse how the technological and scientific advances in H2020 (and earlier FPs) are reflected by patents.


The assumption for using patent analysis to gauge relevance of H2020 and earlier FPs is that, by using patents, one can at least grossly establish whether the FPs have been relevant for a certain type of attempted commercialisation endeavours, namely patenting in fields where patenting is deemed a relevant activity for commercial purposes. In these fields, soaring patent activity can be seen also as reflecting considerable technological as well as scientific advances being made in relation to the prior state-of-the-art. The analysis of relevance of H2020/FP should be considered an (attempted) assessment of relevance of H2020/FP7 for respective patenting activity.

4.2.3.4.1. Short Description of the methodology

‘Patent analysis’ can be understood as an umbrella term for a range of methods that share one thing in common, namely that they are applied to patent data. Based on who is using patent data, one can distinguish also different ways of applying similar methods:

Patent statistics: Using the OECD manual on patent statistics (OECD, 2009) as a starting point, a way of using patent data is “…the measurement of science and technology (S&T), the construction of indicators of technological activity as well as…interpretation of patent indicators.” In this context, patent statistics” refers to the usage of patent data by economists or innovation researchers, who apply statistical methods on patent data as well as, in an adapted manner, methods like



Identif y

citation analysis which are utilised in bibliometrics. The patents are usually looked at from a rather aggregate/macro-level.

Patent analytics: Patents are, however, also analysed by patent applicants, market players in industry and IP service providers like patent attorneys. The object of analysis for the types of patents analysed here is typically much more granular, i.e. often singular patents or patent families; patents that are able to define specific markets; or the patenting activities of certain applicants. The purpose of these analyses is, amongst others, (see Walter & Schnittker, 2016): investigating the state-of-the-art of a technology; avoiding redundant R&D (i.e., R&D that does not need to be undertaken because somebody already patented respective R&D results); avoiding infringing the patent rights of other parties; monitoring competitor activity; analysing strengths and weaknesses of particular patent portfolios; or gathering new ideas for own developments. Because of the wealth of information stored in patent data for competitive purposes, a vivid market has developed regarding in this context regarding ‘patent analytics’ and/or ‘patent landscaping’ (WIPO, 2013).

The boundaries between ‘patent statistics’ as applied by economists and ‘patent analytics’ as applied by users and players of the patent system are blurred, as similar methods are used, and sometimes (e.g. when it comes to analysing complete technology fields), there is a clear overlap. Consequently, a wide range of methods is typically applied to patent data, including: descriptive statistics and econometric modelling performed in bibliographic data; social network analysis (SNA); specifics of bibliometric studies such as citation analysis; different forms of (exploratory) visualisation techniques, including patent landscapes for specific products or technologies; but also different forms of text mining techniques (on the actual content of the patents – abstracts, claims, description) and big data methods.

Applying “patent analysis” for the purpose of the assignment of this expert group may cover therefore a range of approaches, particularly depending on the object of analysis scrutiny (entire wide technology fields or specific products and product classes or applicants, for example). Only a specific selection can be tested within the scope of the assignment.

The assumption for using patent analysis to gauge relevance of H2020 and earlier FPs is that, by using patents, one can a) at least grossly establish whether the FPs have been relevant for a certain type of attempted commercialisation endeavours, namely patenting in fields where patenting is deemed a relevant activity for commercial purposes. Secondly, b) a soaring number of patents in specific technology fields or topics reflects also a technological and scientific advancement to the prior state of the art, as patents have to be novel and need to exhibit an inventive step in order to be granted. Therefore, the analysis of relevance of H2020/FP should be considered an (attempted) assessment of relevance of H2020/FP7 for patenting activity, which in turn reflects technological/scientific advances to a degree in certain fields and with anticipated commercial prospects.

In this context, we attempt to investigate the following questions and their methodological feasibility to measure relevance:



Identif y

• How does the H2020/FPs and their goals, as envisaged in the work programmes and legal documents, compare to general trends in patenting? Do they focus on the same topics and technologies?

• To what extent is there an overlap between researchers active in FP projects vs. the persons named as inventors in patent specifications? If there is a strong overlap in fields that are patenting-prone, one could assume a strong connection between research in FP projects and patenting; if, however, the list of names is quite different, patenting and FP research activity may be considered more detached.

• How do the patents filed by researchers (as inventors) and their host institutions (as applicants) in the FPs compare to peer patenting activity in the fields? Are there topical overlaps? Is there a difference e.g. in terms of whether patents get more or less cited?


In a very first run of the patent analysis, a very simple procedure was used in order to identify those technological fields where patenting activity is above-average increasing and where “hot topics” can be seen in patenting:

STEP 1: Using databases like the OECD Patent Database to identify technology fields with high growth patenting activities

We looked at patenting activity in different technology fields – selected ones and by IPC classes – to identify where patenting activity grew above-average in the past decade. Those technology classes were ranked and the ones with highest growth were selected (i.e., their names used as key words).

STEP 2: Using secondary sources to identify important patenting trends and topics.

In a second step, we also looked at available recent studies and analyses that discussed (current) patent trends20. Important key words describing trends were taken from there, too.

STEP 3: Compare extracted key words with domain expert know-how on H2020/FP domains

The key words were then compiled and provided to the domain experts to gauge whether H2020 and earlier FPs (and the corresponding research) had captured the major patenting trends.

20 e.g., Grevatt, T. (2016): TechInsights' 5 Global Patent Trends for 2016, http://www.eetimes.com/document.asp?doc_id=1328603&page_number=1



Identif y

4.2.3.4.3. Advantages and disadvantages of the method

Advantages:

To our knowledge, using patent data for the purpose of establishing relevance of a research programme has not been yet attempted (unlike impact analysis, where patents have been frequently studied as output of such programmes).As patents are designed to be used for commercial purposes –, patenting activity in certain areas may be used as an indicator, albeit more for expectations of market participants in certain technology areas regarding commercialisation potential of inventions rather than as “true” proxies for innovation.

Patents define enforceable rights in certain technologies, the corresponding freedom-to-operate of patent owners, and against these, eventually, stakes in terms of industrial leadership and competitiveness. This can be seen as advantage for certain types of patentable technologies, innovations, industries and fields.

Disadvantages:

Patents are only relevant for certain technology fields. Fields like software or the outputs of the creative industries (or, in science, of the humanities) do make no or only limited use of patents. A large share of innovations therefore is by principle not traceable through patenting activity, because the underlying subject matter is either not patentable, or there is a low propensity to patent in certain fields/industries.

There is usually a long way to go for most patents until they get commercialised – e.g. prototype building, testing, etc. This variation of technology readiness level makes patents as indicators for actual market-readiness or even market use particularly difficult.

Only a small share of patents can be considered commercially valuable, with distribution of patent value across the stock of patents being highly skewed (see Gambardella, 2005). Many patents can be therefore considered “failed” bets on commercial success and market uptake of innovations.

As with research grants, patents are also only one contributing factor to the success of an innovation or to an impact created by inventions. There are many more factors influencing success of an innovation, including regulation, standards, performance of substitute technologies, or simply acceptance by customers.

It may be difficult to identify clear links between patents and research grants taken out in the course of the FP programmes. A patent by an applicant may be an output of one, several or no FP project(s), e.g. because the patents of research applicants stem from activities not funded by the FPs. There is therefore a need to analyse and identify a link between research project activities and patent activities which is usually not obvious. The current database available to the expert group indicates only some 2 000 patents recorded solely in FP7, which the project participants declared to be outputs of the research projects.



Identif y


Table 15: Mitigation measures to overcome weaknesses of the patent analysis for the identification of subsequent technological or scientific advances


The lack of granularity and specificity when looking at the general/global patenting trends.

The use of exerts’ pre-defined keywords from other methodologies, for searches within the patent databases.

This is to test whether patents in themselves reflect the key goals and topics as defined in H2020 (and not vice versa).

More field-specific approaches and more usage of highly cited patents, e.g. like comparing field specific trends of patents associated with a thematic FP field (and their development) vs. global development, e.g. as possible with the datasets provided for by the EC project ‘Measurement and analysis of knowledge and R&D exploitation flows assessed by patent and licensing data’.21

Technically, this is a departure from the technology-classification approach. The use of patent statistics and bibliometrics could be used to study the overlap of inventor an applicant participation in the FPs, and to compare the general features of these patents to the peer-set of overall patents in the respective fields.

21 https://rio.jrc.ec.europa.eu/en/library/measurement-and-analysis-knowledge-and-rd-exploitation-flows-assessed-patent-and-licensing



Identif y

4.3. Methodologies for the assessment of relevance of European research and innovation framework programmes with: the current priorities based on EU and international strategic policy documents (Q1.2); the needs of society to EU citizens (Q2.2); and the subsequent technological and scientific advances (Q3.2)

4.3.1. Experts’ exploratory approach using human content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2)


The general objective of the proposed methodology is to identify the relevance of European research and innovation programmes (Horizon 2020) using human content analysis.

Specific objective of the methodology:

The specific objective of this methodology is to compare the keywords and phrases extracted for the purpose of Q1.1, Q2.1 and Q3.1 with the texts of legal documents of European research and innovation programmes.


This methodology was developed in order to validate the assumption that the experts’ exploratory approach is a valid and comprehensive approach which produces reliable results, with the goal of detecting key topics in legal documents establishing the framework programme for the purpose of the assessment of relevance.

4.3.1.1. Short description of the methodology

Expert exploratory approach using human content analysis for the assessment of relevance of the European research and innovation programmes shall be implemented by experts through in-depth reading of the legal documents, which establish the European research and innovation programme, and identification of preselected keywords, implicitly or explicitly expressed, to those legal documents. The preselected keywords, phrases and topics can be identified by different methodologies (see chapter 4.2.) specifically designed by the expert group to reflect: (1) key EU and international priorities (Q1.1); (2) the needs of society to EU citizens (Q2.1); and (3) the subsequent technological or scientific advances (Q3.1).

4.3.1.2. Step-by-step description of the methodology

STEP 1: Selection of the legal documents which relevance will be assessed

Methodology: experts’ exploratory approach using human content analysis for the assessment of relevance of H2020

Related Questions: Q 1.2; Q2.2; Q3.2



Identif y

The first step is the selection of the legal documents against which relevance will be assessed. Such legal documents may include for example the legal document (e.g. Regulation) which establishes the framework programme, the legal document (e.g. Decision) which establishes the specific programme implementing the Framework Programme or the legal document such as the Work Programmes implementing the specific programmes and other.

For the assessment of relevance of H2020 the following legal documents were selected:

- the Council Decision of 3 December 2013 (22);

- the 2014–2015 Work Programme adopted in the framework of the specific programme implementing Horizon 2020; and

- the 2016 – 2017 Work Programme also adopted in the framework of the specific programme implementing Horizon 2020.

STEP 2: Selection of levels for the assessment of relevance and classification of the identified keywords according to the structure of the European research and innovation framework programme

After selection of the legal documents, in-depth study of the legal documents with the view to select the level of analysis (e.g. one-level based only on one legal document, or two-level based on two legal documents etc.) and the approach that will be followed for their decomposition (e.g. bottom-up approach or top-down or step-by-step) shall take place.

For H2020 it was decided that the assessment of the content of H2020 will be done in a ‘step-by-step’ approach on the basis of the following parts/texts of the legal documents:

- The Broad Lines of Activities as per Annex I of the Council Decision 2013/743/EU.

- The Calls and Call Topics from the 2014-2015 Work Programme; and

- The Calls and Call Topics from the 2016-2017 Work Programme.

In parallel, the keywords, phrases and topics collected for the purpose of Q1.1, Q2.1 and Q3.1 were allocated in two ways:

(a) Across the H2020 sections. The selection of the sections was based on the section division presented by the official site of European Commission23. The final structure of sections is provided in Table 16.

(22) Council Decision of 3 December 2013 establishing the specific programme implementing Horizon 2020 - the Framework Programme for Research and Innovation (2014-2020) and repealing Decisions 2006/971/EC, 2006/972/EC, 2006/973/EC, 2006/974/EC and 2006/975/EC (2013/743/EU) (OJ L 347, 20.12.2013, p. 965). 23 https://ec.europa.eu/programmes/horizon2020/h2020-sections



Identif y

Table 16: Sections of H2020 considered for the allocation of identified keywords

Section of H2020

I.

Exce

llenc

e Sci

ence

I.1. European Research Council (ERC)

I.2. Future and Emerging Technologies (FET)

I.3. Marie Skłodowska-Curie Actions (MSCA)

I.4. Research Infrastructures, including e-Infrastructures

II.

In

dust

rial

Lea

ders

hip

II.1

. Le

ader

ship

in

enab

ling

and

indu

stri

al

tech

nolo

gies



II.1.3. Space

II.2. Access to risk finance II.3 Innovation in SMEs

III.

Soc

ieta

l Cha

lleng

es III.1 Health, Demographic Change and Wellbeing

III.2 Food Security, Sustainable Agriculture and Forestry, Marine, Maritime and Inland Water Research and the Bioeconomy III.3 Secure, Clean and Efficient Energy III.4 Smart, Green and Integrated Transport III.5 Climate Action, Environment, Resource Efficiency and Raw Materials III.6 Europe in a changing world - Inclusive, innovative and reflective societies III.7 Secure societies – Protecting freedom and security of Europe and citizens


V. Science with and for Society

VI. Cross cutting issues

VI.1 international cooperation VI.2 Social sciences and humanities (SSH) VI.3 Gender

(b) Per methodology applied for their generation i.e. experts’ exploratory methodology, text mining, social media analysis, and patent analysis as illustrated in Figure 26.

Expert approach keywords

Text mining keywords

Social media keywords

Patent keywords

Q1.1 Key words from the policy documents Q2.1 Key words for the social needs Q3.1 Key words for the technology and science

Figure 26: Table for the classification of keywords per used methodology and question



Identif y

Integrated template used is presented in Figure 27.

Q1 keywords

Q2 keywords

Q3 keywords

Section of H2020

Lead

er

Budd

ies

expe

rts’

ex

plor

ator

y

text

min

ing

expe

rts’

ex

plor

ator

y

text

min

ing

Euro

baro

met

er

Soci

al

Med

ia

expe

rts’

ex

plor

ator

y

Soci

al

Med

ia

Pate

nt

keyw

ords

I. Excellence Science II. Industrial Leadership III. Societal Challenges IV. Spreading Excellence & Widening Participation

V. Science with and for Society VI. Cross cutting issues

Figure 27: Layout of the template used for the allocation of keywords per: question and methodology and section of H2020 as needed for the implementation of the experts’

exploratory approach using human content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2)

STEP 3: Organisation of the Expert Group

In the third step organisation of the expert group shall be implemented. In particular, the Expert Group should be organised so as to cover all thematic areas of the identified levels from the selected Legal documents related to the framework programme.

In order to achieve an objective and systematic approach to the assessment of relevance, each expert shall be coupled with another as ‘expert-twinning’ whereby one expert shall have a leading role (‘Leading Expert’), and the other one as a ‘Buddy’ for checking and consulting the leading expert. For each of the H2020 topics identified in the STEP 2, the Leading expert and the Buddy should be assigned.

STEP 4: Tracking the keywords, phrases and topics identified for the purpose of Q1.1, Q2.1 and Q3.1 in the Decision 2013/743/EU, the 2014-2015 Work Programme and the 2016-2017 Work Programme

In the fourth step the keywords, phrases and topics identified for the purpose of Q1.1, Q2.1 and Q3.1 for the specific framework programme (H2020) shall be tracked in the predefined parts/texts of the legal documents by experts, through classical reading.

The template which is illustrated in Figure 28 was developed and used for the tracking of the keywords, phrases and concepts (found for the purpose of Q1.1, Q2.1 and Q3.1) in the Council Decision 2013/743/EU, the 2014-2015 Work Programme (WP) and the 2016-2017 Work Programme.



Identif y

SECTION OF H2020:

Leader:

Buddies:

Experts’

approach keywords

Text Mining keywords

Online keywords

Patent keywords

H2020 section

Council Decision 2013/743/EU or the 2014-2015 WP or the 2016-2017 WP

Q1: Is the Framework Programme in line with the EU and international priorities?

Specific key words found in Council Decision 2013/743/EU (or WPs)

Key words that are not present in Council Decision 2013/743/EU (or WPs)

Common key words identified

Strengths

Weaknesses

Q2 - Is the Framework Programme in line with the needs of society to EU citizens?




Strengths

Weaknesses

Q3 - How well adapted is the FP to subsequent technological or scientific advances?




Strengths

Weaknesses

Figure 28: Layout of the template used for the tracking of the keywords, phrases and topics found for the tracking of the keywords, phrases and concepts (found for the

purpose of Q1.1, Q2.1 and Q3.1) in the Council Decision 2013/743/EU, the 2014-2015 Work Programme and the 2016-2017 Work Programme



Identif y

In order to implement this step for H2020 the following tasks shall be undertaken: (1) The experts will be provided with the list of the keywords identified for Q1.1, Q1.2

and Q1.3 broken down by section of H2020. (2) The experts have to decide if the keywords, phrases and topics identified for Q1.1,

Q1.2 and Q1.3 are present or not present in the Council Decision 2013/743/EU, the 2014-2015 Work Programme or the 2016-2017 Work Programme.

(3) Since the keywords, phrases and topics were allocated according to the method that was used for their development (e.g. experts’ exploratory approach, text mining and social media), the experts shall indicate the terms that were common for all of the methods (e.g. words identified by experts’ exploratory approaches AND text mining AND social media).

(4) Finally, experts shall also identify the strengths and weaknesses of the methods used for the selection of the keywords (e.g. experts’ exploratory approach, text mining, social media).

STEP 5: Making conclusion on relevance

Based on the coverage of the topics identified by the experts according to the above described procedure, the conclusion is made on the assessment of relevance of European research and innovation framework programmes (i.e. H2020) with: the current priorities based on EU and international strategic policy documents (Q1.2); the needs of society to EU citizens (Q2.2); and the subsequent technological and scientific advances (Q3.2)

In Figure 29 the methodology presented step-by-step is illustrated.

Figure 29: Flowchart of the experts’ exploratory approach using human content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2)



Identif y

4.3.1.3. Advantages and disadvantages of the methodology

Advantages:

Human content analysis, allowing for understanding of the context from which presence of keywords, phrases and topics is being identified, especially when it comes to longer phrases and their implicit presence in the legal texts (Council Decision 2013/743/EU and WPs).

The outcome is based on well allocated experts’ thematic expertise to the related sections of the Horizon 2020 and its consequent Work Programmes.

The objectivity of the outcome is checked through the ‘Leader’ and ‘Buddy’ cooperation.

The methodology allows combining use of several methods.

During the implementation of this methodology there is no need to use a special software to process the information input.

This methodology is relatively inexpensive and can achieve quick data collection technique.

Disadvantages

Since the methodology is based on human content analysis, the output may be biased by subjectivity.

The outcome depends on the diversity of the thematic experts as to how well it covers all sections of the H2020.

Using longer specific phrases may result in very limited explicit findings of their presence in the legal texts (Council Decision 2013/743/EU and WPs).

4.3.1.4. Strategies for overcoming disadvantages

Table 17: Mitigation measures to overcome weaknesses of the Expert exploratory approach using human content analysis for the assessment of relevance of H2020 (Q1.2,

Q2.2 and Q3.2)


Since the methodology is based on human content analysis, the output may be biased by subjectivity. The outcome depends on the diversity of the thematic experts as to how well it covers all sections of the H2020. Using longer specific phrases may result in very limited explicit findings of their presence in the legal texts of the framework programme.

The interpretation of the expert can both enhance and decrease the value of the results. Lack of expertise to cover knowledge needed according to section of H2020 may result in the task not being conducted to covering all parts of H2020.

Experts may not be able to find explicit presence of longer phrases in an explicit way.

The objectivity of the method can be increased if at least two control groups of experts (‘Leader’ and ‘Buddy’) participate in the task. Selection and allocation of experts to ensure all parts of H2020 are covered can minimise the subjectivity. Preparation of key words should avoid creation of long phrases as they are found less useful in application of this methodology.



Identif y

4.3.2. Experts’ exploratory approach using computer content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2)


The general objective of the proposed methodology is to identify the relevance of European research and innovation programmes (Horizon 2020) using computer content analysis.


The specific objectives of this methodology are to:

identify explicitly expressed key topics in legal documents of the framework programme;

assess the value of extracted words, phrases and the like from the framework programme for the purpose of the assessment of relevance of the framework programme.


This methodology was developed in order to validate the assumption that the computer aided text mining is a quick tool which produces reliable results, with the goal of detecting key topics in the legal documents of the framework programme for the purpose of the assessment of relevance of the framework programme.


Text mining is a method used in order to provide automated insight in large corpus of text, by extracting the most often-used keywords and phrases as well as recognising topics. Text mining is applied to process text for the purpose of the assessment of the relevance of H2020.

In order to test the methodology, text mining is applied to the III.4 Smart, Green and Integrated Transport section of the Horizon programme:

the Council Decision 2013/743/EU of 3 December 2013 (24);

Part 11 of 2014 -2015 WP ‘Smart, green and integrated transport’ (Societal Challenges)

Part 11 of 2016 -2017 WP ‘Smart, green and integrated transport’ (Societal Challenges)

One expert in the field of the Transport tested the method. The step-by-step process of the method is described in the following section.


Methodology: Experts’ exploratory approach using computer content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2)

Related Questions: Q 1.2; Q2.2; Q2.3



Identif y


The text mining is performed by using the Provalis QDAMiner 4 and Wordstat 7 softwares, and it is conducted according to the following steps (Figure 30):

STEP 1: Pre-processing of the H2020 Decision and WPs The H2020 Decision, WP2014-2015 and WP2016-2017 of the section III.4 Smart, Green and Integrated Transport were downloaded and converted in .txt format by removing all pictures and text formatting. Three separate databases are created in order to conduct text mining (H2020 Decision and two WPs).

STEP 2: Extraction of the full list of the keywords from the H2020 Decision and WPs The full list of the keywords is selected by using the dictionary settings for the key words extraction. Keywords are extracted with the exclusion of the words that are common in English language, e.g. they, we, and, other, and so on. The exclusion dictionary English.exc is provided as part of the Provalis Wordstat software. The software is set to generate the maximum number of 1 000 keywords.

STEP 3: Extraction of the full list of phrases from the H2020 Decision and WPs

The full list of the phrases is extracted from the H2020 Decision and WPs using the following settings: (1) Minimum number of words in a phrase: 2; (2) Maximum number of words in a phrase: 5; (3) Minimum frequency of a phrase: 2.

STEP 4: Extraction of the topics from the H2020 Decision and WPs The list of the most occurring 100 topics is extracted from the H2020 Decision and WPs using the following settings: (1) Minimum combination of keyword and phrases in a topic: 2; (2) Minimum combination of keyword and phrases in a topic: 5; (3) Minimum frequency of a topic: 2.

STEP 5: Evaluation of the keywords, phrases and topics and conclusion on the relevance

One expert in the field of the Transport tested the method, by comparing two sets of keywords: (1) keywords and phrases prepared for the purpose of Q1.1, Q1.2, and Q1.3), and (2) keywords and phrases generated by the text mining of the Decision document and both WPs of the section III.4 Smart, Green and Integrated Transport. Therefore, instead of reading, the expert used only two sets of keywords (from H2020 and WP + from the template). Based on these two sets of keywords the expert tried to assess the relevance.



Identif y

Figure 30: Flowchart of the experts’ exploratory approach using computer content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2)


Advantages:

The methodology is capable to scan in a very short time (e.g. seconds-minutes) a large amount of text that would require hours or even days of work for being read by human experts.

The methodology requires only a trained operator to execute the software, thus limiting the amount of human interaction required and the costs of its execution.

The methodology is per-se objective, and the objectivity of the results produced (i.e. keywords, phrases and topics lists) can be easily documented.

The methodology is implemented in the Provalis QDAMiner 4 and Wordstat 7 softwares, thus it is available on the market and does not require being developed/implemented in software.

The methodology can be carried out with different documents and WPs, thus allowing comparing results in different contexts.

Disadvantages:

The methodology is implemented in the Provalis QDAMiner 4 and Wordstat 7 softwares, thus it is limited by the option that the software allows and does not leave room for modifications.

The methodology does not provide qualitative information, thus not giving indications on the quality of the results.

The evaluation of the results is performed by expert assessment, thus it involves human work-force, which might be expensive, time-consuming and affected by human bias.



Identif y


Table 18: Mitigation measures to overcome weaknesses of the experts’ exploratory approach using computer content analysis for the assessment of relevance of H2020

(Q1.2, Q2.2 and Q3.2)


Lack of interactivity Embedded in the nature of the methodology.

Set a number of intermediate check-points where expert(s) can check the preliminary results, thus allowing human interaction.

Constraints given by the Provalis QDAMiner 4 and Wordstat 7 softwares

Embedded in the nature of the methodology.

Implement custom software, which allows for more degree of freedom, thus implementing custom analyses that may be of specific interest for the application to the H2020.

Missing of qualitative information.


Results from Step 4 (topics) can be preliminary analysed since, compared to keywords and phrases, they can provide more qualitative aspects.

Involvement of human work-force for the evaluation of the results (Step 5).


No specific mitigation measures are envisaged, since the human assessment is highly needed at the end of the methodology.

4.3.3. Text mining approach for the assessment of relevance of H2020 to the identified EU and international priorities (Q1.2)


The general objective of the proposed methodology is to assess if the Horizon 2020 is relevant to the identified EU and international priorities, using text mining methodology if the framework programme is relevant to the identified EU and international priorities


The specific objectives of this methodology are to: compare explicitly expressed key topics in H2020 and WPs and key EU and international policy documents;

evaluate the method of comparison of extracted words, phrases and topics: (1) from H2020 and WPs, and (2) EU and international policy documents - for the purpose of the relevance assessment.


This methodology was developed in order to validate the assumption that the computer aided text mining is a quick tool which produces reliable results in order to assess relevance of H2020, based on the comparison of the legal texts related to the framework programmes with key policy documents reflecting EU and international priorities.

Methodology: Text mining approach for the assessment of relevance of H2020 to the identified EU and international priorities (Q1.2)




Identif y

4.3.3.1. Short Description of the methodology

Text mining is a method used in order to provide automated insight in large corpus of text by extracting the most often used keywords, phrases and topics.

In order to validate the assumption that the computer aided text mining could be used as a tool that produces reliable results with the goal to assess if the Horizon 2020 is relevant to the identified EU and international priorities; the text mining methodology has been developed.

To this end two methodological approaches were used i.e.: (1) the supervised approach and (2) the unsupervised approach.


Provalis QDAMiner 4 and Wordstat 7 softwares were used in order to extract keywords, phrases and topics of the: (1) key documents produced by the key EU and international policy bodies and (2) the Council Decision 2013/743/EU of 3 December 2013. The text mining of key documents was conducted in following steps.

STEP 1: Text mining of the key documents produced by the key EU and international policy bodies

Text mining of the key documents produced by the EU and international policy bodies needs to be conducted in line with the Computer content analysis for the identification of key priorities (as described in Chapter 4.2.1.2). As the result of this methodology, the keywords, phrases and topics should be extracted that would be used for the further analysis.

STEP 2: Preprocessing of the H2020 Decision The H2020 decision was prepared in the txt format with the removing of all of the pictures and text formatting. One database was created in order to conduct text mining.

STEP 2: Extraction of the full list of the keywords from the H2020 Decision

Full list of the keywords was selected using the dictionary settings for the key words extraction. Keywords were extracted using the full list of keywords for the documents with the exclusion of the words that are common in English language, e.g. they, we, and, other, and so on. Exclusion dictionary English.exc is provided as part of the Provalis Wordstat software. The software was set to generate the maximum number of 1000 keywords.

STEP 3: Extraction of the full list of phrases from the H2020 Decision


STEP 4: Extraction of the topics from the H2020 Decision



Identif y

Full list of the most occurring 100 topics was generated with the following settings: (1) Minimum combination of keyword and phrases in a topic: 2; (2) Minimum combination of keyword and phrases in a topic: 5; (3) Minimum frequency of a topic: 2.

STEP 5: Supervised text mining – based on key words and phrases using The European Commission Better Regulations

The Better regulation Toolbox of the European Commission, TOOL #16: IDENTIFICATION / SCREENING OF IMPACTS25 was used in order to identify which common Direct and indirect behavioural changes and policy goals are defined by the EU and international priorities, and by the Horizon 2020 Decision. Text mining analysis was conducted, using selected key words and phrases as a reference for the detection of the social impact. Specific keywords were selected, as the indicators of specific social impact. If the keyword was found in both the H2020 Decision and key documents of the EU and international policy bodies, it was concluded that the H2020 covers the specific social impact. E.g. if keyword gender was found in H2020 Decision and at least one key document of the EU and international policy bodies, it was concluded that H2020 Decision tackles the issue of gender.

STEP 6: Unsupervised text mining – based on the topic extraction

Topic mining was conducted on the paragraph level, using Provalis Wordstat software. The key documents reflecting EU and international priorities were compared with the Horizon 2020 Decision in order to detect to what extent the extracted topics reflect match. Provalis Wordstat was used for extracting 60 topics from the source documents. Topics were extracted on the paragraph level.

STEP 7: Making conclusion on relevance

Based on the comparison (conducted by software) of the topics identified by the topic mining from texts of key policy documents of EU and international bodies as well as by the topic mining from texts of the Council Decision 2013/743/EU, the conclusion is made on the relevance of the H2020 in the relation to the EU and international priorities.

The methodology is presented in the Figure 31 below.




Identif y

Figure 31: Flowchart of the experts’ exploratory approach using the text mining approach for the assessment of relevance of H2020 to the identified EU and international

priorities (Q1.2)


Advantages:

The presented method can be completely automated. The presented method can be conducted by one independent

expert. The time needed for conducting the analysis is very short

(maximum 1-2 days). The method is objective, and the process can be easily repeated by

other expert with the same results (using the same set of documents, same algorithms and same software).

Disadvantages:

The analysis was based on the referenced key documents, that reflect the social impact of the analysed organizations/groups (European Union, G7, OECD, Paris protocol, UN, WHO). However, it is possible that these documents do not fully reflect the depth and the breath of the social impact generated by the observed organizations/groups. Two options are possible: (1) documents present some impact that is actually not relevant for the organization (e.g. only for the marketing purpose), and (2) documents does not present some impact that is actually relevant for the organization (e.g. migration has become an urgent matter that is not taken into account to the substantial matter in Europe 2020, since at that point in time it was not relevant to that high extent as today).

The supervised analysis was based using the European Commission Better Regulations framework as a proxy for the social impacts, grouped into economic, societal, environmental, and human rights impacts. It would be possible to develop



Identif y

a new framework by the Evaluation group, or to use some other framework, and the results of the research would probably be different.

The keywords and phrases were extracted using text mining methodology, which could be considered as the objective endeavour. However, the keywords that were used as the presence of the social impact in the document (for the supervised text mining) were selected subjectively by the researcher, although the control of the keyword relation and the social impact was conducted using the text mining Keyword-in-context analysis. Text documents that list the result of this analysis are also provided.

Different algorithms used for the topic mining could generate different result for the unsupervised text mining.


Table 19 describes mitigation issues to the disadvantages listed in the section above.

Table 19: Mitigation measures to overcome weaknesses of the text mining approach for the assessment of relevance of H2020 to the identified EU and international priorities

(Q1.2)


Selection of the key documents that may not fully reflect the depth and the breath of the social impact generated by the observed organizations/groups

Some of the social needs are not tackled by the key documents.

The additional input key documents could be used in order to provide broader outlook to the key EU and international priorities.

The supervised analysis was based using the European Commission Better Regulations framework as a proxy for the social impacts.

Some of the social needs may not be defined by the European Commission Better Regulations framework.

Other framework should be used for the testing of the comprehensive definition of the social needs.

Selection of the keywords indicating social impact (for the supervised text mining) may not be objective.

Some of the social needs may be detected by present by using the wrong keyword.

Additional expert(s) could be used for detecting and testing the used keywords.

Different algorithms used for the topic mining could generate different result for the unsupervised text mining.

Different topics could be generated by the usage of different algorithms.

Several algorithms and software should be used for testing the results of the topic mining (for the unsupervised text mining).



Identif y

4.3.4. Social media analysis for the assessment of relevance of H2020 (Q1.2, Q2.2, Q3.2)


The general objective of the proposed methodology is to assess if the Horizon 2020 is relevant to the citizens’ voices.


The specific objectives of this methodology are to:

analyse twitter accounts and websites related to H2020.

assess whether the identified NSEUCs and STECSIAs are present in them.


In order to validate the assumption that the social media analysis is meaningful tool which produces reliable results in order to assess relevance of H2020, based on the twitter and websites related to H2020 analysis and to assess whether the identified NSEUCs and technological and scientific advances are present in them.


Social media analysis is proposed to complement the assessment of relevance of Horizon 2020. The main objective is to analyse twitter accounts and websites related to H2020 in order to assess whether the identified NSEUCs and technological and scientific advances are present in them. The analysis allows observing whether the contributions extracted from citizens’ voices are included or not in H2020.


STEP 1: Selection of keywords to compare with H2020 related sources

The list of NEUSCs (see chapter 4.2.2.5.2.2.) and technological and scientific advances (see chapter 4.2.3.3.2.2) obtained from the bottom-up approach of social media has been made operational into searchable keywords.

STEP 2: Selection of H2020 official twitter accounts and funded project websites

The criteria to select the sources for conducting an initial exploratory analysis are:

a) a selection Twitter accounts related to H2020; and

b) at least, one project per H2020 section.

In the following table, the list of the selected sources is provided:

Methodology: Social media analysis for the assessment of relevance of H2020

Related Question: Q 1.2; Q2.2; Q3.2



Identif y

Table 20: List of the selected sources

Source Description Profile

Twitter

#H2020 Hashtag of H2020

@EU_H2020 The official account for the EU's #H2020 research and innovation programme. Managed by the EC's Research & Innovation DG.

@ERC_research EU's European Research Council (#ERC) funds top researchers in Europe, of any nationality, helping them pursue pioneering ideas at the frontiers of knowledge.

@myCordis Official CORDIS account. News and information on EU-funded research projects & results #FP7 #Horizon2020 #CORDISpacks #ribfp7 #researcheu #h2020

@H2020SME We invest in highly innovative European SMEs. #SMEinstrument of #Horizon2020 / #H2020. Official account managed by @EU_commission @EU_EASME

@NCPHorizon2020 The National Contact Point for Horizon 2020 advises companies and researchers applying for EU funding in Research and Innovation #H2020 HQ @Luxinnovation

Luxembourg • horizon2020.lu

@EsHorizonte2020 Toda la información sobre el Programa Marco de Investigación e Innovación de la UE, Horizonte 2020 [All information on the Framework Programme for Research and Innovation EU Horizon 2020]

España • ESHorizonte2020.es

@Innovation2020 #Innovation & #Creativity #Research Projects. Industry SME R&D Innovation #Funding in Europe USA & ASIA #Strategy #Entrepeneurship #Startups #Tech #IoT #H2020

@H2020partners Horizon 2020 Partnerships. EU R&D Projects Networking. Find & Announce Project Partners for H2020 (not an EC official account) follow @EU_H2020 @innovationunion

At least one project for each section of H2020 has been selected (except in the case of the Cross cutting section), analysing their websites and twitter accounts. There are two societal challenges with more than one.

Table 21: Sections of H2020 considered for the allocation of identified keywords

Section of H2020 Website Twitter account

I.

Exce

llenc

e Sci

ence

I.1. European Research Council (ERC)

RECIRC http://recirc.nuigalway.ie @RECIRC_

I.2. Future and Emerging Technologies (FET)

HBP SGA1 https://www.humanbrainproject.eu

@HumanBrainProj



Identif y


I.3. Marie Skłodowska-Curie Actions (MSCA)

CHESS http://www.chessitn.eu @CHESS_ITN

I.4. Research Infrastructures, including e-Infrastructures

EPOS IP https://www.epos-ip.org @EPOSeu

II.

In

dust

rial

Lea

ders

hip

II.1

. Le

ader

ship

in e

nabl

ing

and

indu

stri

al t

echn

olog

ies


POINT - P Over IcN http://point-h2020.eu

@pointh2020


IZADI-NANO2INDUSTRY http://www.izadinano2industry.eu/

@IzadiNano

II.1.3. Space G MOTIT - Galileo- http://gmotit.pildo.com/

@Gmotit

II.2. Access to risk finance

Progress TT http://www.progresstt.eu/ @ProgressTTEU

II.3 Innovation in SMEs This section has been covered with the twitter account @H2020SME

III.

Soc

ieta

l Cha

lleng

es

III.1 Health, Demographic Change and Wellbeing

Tack SHS http://tackshs.eu @TackshsProject


Emphasis Project http://www.emphasisproject.eu/

@EmphasisProject

III.3 Secure, Clean and Efficient Energy

WETFEET http://www.wetfeet.eu/ @Wetfeetproject


Infralert http://infralert.eu Synchro-NET http://www.synchronet.eu Dora Project https://dora-project.eu/

@infralert_eu @Synchro_Net @Project_DORA


Maslowaten http://maslowaten.eu @maslowaten

III.6 Europe in a changing world - Inclusive, innovative and reflective societies

Bilat USA 4.0 http://www.euussciencetechnology.eu/ EULAC FOCUS http://eulac-focus.net

@BILATUSA40 @EULAC_FOCUS



Identif y


III.7 Secure societies – Protecting freedom and security of Europe and citizens

INSPEC2T Project http://inspec2t-project.eu @INSPEC2T


NCP_WIDE.NET http://www.ncpwidenet.eu/ @NCP_WIDENET


FutureTDM http://project.futuretdm.eu/

@FutureTDM

VI. Cross cutting issues

VI.1 international cooperation - -

VI.2 Social sciences and humanities (SSH) - -

VI.3 Gender - -

STEP 3: Data collection and analysis

The data collection has followed different steps for each source. In the case of twitter accounts, the tweets in each account were converted into PDF files in order to facilitate the search of the keywords. Furthermore, the accounts with conversations linked to the hashtag #H2020 were also analysed. In the case of project websites, the analysis included the website sections describing the project and the results obtained. An excel sheet for each source was designed to collect data.

Figure 32: Example of template for Twitter accounts analysis selected

Data Analysis has been structured considering the level of presence (high, medium, low, no presence) of the NEUSC and the technological and scientific advances in the sources analysed.



Identif y


Advantages:

The methodology allows the analysis of Twitter accounts related to H2020, exploring the contents highlighted. The resulting information is useful to check whether citizens’ voices are present in the Twitter accounts selected.

The methodology enables to monitor which priorities of the programme are present in H2020 project’s Twitter accounts and websites selected in order to check whether the identified NSCEUs/STECIAs are considered or not.

Disadvantages:

The use of the R-programme implies to collect only tweets published in the last 7/9 days; therefore it has been used only for the analysis of hashtag #H2020. The data of tweets available in each twitter account selected has been downloaded manually through the conversion in pdf file in order to analyse if keywords selected were present or not.

The selection of keywords analysed is limited to the results obtained in the period covered from bottom up approach.


Table 22: Mitigation measures to overcome weaknesses of the Social media analysis for the assessment of relevance of H2020 and WPs (Q1.2, Q2.2 and Q3.2)


Historical data The collection of data is limited to a short period

To define a monitoring process to provide a monthly or yearly overview with more covered topics on twitter accounts selected.

Amount of Data available The selection of twitter accounts available

Selection of relevant twitter accounts available related to H2020 in order to analyse them for the assessment of relevance. The selection will depend on the aim of the specific need analysis.



Identif y

CHAPTER 5: Application of the methodologies to the relevance questions and key findings in different sections of Horizon 2020

Key findings are presented across different sections of Horizon 2020. Since text mining and social media were applied only partially, this part of the document focuses on the experts’ exploratory approach using human content analysis: (1) examples of the keywords that are covered in legal documents are presented – in most of the cases for Horizon 2020 establishment act, and (2) detailed report on the keywords that are not covered in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes are presented.

Experts identified if the keywords reflecting the EU and International priorities, the needs of EU citizens, and the subsequent technological and scientific advances are directly or indirectly present in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes.

The relevance methodologies were not applicable in the case of Pillar 1-Excellent Science components (i.e. ERC, FET, MSCA, research infrastructures) because of its bottom-up nature.

The key finding of Horizon 2020 is presented in the following section:



II.1.3. Space

II.2. Access to Risk Finance



III.2. Food Security, Sustainable Agriculture and Forestry, Marine, Maritime and Inland Water Research and the Bioeconomy (SC2)

III.3. Secure, Clean and Efficient Energy (SC3)

III.4. Smart, Green and Integrated Transport (SC4)

III.5. Climate Action, Environment, Resource Efficiency and Raw Materials (SC5)




V. – Science with and for Society

VI.1. Cross-cutting Issues: International Cooperation

VI.2. Cross-cutting Issues: Social Sciences and Humanities



Identif y

VI.3. Cross-cutting Issues: Gender.


In the case of II.1.1, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act, and in the 2016-2017 work programme. The needs of EU citizens are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advances are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: boost investment; digital single market; digital technologies; enhance enabling technology; growth and jobs; high-speed internet; increase competitiveness; innovation 2.0 and entrepreneurship; smart grids using ICTs; strengthen innovation chain; improve research and innovation.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: 5G; adaptive design; age appropriate design; cloud computing; data anonymization; data quality; data storage; datification; quantum; robotics; security; wireless; social media; virtual reality; facial recognition; online shopping; personal robot assistant; ubiquitous communication. ; iPad; Soundcloud; Amazon; Twitter; Periscope; iPhone; Snapchat; Data protection; Apple; Android; Internet; Facebook; Software; Wikipedia; Youtube; open source-javascript runtime environment; social networks; online shopping; VPNs (Virtual Private Network); RAID (redundant array of independent disks).

Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: Big Data; Speech technology; Advanced machine learning; Quantum computing; eHealth; Software alternative to pesticides; ICT for safer environment; autonomous vehicles; mobile communication; smartphone; software-controlled hardware; software patents; Ultraprivate Smartphones; Neuromorphic Chips; Microscale 3-D Printing; Mobile Collaboration; Oculus Rift; 3-D Images; Alternative online payments; Conversational Interfaces; Robots That Teach Each Other; Internet devices powered by Wi-Fi and other telecommunications signals; Robots that cooperate; Artificial Intelligence.

However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: bridge digital divides and cybercrime. All of the relevant keywords are found in both 2014-2015 and 2016-2017 work programmes.



Identif y


In the case of H2020 section ‘Nanotechnologies, Advanced Materials, Advanced Manufacturing and Processing, and Biotechnology’, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities, the needs of EU citizens and the keywords related to the subsequent technological and scientific advances are to a high degree covered in the Horizon 2020 documentation, both in the establishment act as well as the 2014-2015 and the 2016-2017 work programmes.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: Improve research and innovation; strengthen innovation chain; increase competitiveness; enhance enabling technology; stimulate investment in new technologies. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: nature-based solutions; eco-innovation; smart insulation; vacuum insulation; aerogel insulation; biological building materials; phase change materials; nanotechnologies; graphene; molecular; microsensors; nanosafety; nanopharmaceutical; efficiency; electronics; photonics; nanodevices; nanomaterials; nanosystems; miniaturisation; nanorisk; coating; laser; composites; metallurgy; hybrid; carbine; biomaterials; biotechnology; bioengineering; biodegradable; biomanufactiring; KET; GMO; pharmaceutical; cell; bioprocessing; sustainability; genomics; molecular; biosystems; biocatalysis; bioinformatics; metagenomics; LCA; SME; composites; lightweight; 4.0; circularity; robotics; STEM_CELL_RESEARCH; STEM_CELL_THERAPY. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: Organic semiconductor; Black phosphorus; 2D Materials; Nanosensors and the Internet of Nanothings; Phenolic carbon; Tissue engineering; Biofibre-based pyjamas; Medical technology; autonomous vehicles; mobile communication; smartphone; software-controlled hardware; software patents; standard essential; technology convergence.

However, the following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act: nature-based solutions; eco-innovation; miniaturisation; GMO and robotics. However, as it was above mentioned, these keywords were found in in 2014-2015 and 2016-2017 work programmes.

II.1.3. Space



Identif y

In the case of II.1.3, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act, 2014-2015, and in the 2016-2017 work programmes. The needs of EU citizens are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. Also the subsequent technological and scientific advance is covered to a high degree.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: research and innovation; increase competitiveness; enabling technology. Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: navigation; geoinformation; climate; satellite; Copernicus; Galileo; surveillance; tracking; weather; safety; telecommunications; environment. Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: Rosseta mission; Agile Robots; Cube Sat.


II.2. Access to Risk Finance

In the case of II.2, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act, 2014-2015, and in the 2016-2017 work programmes. The needs of EU citizens are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords.

For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: boost investment; increase competitiveness; fund absorption; effective financial instruments; high-risk loans; innovative start-ups; trade and investment; financial sector; public finances; capital markets; financial instruments; financial free trade; improve the business environment; access to finance; funding mechanisms.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: SME; finance; micro-enterprise; small enterprise; crisis; banking; tax; competitiveness; agreements; stakeholders; transparency; business associations;



Identif y

economic consultancies; boost economic growth; competition authority; economic effectiveness.

However, the following keywords from the key strategic documents and needs of EU citizens were not covered in Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: micro-enterprise; crisis; tax; transparency.


In the case of II.2, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act, 2014-2015, and in the 2016-2017 work programmes. The needs of EU citizens are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. Also the subsequent technological and scientific advance is covered to a high degree.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: improve research and innovation; strengthen innovation chain; boost investment; increase competitiveness; entrepreneurship; innovative start-ups; improve the business environment.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: new enterprises; low unemployment; digital skills; digital economy; industry 4.0; entrepreneurship; knowledge transfer.

Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: alternative currencies (bitcoin & lebocoin); Computer technology; Electrical machinery; Digital communication; Medical technology; autonomous vehicles; mobile communication; Smartphone; software-controlled hardware; software patents.



In the case of SC1, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act – 2013 and to the medium degree in the 2014-2015 and the 2016-2017 work programmes. The needs of EU citizens are to a medium degree covered in the Horizon 2020 establishment act – 2013 and the 20142015 and 2016 -2017 work



Identif y

programme. When looking at the subsequent technological and scientific advances, the identified keywords are to a medium degree covered in the Horizon 2020 establishment act – 2013 and the 20142015 and 2016 -2017 work programme. The coverage is presented in the following tables.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act – 2013 and the 2014-2015 and 2016 -2017 work programme are: prevention and treatment, public health, antimicrobial resistance, health systems, and mental health. Non-communicable disease was covered only in 2014-2015. Infectious disease and health care was covered in Horizon 2020 establishment act, and 2016-2017 work programmes. Health targets and neglected tropical diseases were not covered by any of the documents.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: health service; health literacy; poverty; ageing; sustainable health; climate change; wellbeing; cancer; cardiovascular; prevention; care; illness; vaccines; participation; tuberculosis; mental illness; drugs. The following kewords were not covered by any of the key documents: reproductive health; nervous system; Asperger-Syndrome; Zika virus; Lyme disease; multiple sclerosis; syphilis; fibromyalgie; meningitis; and human papiloma virus.

Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: vaccines; Health and safety; nutrition; HIV; diabetes; Brain Mapping; Genome Editings; Simple blood tests for cancer; Growing human brain cells for diagnostics; Immune Engineering; Ageing; DNA sequencing; Psychology research. The The following kewords were not covered by any of the key documents: Optogenetics; Systems Metabolic Engineering; Molecular methods; Internet of DNA; DNA App Store; Chips that mimic the brain; Manipulating memories; A vaccine against Ebola; Bioengineered yeasts capable of producing the baine; Proteins by design; Metalenses.

III.2. Food Security, Sustainable Agriculture and Forestry, Marine, Maritime and Inland Water Research and the Bioeconomy (SC2)

In the case of SC2, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities, the needs of the EU citizens and the subsequent technological and scientific advances are to a high degree covered in both the Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: food safety; hunger; rural infrastructure; plant and livestock gene banks; agricultural in developing countries; oceans and international law; ocean acidification; agricultural productivity; small-scale food producers (including women, indigenous peoples, family farmers, pastoralists and fishers); sustainable consumption; sustainable



Identif y

food production; resilient agricultural practices; land and soil quality; regulate harvesting; overfishing, illegal fishing; plans to restore fish stocks; sustainable use of marine resources; limit extreme food price volatility; maintain genetic diversity of seeds; maintain cultivated plants; maintain farmed and domesticated animals; maintain wild species; natural resources; green growth; resource efficient; sustainable development goals; infrastructure investment; natural disasters; food security; food nutrition; renewable energy; rural development; climate and energy; climate action.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: sustainability; trade; agriculture; biodiversity; food; security; sustainable; forestry; inland; water; bio-economy; rural; hunger; land; development; fisheries; sea; production; consumption; impact; biological; ecological; biotechnology; healthy; consumers; acidification; degradation; ocean; biomass; fresh; drinking; waste; ecosystem; aquatic; overfishing; circular economy; desalination; food they buy; everyday products; worry about the quality.

Finally, these keywords from the subsequent technological and scientific advances were found present in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: sustainable intensification; ocean acidification; rural development; high nature value farming; agro-ecology; Protecting biodiversity; Agricultural Drones; Reverse-osmosis desalination plant ; Supercharged Photosynthesis; Precise Gene Editing in Plants; Evolutionary Biology.

The following keywords from the key strategic documents, needs of EU citizens and subsequent technological advances were not covered in Horizon 2020 establishment act, the 2014-2015 work programme and the 2016-2017 work programme: poverty; industrialisation; GMO.

III.3. Secure, Clean and Efficient Energy (SC3)

In the case of SC3, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 documentation, both in the establishment act as well as the 2014-2015 and the 2016-2017 work programmes. The needs of EU citizens and subsequent technological and scientific advances are to a medium degree covered in the Horizon 2020 documentation, both in the establishment act as well as the 2014-2015 and the 2016-2017 work programmes.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: decarbonising the economy; reduce GHG; increase renewable sources use; increase energy efficiency; smart grids using ICTs; facilitate energy investments; clean energy; diversify the energy sources; reform energy policy;; natural res; low carbon economy; renewable energy; climate and energy; greenhouse gas emissions; green



Identif y

growth; climate action; and reduce emissions. Reduce energy dependency and European Energy Union (only in 2014-2015 and the 2016-2017 work programmes).

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: bioenergy; biofuel; building performance; building renovation; carbon; climate change; CO2; competitiveness; consumers; consumption; decarbonisation; electricity; emissions; end-user; energy efficiency; forest; fossil; geothermal; health; heating; local; market; nuclear; pollution; pretreatment; recycling; refurbishment; renewable; renewable energy; renewables; renovation; resilience; resources; risk; secure; solar; storage; sustainability; sustainable development; transport; wind; energy; smart appliances. Following keywords were found in only in Horizon establishment act, but not in the work programmes: buildings recyclability; circular economy; cradle to cradle; energy flexibility; green infrastructure; lifestyle quality; multi generation living; resource use inefficiency; source separation; sustainable living; vertical gardens; solar panels; air pollution; types of waste; environmental decisions; circular economy; biological architecture. Following keywords were covered in Horizon establishment and in only one of the programmes: batteries; biodiversity; building energy; building integrated renewables; carbondioxide; deforestation; eco design; eco-districts; eco-innovation; ecological sites; fusion; households; minerals; mitigation; on-site renewables; smart grid; sustainable building; energy saving; garden waste. Following keywords were present only in one of the work-programmes, but not in the Horizon establishment act: hydrocarbons; indoor pollution; LCA; Poor indoor environment; prosument; green housing; health impact; insulation; Nearly zero-energy buildings; smart insulation; wind farms. Following keywords were not present in any of the documents: adaptive design; aerogel insulation; age appropriate design; artificial intelligence; biological building materials; blue infrastructure; city farming; climate policies; district energy efficiency; high performance ventilation; home apps; hybrid ventilation; nature-based solutions; overconsumption; passive house; phase change materials; robotics; sustainable lifestyle; big polluters; agricultural pollution; home appliances; soil degradation; IAEA; flixbus (social public transportation initiatives); sustainable neighbourhood.

Finally, these keywords from the subsequent technological and scientific advances were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: pollution; emissions; low-carbon; smart buildings; sustainable; renewable; electricity; sustainability AND energy; Hybrid airships ; Smart Wind and Solar Power; High-efficiency solar panels. Following keywords were found in only one of the documents: smart cities; local energy; pollution; ecological architecture. Green housing was not covered in any of the documents.

III.4. Smart, Green and Integrated Transport (SC4)

In the case of SC4, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and



Identif y

International priorities, needs of EU citizens and the subsequent technological and scientific advances are to a high degree covered in the Horizon 2020 documentation, both in the establishment act as well as the 2014-2015 and the 2016-2017 work programmes.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: decarbonising the economy; reduce greenhouse gas emissions; increase renewable sources use; modernise transport sector; develop green technologies; increase competitiveness; natural resources; resource efficient; infrastructure investment; low carbon economy; renewable energy; aviation security; climate and energy; greenhouse gas emissions; low carbon; smart growth; climate action; international civil aviation; reduce emissions. Smart grids using ICTs was present in Horizon 2020 establishment act and the 2016-2017 work programme, while green growth was present only in 2014-2015 work programme.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: transport; electric cars; smart; travel; oil; decarbonisation; infrastructure; mobility; trade; network; greenhouse; emissions; climate; green transport; electric vehicles; e-mobility; low carbon; environment; carbon footprint; emissions; GHGs; clean air; CO2; decarbonisation; green transport; green; integrated; transport; electric ; vehicle; road; emissions; greenhouse; aviation; mobility; congestion; efficiency; electrical; logistics; noise; traffic; pollution; climate; urban; regional; car; Mobility (transport); Public Transport; routes. Battery was present in Horizon 2020 establishment act and the 2016-2017 work programme, while Mediterranean Corridor was not present in any of the documents.

Finally, these keywords from the subsequent technological and scientific advances were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: low carbon transport; Urban mobility; Electric vehicles; air transport; pollution AND transport; climate AND transport ; Agile Robots; Car-to-Car Communication; Reusable Rockets; Autopilot cars.

III.5. Climate Action, Environment, Resource Efficiency and Raw Materials (SC5)

In the case of SC5, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities and the needs of EU citizens are to a high degree covered in the Horizon 2020 documentation, both in the establishment act as well as the 2014-2015 and the 2016-2017 work programmes, while to a medium degree covered when looking at the subsequent technological and scientific advances.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: water scarcity; trafficking of protected species; mountain ecosystems;



Identif y

biodiversity and ecosystems; coastal and marine areas; decarbonising the economy; green technologies; terrestrial and inland freshwater ecosystems; climate change; global warming; water quality; energy efficiency; recycling; water-use efficiency; limit temperature increase; low-carbon economy; hazardous chemicals and materials; biodiversity loss; greenhouse gas emissions; pollution; tackle resource scarcity; wastewater treatment; water efficiency; water harvesting; water recycling and reuse technologies; climate change; climate action; climate and energy; climate finance; energy efficiency; GHG emissions; global emissions; green growth; greenhouse gas emissions; industrial policy; infrastructure investment; low carbon; natural disasters; natural resources; reduce emissions; renewable energy; resource efficiency; sustainable development; low carbon economy; raw materials. Following keywords were present in only in one of the work programmes: sanitation and hygiene; desertification; desalination; food waste at the retail and consumer levels.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: adaptation; biodiversity; biological architecture; circularity; climate action; climate change; climate policies; deforestation; eco-innovation; ecosystems; energy efficiency; environment quality; equilibrium of natural and built environment; green infrastructure; greenhouse; high performance ventilation; hybrid ventilation; mineral resources; mitigation; overconsumption; pollution; raw material; renewable energy; resilience; resource; resource assessment; risks; scarcity; sustainability; sustainable lifestyle; sustainable living; sustainable urban environment; temperature increase; vulnerability; waste; circular economy; waste management; renewable energy; separate waste; environmental protection; energy efficient; reduce waste; water pollution; efficient use of resources; energy consumption; generating too much waste; natural resources; raw materials; plastic bags; recycled materials; electronic waste; water consumption; climate change ; environment ; sustainability; environmental risk ; wind energy; solar energy ; recycling; renewable energy. Following keywords were present in Horizon 2020 establishment act and only one of the work programmes: material loops; recycling; teledetection; household waste; waste for recycling, resulable, globalization. Following keywords were not present in Horizon 2020 establishment act, but were present in only one of the work programmes: nature-based solutions; buildings recyclability; food waste; plastic waste, noise pollution, food waste. Following keywords were not found in any of the documents: aerogel insulation; city farming; footprint; greening cities; home apps; indoor environment; smart insulation; micropollutants; rural; smart appliances; vertical gardens.

Finally, these keywords from the subsequent technological and scientific advances were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: development strategies; economic and development cooperation; economic crisis; education and training; education systems; emerging economies; employment rate; equal opportunities; financing for development; growth and jobs; poverty eradication; quality education; quality of life; respect for human rights; single market; social protection; sustainable and inclusive growth. However, the following keywords were not covered by any of the documents: Artificial photosynthesis; Data base on climate change; Energy storage; Green light-emitting diodes; nuclear fusion;



Identif y

Plasma pre-treatment detecting deformations and movements landslides; Recyclable thermoset plastics.


In the case of SC6, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a low degree covered in the Horizon 2020 establishment act – 2013 and the 2016-2017 work programmes and to a medium degree covered in the and the 2014-2015 work programme. The needs of EU citizens are to a medium degree covered in the Horizon 2020 establishment act – 2013 and to a high degree covered to the 2014-2015 work programme and the 2016-2017 work programme. When looking at the subsequent technological and scientific advances, the identified keywords are to a high degree covered in the Horizon 2020 and the 2016-2017 work programme and to a medium degree covered in the 2014-2015 work programme.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: development strategies; economic and development cooperation; economic crisis; education and training; education systems; emerging economies; employment rate; equal opportunities; financing for development; growth and jobs; poverty eradication; quality education; excellent science, quality of life; respect for human rights; single market; social protection; sustainable and inclusive growth. The following keywords were covered by the Horizon 2020 establishment act and at least one of the work programmes: enhance performance of education systems; increase development of new skills; poverty; growth and jobs; live in dignity; simplify regulations; well-being of societies; inclusive institutions; digital economy; fundamental rights; persons with disabilities; single currency; boost for new jobs; speed internet. The smaller number of keywords was present in one or both work programmes, but was not present in Horizon 2020 establishment act: fight corruption; fiscal consolidation; fiscal transparency; information sharing; neighbourhood policy; refugees and other displaced persons; anti-corruption capacity building; provide access to justice; social protection policies; internal market; money laundering, debt on a sustainable path.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: innovation talents and skills; public sector innovation; health innovation; inclusive innovation; citizens empowerment; partnerships with citizens; economic growth; quality of life; economic and financial crisis; competition policy; level of education; employment opportunities; facing financial difficulties; social relationships; solidarity and responsibility; household income; medical expenses; single market; social factors; financial crisis; financial incentives; single currency; economic analyses; economic crisis; employment opportunities; dialogue & politics; justice; migration; politics; public policies; social inclusion; work; solidarity; jobs; leadership; public opinion;



Identif y

political issues; peace; human rights; politics. The following keywords were present in at least one of the documents, but not all of them: public sector reform; Refugees; Social Innovation; Discrimination; Islam; bill of rights; knowledge creation and diffusion; Sustainable Tourism.

Finally, these keywords from the subsequent technological and scientific advances were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: social inclusion; migration management; migration; natural heritage; demography; Competition; Rights of persons with disabilities; Education; Employment. The following keywords were present in at least one of the documents, but not all of them: income inequality; poverty; justice; sectorial competitiveness; dignity.


In the case of SC7, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a low degree covered in the Horizon 2020 establishment act – 2013 and in the 2016-2017 work programme and to a medium degree covered in the 2014-2015 work programme. The needs of EU citizens are to a medium degree covered in both the Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes. When looking at the subsequent technological and scientific advances, the identified keywords are to a high degree covered in both the Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: justice; well-being of societies; climate-related extreme events; economic, social and environmental shocks and disasters; secure cyberspace; law enforcement cooperation; organised crime; cybercrime; terrorism; radicalisation; manage migration; combat all forms of organized crime; peace and security; terrorist groups; countering terrorism and violent extremism; cyber security. The following keywords were present in at least one of the documents, but not all of them: personal data protection; fundamental rights; mutual trust; promote peaceful and inclusive societies; access to justice; human trafficking; smuggling; cross-border crime; legal migration; anticipate events early; stronger security and defence; cooperation in defence procurement; reinforce the work of Frontex; rapid border interventions; violent extremism; data protection; international law; social and territorial cohesion; energy security; border security; early warning; exit from the crisis; fight corruption; anti-corruption capacity building; inclusive institutions; nationality discrimination; gender discrimination; racial discrimination; ethnic discrimination; religion discrimination; belief discrimination; discrimination for disability; age discrimination; sexual orientation discrimination; discrimination for people belonging to a minority; ensure equal opportunity; reduce illicit financial and arms flows; substance abuse, narcotic drug abuse and alcohol; global governance; vulnerable situations; financial stability; hazardous chemicals. The following keywords were not present in any of the documents: neighbourhood policy; fight abuse;



Identif y

reduce exploitation; modern slavery; child labour; child soldiers; nuclear safety; terrorist financing; chemical weapons; common asylum; free movement; Minsk agreements; tax evasion.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: borders; cooperation; crime; crisis; cybercrime; defence; energy; environment; extremism; freedom; human rights; mobility; peace; police; policy; privacy; protection; radical; radicalization; rights; risks; security; surveillance; terrorism; threat; transparency; privacy and data; terrorism; justice; security;

The following keywords were present in at least one of the documents, but not all of them: aid; citizenship; equality; fairness; immigration; inclusion; inequality; jobs; migration; poverty; Schengen area; social exclusion; violence; data security; daily life to protect; dependence on technology; violence; bullying; refugees; coping with the crisis; social exclusion.

The following keywords were not present in any of the documents: prisons; anti-Islam; asylum; bioweapon; xenophobia; flexicurity; introducing heavier fines for offenders; loan sharks; financial supervision.

Finally, these keywords from the subsequent technological and scientific advances were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: surveillance; Military equipment; Crime and terrorism; Ultraprivate Smartphones. The following keywords were present in at least one of the documents, but not all of them: extremism, Asylum and protection of removal.



In the case of IV, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act, to a medium degree covered in the 2014-2015 work programme and to a low degree covered in the 2016-2017 work programme. The needs of EU citizens are to a high degree covered in the H2020 establishment act while to a medium degree covered in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advances were not applicable for analysis to the section IV of Horizon 2020.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: improve research and innovation; strengthen innovation chain; enhance performance of education systems. The following keywords were present in at least one of the documents, but not all of them: reinforce higher education; boost for new jobs.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: excellence; participation; mobility; education; teaming; collaboration;



Identif y

capacity; citizens involvement. The following keywords were present in at least one of the documents, but not all of them: job, twinning, talent, skills.

The subsequent technological and scientific advances were not applicable for this section of Horizon.

V. – Science with and for Society

In the case of V, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU and International priorities are to a high degree covered in the Horizon 2020 establishment act and in the 2016-2017 work programme, while to a medium degree covered in the 2014-2015 work programme. The needs of EU citizens are to a high degree covered in the H2020 documentation, the establishment act and in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advances were not applicable for analysis to the section V of Horizon 2020.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: boost for new jobs; enhance performance of education systems; reinforce higher education; strengthen innovation chain; improve research and innovation. The following keyword was present in at least one of the documents, but not all of them: sustainable development and lifestyles. The keyword „complete tertiary education“ was not covered by any of the documents.

Additionally, the following keywords emerging from the needs of society to EU citizens were found in the Horizon 2020 establishment act and the 2014-2015 and the 2016-2017 work programmes: awareness; citizen; education; employment; engagement; ethics; gender; integrity; knowledge; layman; public; schools; science; social; society; transdisciplinary; Citizens involvement; social sciences. The following keywords were present in at least one of the documents, but not all of them: investment; preparedness; participation.

The subsequent technological and scientific advances were not applicable for this section of Horizon.

VI.1. Cross-cutting Issues: International Cooperation

In the case of the H2020 cross-cutting issue related to international cooperation, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU an International priorities are to a high degree covered in the Horizon 2020 establishment act – 2013 and the 2016-2017 work programme, while they are to a medium degree covered in the 2014-2015 work programme. The needs of EU citizens are to a medium



Identif y

degree covered in the Horizon 2020 establishment act – 2013, while they are to a high degree covered in both 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords.

For example, the following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: simplify regulations; strengthen capacity for early warning, risk reduction and management of national and global health risks in developing countries; keep markets open; encourage climate-resilient sustainable development by promoting international cooperation; ease access to markets; better cooperation with third countries; enhance international support for implementing effective and targeted capacity-building in developing countries to support national plans to implement all the Sustainable Development Goals; recruitment, development, training and retention of the health workforce in developing countries, least developed countries and small island developing States; transboundary cooperation in water resources management; international support to developing countries; international cooperation on and access to science, technology and innovation. The following keywords related to the needs of EU citizens were found in the Horizon 2020 establishment act: open markets; EU enlargement; public-private partnerships (PPPs); international relations; EU enlargement.

The following words emerging from key strategic documents worldwide and social needs of EU citizens were not found in the Horizon 2020 establishment act: free trade Agreement with the USA; encourage trade liberalization efforts; fight protectionism; trust in transatlantic relations; assist developing countries in attaining long-term debt sustainability through coordinated policies; duty-free and quota-free market access on a lasting basis for all least developed countries; development, transfer, dissemination and diffusion of environmentally sound technologies to developing countries; international cooperation and capacity-building support to developing countries in water- and sanitation-related activities and programmes; international trade; production-consumption systems; NGOs. The following words emerging from key strategic documents worldwide and social needs of EU citizens were not found in the 2014-2015 work programme: free trade Agreement with the USA; encourage trade liberalization efforts; fight protectionism; ease access to markets; assist developing countries in attaining long-term debt sustainability through coordinated policies; duty-free and quota-free market access on a lasting basis for all least developed countries; recruitment, development, training and retention of the health workforce in developing countries, least developed countries and small island developing States. The following words emerging from key strategic documents worldwide and social needs of EU citizens were not found in the 2016-2017 work programme: free trade Agreement with the USA; encourage trade liberalization efforts; fight protectionism; duty-free and quota-free market access on a lasting basis for all least developed countries; recruitment, development, training and retention of the health workforce in developing countries, least developed countries and small island developing States.

VI.2. Cross-cutting Issues: Social Sciences and Humanities



Identif y

In the case of the H2020 cross-cutting issue related to social sciences and humanities, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU an International priorities are to a high degree covered in the Horizon 2020 establishment act – 2013, 2014-2015 and the 2016-2017 work programme. The needs of EU citizens are to a medium degree covered in the Horizon 2020 establishment act – 2013, 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act: improve research and innovation; strengthen innovation chain; poverty; justice; fundamental rights; mutual trust; fight abuse; reduce trafficking; hunger; access to safe, nutritious and sufficient food by all people including infants; access to essential health-care services; resilience of the poor and those in vulnerable situations. The following keywords related to the needs of EU citizens were found in the Horizon 2020 establishment act: employment; unemployment; inclusion; wellbeing; poverty; sharing; education; social inclusion.

The following keywords emerging from key strategic documents worldwide were found in the 2014-2015 and 2016-2017 work programmes: improve research and innovation; access to essential health-care services; access to safe, nutritious and sufficient food by all people including infants; mutual trust; fundamental rights; personal data protection; strengthen innovation chain; poverty; justice. The following keywords related to the needs of EU citizens were found in the 2014-2015 and 2016-2017 work programmes: sharing; education; employment; social inclusion; inclusion; wellbeing; unemployment; poverty.

The following keywords emerging from key strategic documents worldwide were not found in the Horizon 2020 establishment act: personal data protection; reduce exploitation; fight all forms of violence; fight torture of children; extreme poverty; nationality discrimination; sex discrimination; racial discrimination; ethnic discrimination; religion discrimination; belief discrimination; discrimination for disability; age discrimination; sexual orientation discrimination; discrimination for people belonging to a minority; adopt social protection policies. The following keywords emerging from key strategic documents worldwide were not found in the in the 2014-2015 and 2016-2017 work programmes: fight abuse; reduce exploitation; reduce trafficking; fight all forms of violence; fight torture of children; extreme poverty; hunger; nationality discrimination; sex discrimination; racial discrimination; ethnic discrimination; religion discrimination; belief discrimination; discrimination for disability; age discrimination; sexual orientation discrimination; discrimination for people belonging to a minority. The following keywords related to the needs of EU citizens were not found in the 2014-2015 and 2016-2017 work programmes: cultural integration.



Identif y

VI.3. Cross-cutting Issues: Gender

In the case of the H2020 cross-cutting issue related to gender, the overall outcome of the experts’ exploratory approach using human content analysis revealed that the identified keywords from the EU an International priorities are to a medium degree covered in the Horizon 2020 establishment act – 2013, and high degree in 2014-2015 and the 2016-2017 work programme. The needs of EU citizens are to a high degree covered in the Horizon 2020 establishment act – 2013, 2014-2015 and 2016-2017 work programmes. The subsequent technological and scientific advance was not evaluated since the approach did not generate relevant keywords.

The following keywords emerging from key strategic documents worldwide were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: gender equality; give women equal rights to economic resources; give women equal rights to financial services, inheritance and natural resources; women’s equal opportunities for leadership; capacity-building such as education and training of women in Science, Technology, Engineering and Mathematics (STEM) fields; sex discrimination; promote and protect women’s rights; support policy for the promotion of gender equality and the empowerment of all women and girls; promote women empowerment; protect women migrants; recognize the role of women as key actors to ensure peace and security; give women equal rights to ownership; give women equal rights control over land and property; eliminate violence against women and girls. The following keywords related to the needs of EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes: gender; societal; vulnerability; education; awareness.

The following keywords emerging from key strategic documents worldwide were not found in the Horizon 2020 establishment act: recognize the role of women as key actors to ensure peace and security; give women equal rights to ownership; give women equal rights control over land and property; eliminate violence against women and girls; protect women migrants, adolescent girls, maternal mortality, sexual and gender based violence. All of the keywords related to the needs of EU citizens were found in the Horizon 2020 establishment act, 2014-2015 and 2016-2017 work programmes.

Table 23: Degree of matching of H2020 keywords and phrases

H2020 sections

Q1: Is Horizon 2020 in line with the EU and

international priorities?

Q2: Is Horizon 2020 in line with the needs of

the EU citizens?


subsequent technological or

scientific advances?





Identif y

H2020 sections




the EU citizens?






II.1.3. Space High (A; B; C) High (A; B; C) High (A; B; C)


High (A; B; C) High (A; B; C) N.A.

II.3 Innovation in SMEs High (A; B; C) High (A; B; C) High (A; B; C)


High (A)

Medium (B, C) Medium (A; B; C) Medium (A; B; C)




High (A; B; C) Medium (A; B; C) High (A; B; C)




High (A; B; C) High (A; B; C) Medium (A; B; C)




Medium (A; B; C Medium (A; B; C Medium (A; B; C


High (A; B; C) High (A; B; C) N.A.



Identif y

H2020 sections




the EU citizens?





High (A)

Medium (B, C)

High (A)

Medium (B, C) N.A.

VI.1 Cross-cutting Issues. 1.International cooperation

High (A; C)

Medium (B)

Medium (A)

High (B; C) N.A.

VI.2 Cross-cutting Issues. Social Sciences and Humanities (SSH)

Medium (A; B; C) High (A; B; C) N.A.

VI.3. Cross-cutting Issues. Gender

Medium (A)

High (B; C) High (A; B; C) N.A.

A- Horizon 2020 establishment act – 2013

B – Horizon 2020 Work Programme 2014-15

C - Horizon 2020 Work Programme 2016-17

Notes: (i) The approach was not applicable to the following sections of Horizon 2020: I.1. European Research Council I.2. Future and Emerging Technologies; I.3. Marie Skłodowska-Curie Actions; I.4. Research Infrastructures, including e-Infrastructures; and for Q3. for several other sections; (ii) The comparison was conducted according Horizon 2020 sections, indicating the following levels of relevance related to the degrees of matching of Horizon 2020 and work programmes with the extracted keywords and phrases: none, very low (matching up to 25% of the keywords), low (from 25% up to 50%), medium (from 50% up to 75%), and high (from 75% to 100%); (iii) N.A. – not applicable



Identif y

CHAPTER 6: Conclusions

European Union framework programmes for research and innovation aim to support development of the society and world-leading economy based on knowledge and innovation. At the same time, their goal is to contribute to tackling societal challenges and among them to promote growth and jobs in the EU. Since the beginning of the framework programmes, the goal was to achieve the scientific and technical objectives to be pursued at Community level in line with the priorities for the society. In other words, framework programmes for research and innovation goal is to be relevant and of utility to the society.

The EU framework programmes are set in different point in time and are conducted during a planned period. Horizon 2020 was first planned in 2011, and was launched in 2013. It is planned to last until 2020. This report is written in 2016, and already a number of changes have occurred in Europe and the world having strong impact to the needs of society. The strongest examples of these are migrant movement of refugees coming from different parts of the world and security threats that European societies face nowadays. There is a time discrepancy between the set goals of the framework programme that tend to be fixed on a particular time and challenges that our society is confronting. This is due to the goals of the framework programmes were set at the beginning of the programme, while the challenges of society are changing over time. Therefore, the relevance analysis is significant to both to the intervention and to the society, since the intervention should tackle the needs or problems relevant to the society in the current period of time. In this sense, the relevance assessment task cannot be an analysis once conducted and for all, but an ongoing endeavour constituting an important inherent part of the structure of H2020 or future framework programmes. A monitoring system of the intervention should also go along with opening up new venues to identify newly emerged political priorities, needs of society for EU citizens and new technological and scientific advances, thus to guarantee that the programme effectively respond to them.

In order to support this challenge, an expert group was formed in order to develop new methodologies to assess the relevance of European research and innovation framework programmes, and also to apply them in order to assess the relevance of Horizon 2020. This publication presents the methodologies developed for the assessment of relevance of European research and innovation framework programmes as well as the implementation of these to Horizon 2020. In order to assess the relevance, three basic questions (Q1, Q2 and Q3) were formulated that have been further operationalised in sub-questions. The questions refer to the relation of the framework programme with the EU and international priorities (Q1), with the needs of EU citizens (Q2), and with the subsequent technological or scientific advances. The proposed approach includes the following methodologies for providing answers to these questions: experts’ exploratory approach (from source



Identif y

documents, from online sources and from tacit knowledge); computer content analysis (from source documents); social media content analysis, bibliometric analysis and patent analysis. These methods were developed and tested on the Council Decision 2013/743/EU and on the Work Programmes (2014-2015 and 2016-2017). The development of these methodologies was based on strong and effective teamwork approach.

6.1. Concluding remarks on the relevance-assessing methodologies

The expert group developed and implemented the following methodologies for testing the relevance of Horizon 2020:

a) Expert exploratory approach using human content analysis for the processing of H2020 and WPs (Q1.2, Q2.2, Q3.2)

b) Expert exploratory approach using computer content analysis for the processing of H2020 and WPs

c) Text mining approach for the assessment of relevance of H2020 and WPs (Q1.1)

d) Social media approach for the assessment of relevance of H2020 and WPs (Q1.2, Q2.2, Q3.2)

In the following table, a graphic summary is provided:

Table 27. Methodology usage per question

Q1 Q2 Q3

Step 1. Identification

of the EU and

international priorities


relevance


of societal needs



of subsequent

technological or scientific advances


Expert exploratory

human-content

Text mining approach

(only for the Horizon 2020 establishment

act)

NOT APPLICABLE

NOT APPLICABLE

NOT APPLICABLE

Social media NOT APPLICABLE

NOT APPLICABLE

Bibliometric, patent analysis and scientific achievements

NOT APPLICABLE

NOT APPLICABLE

NOT APPLICABLE

NOT APPLICABLE

NOT APPLICABLE

Applicability of each methodology per question has been further explored, and some concluding remarks can be delivered for each of them. First, experts’ exploratory approach was the most widely used approach in the process, since it allows the usage of



Identif y

the tacit knowledge of experts and its combination with the relevant knowledge sources (e.g. key documents of EU and international bodies; Eurobarometer). The privilege of working with a highly selected group of experts has allowed testing this approach in practice. Given that the European Commission has relatively access to a pool of qualified experts, this approach could be considered for future assessments as it showed highest applicability throughout the analysis of keywords in all basic questions (Q1, Q2 and Q3), whereby other methods had applicability limitations. On the other side, its disadvantages have also become evident, in most of the cases referring to the subjective nature of the human judgment as well as the financial cost involved. Thus, in terms of applicability it is necessary to differentiate between the approach using human content analysis and the one using computer based analysis. Regarding the Expert exploratory approach using human content analysis for the processing of H2020 and WPs (Q1.2, Q2.2, Q3.2), it has proven to be applicable in most of the sections of H2020. In this way, the most useful method is expert exploratory approach (for Q1, Q2 and Q3), while text mining is the second most useful (for Q1), followed by online content analysis (for Q2 and Q3). Patent was the least applicable method. The applicability of the Expert exploratory approach using computer content analysis for the processing of H2020 and WPs was tested on the section III.4. Smart, Green and Integrated Transport. Keywords, phrases, and topics extraction can work as a valuable technique to provide synthetic and concise information for evaluating the development of the WPs across a framework period. Keywords and phrases look to be more meaningful than topics, so it is suggested to limit the analysis only to these two categories. However, this opinion is valid solely within the boundaries of the present analysis, and cannot be generalised to the full H2020 framework. The method has several limitations. The most important limitation is that the topic extraction from the legal documents missed several important topics, which was not the case with the exploratory approach using human content analysis. At this stage, the exploratory approach using computer content analysis should be refined in the future, and its effectiveness heavily depends on the quality of the algorithm used for the topic extraction.

Second, text mining approach has demonstrated the ability to process in a faster way large amount of texts. In current times, it is necessary to consider new venues that allow exploring large sets of data in order to grasp broader strokes about emerging trends and existing gaps. These methods shall be used in combination with expert interpretation that allows for a major refinement of the procedure as well as the findings obtained. For the present endeavour, the applicability of the Text mining approach for the assessment of relevance of H2020 and WPs was tested on the Q1.2 level, using the supervised and unsupervised approach, based on the extracted keywords, phrases and topics of the: (1) key documents produced by the key EU and international policy bodies and (2) H2020 Decision document. The results revealed that the approach is useful at the Q1.2 level, both for the supervised and unsupervised approach, with some limitations.

Third, Social media content analysis has revealed the importance of the vivid discussion on the social media as the indicator of the social needs or the values posed into particular subsequent scientific and technological advances. It allows for capturing the



Identif y

voices of lay citizens who might not be represented by the experts or stakeholders usually consulted. However, the changing nature of social media should also be taken into account, since trends are emerging and disappearing in fast manner. Nonetheless, they are still important for understanding the current worries and hopes of citizens that cannot be dismissed in nowadays society. In terms of its applicability, Social media approach for the assessment of relevance of H2020 and WPs has been tested in identifying which of the NEUSCs and STECIAs (operationalized into keywords) obtained through the above mentioned bottom up approach are present in those selected twitter account/project websites related to H2020 (Official or from a funded project).

Finally, Bibliometric, patent analysis and scientific achievements advantages and disadvantages are widely discussed in the literature, and were confirmed in this work. The inputs of not a selected group of experts but the exploration of those provided by the academic community at large are not less important, for what they should be also considered to be further explored. The use of the scientific achievements selections, like MIT Technology Review yearly selection of breakthrough technologies, and Science journal selection of the science breakthrough of the year (2014-2016), were proved to be useful for the generation of identifying the emerging scientific topics. Limitations are found in terms of attribution and the refinement of the results.

Applying relevanc


Identif y

Table 24: Summary on applicability of the expert exploratory approach using human content analysis WPs

H2020 section

Q1. EU and international policy documents keywords

Q2. Needs of society to EU citizens keywords

Q3adv

Expert Text mining

Online (social media)

Patent Expert Text mining


Patent Exp


(Decision + WP level)




(De+ Wleve


(WP level)

(WP

level)

(leve

II.1.3. Space

(Decision + WP level; partially)





(Decision + WP level; partially

II.3 Innovation in SMEs








K (paDecleve





(De+ Wleve

Applying relevanc


Identif y

H2020 section



Q3adv

Expert Text mining




Patent Exp






(De+ Wleve




(De+ Wleve







(De+ Wleve



(WP level; partially)



(De+ Wlevepar

III.7 Secure societies – Protecting freedom and security of Europe and its citizens





(De+ Wleve









Applying relevanc


Identif y

H2020 section



Q3adv

Expert Text mining




Patent Exp

VI.1 Cross-cutting issues. 1.International cooperation





VI.2 Cross-cutting issues. Social sciences and humanities (SSH)




VI.3. Cross-cutting issues. Gender




Total 14 10 0 0 17 6 8 0

Applying relevanc


Identif y

Besides the limitations considered in each methodology, the presented overall work also has other ones that stem from the specific composition of the expert group and from the specific point in time when the methodologies were developed. Often the whole is greater than the sum of its parts. To our belief this is also the case here. The synergy occurred in the expert group was due to the result of the different expertise, background knowledge and approaches. Drawing on this, it is possible to conclude that all methodologies can provide the most valuable results when they are combined together, as they are considered complementary in different manners. First, their combination allow to triangulate the identification of political priorities, needs of citizens and scientific and technological advances as well as the assessment of H2020 relevance integrating different needed perspectives and actors: citizens (social media, Eurobarometer, Google); selected group of experts (members of the present group); and the academic community at large (Google scholar, patent analysis). Traditionally, consultations reach those sectors who usually participate as academics and major stakeholders with the combination of the hereby presented methodologies, what allows to reach out beyond. Second, in most methodologies a combination of bottom-up and top-down approaches coexist, meaning that the search pre-selected keywords (top down) identified by experts or software are complemented by new approaches that are attempting to go beyond this limitation by analysing directly what citizenry express (bottom up). Third, the hereby presented methodologies are also complementary in terms of combining both human directed analyses with innovative software interventions, allowing for searching large sets of data but with the refinement of experts’ views. Finally, while there are methodologies that involve different levels of complexity, others are easily accessible to be conducted involving distinct requirements of resources, an aspect that also worth to be considered.

6.2. Concluding remarks on the application of relevance-assessing methodologies to Horizon 2020 on the both the Decision level and Working programme level (WP2014-2015 and WP2016-2017)

The concluding remarks on the application of relevance-assessing methodologies are presented in relation to the methodologies used, and in relation to each Question. In this sense, although the relevance of some sections of H2020 could not be assessed implementing the Expert exploratory approach using human content analysis for the processing of H2020 and WPs (Q1.2, Q2.2, Q3.2), the method revealed that most of the sections of H2020 are still relevant today. In terms of the conclusions achieved, using the Expert exploratory approach using computer content analysis for the processing of H2020 and WPs was conducted only for the section III.4. Smart, Green and Integrated Transport, which confirmed the positive conclusion of the previous method. The use of Text mining approach for the assessment of relevance of H2020 and WPs, in relation to the Q1.2, leads to conclude that some topics could be more present at the H2020 Decision level. Supervised analysis revealed that these topics are the following: Trade and investment flows, Working conditions, Effects on income, distribution and social inclusion, and Dignity. Topic mapping unsupervised analysis shows that following topics

Applying relevanc


Identif y

could be more present at the Horizon 2020 Decision level: Fight corruption, Response, refugees, Single currency, Violent extremism, and Women. Finally, the conclusions on the relevance of H2020 using the Social media approach for the assessment of relevance of H2020 and WPs allowed identifying NSEUCs that are not present in H2020, e.g.: Stop Evictions, Mediterranean Corridor, and Prisons among others. In the following table, a summary is provided.

Applying relevanc

163 | P a g e EXPERT GROUP ON EVALUAND

Identif y

Table 25: Conclusion on applicability of the expert exploratory approach using human content analysisWPs

H2020 section

Q1. EU and international policy documents keywords Q2. Needs of society to EU citizens keywords Q3. Subseque

a

Expert Text mining


Patent Expert Text mining Online (social media)

Patent Expert


Keywords are adequate and approach is valid

No keywords relevant to ICT generated by the method

N.A. N.A.




N.A.



Keywords are adequate and approach is partially valid (on the WPs level), with some disadvantages

No keywords relevant to NANO generated by the method

N.A. N.A.


Only few keyword relevant to NANO is extracted, mainly in the field of the Biotechnology


N.A.


II.1.3. Space

Keywords are adequate not adequate; approach is not valid

No keywords relevant to SPACE generated by the method

N.A. N.A.

Keywords are only partially adequate; approach is partially valid

Number of keywords generated is too small and generic


N.A.

Keywords are adequate not adequate; approach is not valid


Keywords are adequate and approach is

Keywords are adequate and approac

N.A. N.A.

Keywords are adequate and approach is

Keywords are only partially adequate; approach is

Number of keywords generate

N.A.

No keywords relevant to the section were

Applying relevanc


Identif y

H2020 section


a

Expert Text mining



Patent Expert

valid. h is valid.

valid. partially valid.

d is too small.

generated by the method.

II.3 Innovation in SMEs

Keywords are adequate and approach is valid.

Number of keywords generated is too small.

N.A. N.A.




N.A.

No keywords relevant to the section were generated by the method.


Keywords are partially adequate and approach is partially valid at the Decision level, but is not useful at WPs level.

Keywords are partially adequate and approach is valid.

N.A.

N.A.


Keywords are not relevant, approach is not valid.

Keywords are partially adequate and approach is partially valid.

N.A.

Keywords are partially adequate and approach is partially valid at the Decision level, but is not useful at WPs level.

III.2 Food Security, Sustainable Agriculture and

Keywords are not adequate and

Keywords are not adequate and

Keywords are not adequat

Keywords are not adequat

Keywords are adequate and


Keywords are partially adequat

N.A.


Applying relevanc


Identif y

H2020 section


a

Expert Text mining



Patent Expert

Forestry, Marine, Maritime and Inland Water Research and the Bioeconomy

approach is not valid.

approach is not valid.

e and approach is not valid.

e and approach is not valid.

approach is valid.

approach is valid.

e and approach is partially valid.

approach is valid.



Keywords are partially adequate and approach is valid for H2020 and WP2016-2017, and not for WP2014-2015.

N.A.

N.A.


Keywords are partially adequate and approach is valid for H2020 and WP2016-2017, and not for WP2014-2015.


N.A.



Keywords are not adequate and approach is not valid.

Keywords are partially adequate and approach is partially valid

N.A.

N.A.


N.A.


N.A.





N.A.

N.A.



Keywords are partially adequate and approach is partially

N.A.


Applying relevanc


Identif y

H2020 section


a

Expert Text mining



Patent Expert

valid.



Keywords are partially adequate and approach is partially valid, at the WPs level.

N.A.

N.A.


Number of adequate keywords is too small.


N.A.


III.7 Secure societies – Protecting freedom and security of Europe and its citizens



N.A.

N.A.




N.A.




N.A.

N.A.

N.A.


N.A.

Keywords are partially adequate and approach is

N.A.

N.A.

Applying relevanc


Identif y

H2020 section


a

Expert Text mining



Patent Expert

partially valid



N.A.

N.A.

N.A.


N.A.


N.A.

N.A.

VI.1 Cross-cutting issues. 1.International cooperation



N.A.

N.A.


N.A.


N.A. N.A.

VI.2 Cross-cutting issues. Social sciences and humanities (SSH)


Keywords are adequate and approach is valid at the H2020 and WP2016-2017 level.

N.A.

N.A.


N.A.


N.A. N.A.

VI.3. Cross-cutting issues. Gender

Keywords are adequate

Keywords are adequat

N.A.

N.A.

Keywords are adequate

N.A.

Number of adequat

N.A. N.A.

Applying relevanc


Identif y

H2020 section


a

Expert Text mining



Patent Expert

and approach is valid.

e and approach is valid.

and approach is valid.

e keywords is too small.

Note: N.A. – Not applicable method for the section or cross-cutting issue of H2020.



Identif y

A summary of the main results of the application of the relevance-assessing methodologies is provided here per question. In relation to the Q1 “Is Horizon 2020 in line with the EU and international priorities?”, the application of different methodologies for analysing the relevance of Horizon 2020 compared to the EU and international priorities revealed a high degree of matching at all the three levels of assessment (Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes) for the all of the sections of H2020, except for the III.1 Health, Demographic Change and Wellbeing, III.7 Secure societies and VI.1 Cross-cutting Issues: International Cooperation (with the high level of coverage for the Horizon 2020 establishment act – 2013, and medium level of coverage for the work packages), and VI.3. Cross-cutting Issues: Gender (the degree of matching was from a medium to a high one -with the medium level of coverage for the Horizon 2020 establishment act – 2013, and high level of coverage for the work packages). A graphical representation is provided in the following Figure:

Figure 33 – The relevance of specific H2020 objectives to address European objectives

(4 – High, 3 – Medium, 2 – Low, 1 – Very Low)

Source: 'Applying relevance-assessing methodologies to Horizon 2020' report Graph: Commission compilation based on degree of matching provided by Expert Group

In relation to Q2 “Is Horizon 2020 in line with the needs of the EU citizens?”, the application of different methodologies for analysing the relevance of Horizon 2020 compared to the needs of the EU citizens showed a high degree of matching at all the three levels of assessment (Horizon 2020 establishment act – 2013, the 2014-2015 and



Identif y

the 2016-2017 work programmes) in all section of the LEIT. Throughout the Societal Challenges, the application of different methodologies for analysing the relevance of Horizon 2020 compared to the needs of the EU citizens showed a high degree of matching at all the three levels of assessment in the case of SC2, SC4, SC5, and SC6), and a medium match for the SC1, SC3, and SC7. The section of H2020 concerning science with and for society and the section on spreading excellence and widening participation showed a high degree of match at all the three levels of assessment. In the cross-cutting issues, a high degree of match at all the three levels of assessment was found in the social sciences and humanities, and gender, while a medium and high degree of match was found in the international cooperation. In the Figure below, a graphical representation is provided:

Figure 34 – The relevance of specific H2020 objectives to address the needs of EU citizens



Different methodologies for analysing the relevance of Horizon 2020 compared to the needs of the EU citizens were not applicable to several sections: II.2. Access to Risk Finance, IV. Spreading Excellence and Widening Participation, V. Science with and for Society, VI.1 Cross-cutting Issues. 1.International Cooperation, VI.2 Cross-cutting Issues. Social Sciences and Humanities (SSH).

Finally, in order to respond to Q3 “How well adapted is Horizon 2020 to the subsequent technological or scientific advances?,” it is necessary to highlight that the methodology used for the extraction of the keywords indicating the subsequent technological or



Identif y

scientific advances did not generate the related keywords. For example, the keywords identified as the indicators of the subsequent technological or scientific advances were not related to social science and humanities. The following sections of the LEITS showed a high degree of matching at all the three levels of assessment (Horizon 2020 establishment act – 2013 and the 2014-2015 and the 2016-2017 work programmes): II.1.1. Information and Communication Technologies (ICT); II.1.2. Nanotechnologies, Advanced Materials, Advanced Manufacturing and Processing, and Biotechnology (NANO); II.3. Innovation is SMEs, and II.1.3. Space. Throughout the Societal Challenges, the application of different methodologies for analysing the relevance of Horizon 2020 compared to the needs of the EU citizens showed a high degree of matching at all the three levels of assessment in the case of SC2, SC3, SC4, and SC6, while at SC1 and SC7 a medium degree of matching was identified. In the following figure a visual summary is provided:

Figure 35 – The relevance of specific H2020 objectives to address subsequent technological developments





Identif y

References

Bakkalbasi, N. a. (2006). Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomedical digital libraries, 3(1), 7.

Carley, K. (1993). Coding choices for textual analysis: A comparison of content analysis and map analysis. Sociological methodology, 75-126.

Council Decision 2013/743/EU. (OJ L 347 of the 20.12.2013). Council Decision of 3 December 2013 establishing the specific programme implementing Horizon 2020 - the Framework Programme for Research and Innovation (2014-2020) and repealing Decisions 2006/971/EC, 2006/972/EC, 2006/973/EC, 2006/974/EC and 2006/975/EC.

European Commission. (2015a). Better Regulation Guidelines. SWD(2015) 111 final. Retrieved August 8, 2016, from http://ec.europa.eu/smart-regulation/guidelines/docs/swd_br_guidelines_en.pdf

European Commission. (2015b). Better Regulation "Toolbox". Retrieved August 8, 2016, from http://ec.europa.eu/smart-regulation/guidelines/docs/br_toolbox_en.pdf

EUROSTAT. (2009). Individuals using the internet for posting messages to social media sites or instant messaging. Retrieved from http://ec.europa.eu/eurostat/web/products-datasets/-/isoc_cismt

EUROSTAT. (2015). 2015 Year in Review. Types of social media used. Retrieved from http://ec.europa.eu/eurostat/web/products-datasets/-/tin00084

McCrudden, M., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational psychology review, 19(2), 113-139.

Regulation No 1291/2013. (OJ L 347 of 20.12.2013). Regulation No 1291/2013 of the European Parliament and of the Council of 11 December 2013 establishing Horizon 2020 - the Framework Programme for Research and Innovation (2014-2020) and repealing Decision No 1982/2006/EC.

Reillon, V. (2015). Overview of EU Funds for research and innovation. Retrieved September 9, 2016, from http://www.europarl.europa.eu/RegData/etudes/BRIE/2015/568327/EPRS_BRI(2015)568327_EN.pdf

https://ec.europa.eu/programmes/horizon2020/h2020-sections

http://thismatter.com/economics/technological-advances

http://newsroom.fb.com/news/2015/12/2015-year-in-review/

https://2015.twitter.com/

http://www.trendone.com/en/trend-universe/micro-trends.html

http://ec.europa.eu/rea/breakthroughs/index_en.htm

https://rio.jrc.ec.europa.eu/en/library/measurement-and-analysis-knowledge-and-rd-exploitation-flows-assessed-patent-and-licensing



Identif y

List of Tables

Table 1: Mitigation measures to overcome weaknesses of the exploratory text analysis for the identification of priorities as explicitly and implicitly given in key EU and international policy documents ................................................................ 48 Table 2: Mitigation measures to overcome weaknesses of the computer content analysis for the identification of current priorities as explicitly given in key EU and international policy documents ........................................................................ 54 Table 3: Mitigation measures to overcome weaknesses of the experts’ exploratory methodology for the identification of the needs of society to EU citizens ...................................................................................................................................... 60 Table 4: Mitigation measures to overcome weaknesses of the experts’ exploratory methodology for analysis of Eurobarometer and EC public consultation reports for the identification of the needs of society to EU citizens ............... 67 Table 5: Structure adopted for the “Keywords Expert Exploratory Approach” (as per H2020) .......................................... 69 Table 6: Mitigation measures to overcome weaknesses of the experts’ keywords selection for the identification of the needs of society to EU citizens ............................................................................................................................................ 74 Table 7: Mitigation measures to overcome weaknesses of the Computer content analysis for the identification of the needs of society to EU citizens from Eurobarometer surveys ............................................................................................. 78 Table 8: Mitigation measures to overcome weaknesses of the social media analysis for the identification of the needs of society to EU citizens ........................................................................................................................................................... 87 Table 9: The four processes of technological advances ...................................................................................................... 90 Table 10: Definitions of ‘scientific advance’ and ‘technological advance’ .......................................................................... 90 Table 11: Examples of sources of information utilised for the experts’ exploratory approach for the identification of the subsequent technological and scientific advances ............................................................................................................. 91 Table 12: Mitigation measures to overcome weaknesses of the experts’ exploratory approach for the identification of subsequent technological and scientific advances ............................................................................................................. 95 Table 13: Mitigation measures to overcome weaknesses of the bibliometric analysis for the identification of subsequent technological and scientific advances ................................................................................................................................. 99 Table 14: Mitigation measures to overcome weaknesses of the social media analysis for the identification of subsequent technological and scientific advances ............................................................................................................................... 108 Table 15: Mitigation measures to overcome weaknesses of the patent analysis for the identification of subsequent technological or scientific advances ................................................................................................................................. 114 Table 16: Sections of H2020 considered for the allocation of identified keywords .......................................................... 117 Table 17: Mitigation measures to overcome weaknesses of the Expert exploratory approach using human content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2) ...................................................................... 121 Table 18: Mitigation measures to overcome weaknesses of the experts’ exploratory approach using computer content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2) ...................................................................... 125 Table 19: Mitigation measures to overcome weaknesses of the text mining approach for the assessment of relevance of H2020 to the identified EU and international priorities (Q1.2) ......................................................................................... 129 Table 20: List of the selected sources ................................................................................................................................ 131 Table 21: Sections of H2020 considered for the allocation of identified keywords .......................................................... 131 Table 22: Mitigation measures to overcome weaknesses of the Social media analysis for the assessment of relevance of H2020 and WPs (Q1.2, Q2.2 and Q3.2) ............................................................................................................................ 134 Table 23: Degree of matching of H2020 keywords and phrases ...................................................................................... 151 Table 24: Summary on applicability of the expert exploratory approach using human content analysis for the processing of H2020 and WPs ............................................................................................................................................................ 158



Identif y

List of Figures

Figure 1: Conceptual representation of evaluation criteria and the intervention logic by European Commission (European Commission, 2015b)............................................................................................................................................................ 36 Figure 2: Overview of the five evaluation criteria related to the evaluation process of European Commission (European Commission, 2015a)............................................................................................................................................................ 37 Figure 3: The continuity of the eight EU framework programmes for the research and innovation .................................. 38 Figure 4: Relation of challenges in society and goals of the research and innovation framework programmes (hypothetical example) ....................................................................................................................................................... 39 Figure 5: The three basic questions and sub-questions which determine the framework for the assessment of relevance of European research and innovation framework programmes ......................................................................................... 40 Figure 6: Layout of the template used for the extraction and representation of concepts reflecting EU and international priorities from strategic policy documents in the context of STEP 4 of the exploratory text analysis conducted by experts ............................................................................................................................................................................................ 45 Figure 7: Flowchart of the exploratory text analysis for the identification of current priorities as explicitly and implicitly given in key EU and international policy documents .......................................................................................................... 47 Figure 8: Flowchart of the computer content analysis for the identification of current priorities as explicitly given in key EU and international policy documents .............................................................................................................................. 53 Figure 9: Layout of the template used for the identification of the needs of society to EU citizens (STEP 1) in the context of experts’ exploratory methodology .................................................................................................................................. 57 Figure 10: Layout of the template used for the identification of the main category within the multi-dimensionality aspect of the identified needs of society to EU citizens (STEP 3) in the context of the experts’ exploratory methodology ........... 58 Figure 11: Flowchart of the experts’ exploratory methodology for the identification of the needs of society to EU citizens ............................................................................................................................................................................................ 59 Figure 12: Layout of the template used for the validation of pre-identified topics related to potential emerging challenges for society (STEP 2) for the implementation of the experts’ exploratory approach from Eurobarometer for the identification of the needs of society to EU citizens ............................................................................................................ 64 Figure 13: Flowchart of the experts’ exploratory methodology for analysis of Eurobarometer and EC public consultation reports for the identification of the needs of society to EU citizens.................................................................................... 66 Figure 14: Layout of the excel template used for collecting experts’ input (keywords) (STEP 1) for the implementation of the experts’ selection of keywords for the identification of the needs of society to EU citizens ......................................... 71 Figure 15: Flowchart of the experts’ keywords selection for the identification of the needs of society to EU citizens ....... 72 Figure 16: Flowchart of the computer content analysis for the identification of the needs of society to EU citizens from Eurobarometer surveys ....................................................................................................................................................... 77 Figure 17: Flowchart of the top-down approach for social media analysis for the identification of the needs of society to EU citizens ........................................................................................................................................................................... 84 Figure 18: Layout of the template used for Identifying NSEUCs from Twitter Trending Topics of 15 European Countries (STEP 2) of the bottom-up approach of social media analysis ............................................................................................ 85 Figure 19: Flowchart of the bottom-up approach for social media analysis for the identification of the needs of society to EU citizens ........................................................................................................................................................................... 86 Figure 20: Layout of the excel template used for collecting experts’ input (keywords) (STEP 3) for the implementation of the Experts’ exploratory approach for the identification of subsequent technological and scientific advances ................ 92 Figure 21: Flowchart of the experts’ exploratory approach for the identification of subsequent technological and scientific advances .............................................................................................................................................................. 93 Figure 22: Steps of the Google Scholar Exploratory Approach Methodology on identifying subsequent technological and scientific advances .............................................................................................................................................................. 98 Figure 23: Flowchart of the top-down approach for social media analysis for the identification of subsequent technological and scientific advances ............................................................................................................................... 105 Figure 24: Layout of the template used for identifying technological and scientific advances from Twitter Trending Topics of 15 European Countries (Step 2) of the bottom-up approach of social media analysis ................................................. 106 Figure 25: Flowchart of the top-down approach for social media analysis for the identification of subsequent technological and scientific advances ............................................................................................................................... 107 Figure 26: Table for the classification of keywords per used methodology and question ............................................... 117 Figure 27: Layout of the template used for the allocation of keywords per: question and methodology and section of H2020 as needed for the implementation of the experts’ exploratory approach using human content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2) ............................................................................................... 118



Identif y

Figure 28: Layout of the template used for the tracking of the keywords, phrases and topics found for the tracking of the keywords, phrases and concepts (found for the purpose of Q1.1, Q2.1 and Q3.1) in the Council Decision 2013/743/EU, the 2014-2015 Work Programme and the 2016-2017 Work Programme ........................................................................ 119 Figure 29: Flowchart of the experts’ exploratory approach using human content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2) ....................................................................................................................................... 120 Figure 30: Flowchart of the experts’ exploratory approach using computer content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2) ....................................................................................................................... 124 Figure 31: Flowchart of the experts’ exploratory approach using the text mining approach for the assessment of relevance of H2020 to the identified EU and international priorities (Q1.2) .................................................................... 128 Figure 32: Example of template for Twitter accounts analysis selected ........................................................................... 133



Identif y

Appendix 1. Summary of the overall methodological approach

The methodology for the assessment of relevance is defined by the expert group as the process to determine whether the original objectives of the framework programme are still relevant and to define the extent they still match the present needs and challenge.

New innovative methodologies were developed by the expert group for assessing the relevance of the European research and innovation framework programmes.

To this end, the assessment of relevance of framework programmes has been structured upon three basic i.e.:

Q1. Is the framework programme in line with the EU and international priorities?

o Q1.1 How to identify the EU and international priorities? o Q1.2 How to asses if the Horizon 2020 is relevant to the identified EU and international

priorities?

Q2. Is the programme in line with the needs of society to EU citizens?

o Q2.1 How to identify the needs of society to EU citizens? o Q2.2 How to asses if the Horizon 2020 is relevant to the identified needs of society to EU

citizens?

Q3. How well adapted is the intervention to the subsequent technological or scientific advances?

o Q3.1 How to identify the subsequent technological or scientific advances? o Q3.2 How to asses if the Horizon 2020 is relevant to the identified subsequent

technological or scientific advances?

The following tables present the methodologies that were developed for providing answers to the above questions.

Table 1-I: Summary of the methodologies developed for answering sub-question Q1.1: How to identify the EU and international priorities?

Methodology Knowledge source Process Outcome Software

usage

Q1.1 How to identify the EU and international priorities?

Exploratory text analysis conducted by experts

Strategic policy documents published by EU and international institutions

Human content analysis of the key EU and international priorities from the strategic policy documents

Keywords and phrases (concepts) indicating the key EU & international priorities

None

Computer content analysis for the identification of key priorities

Strategic policy documents published by EU and international institutions

Software content analysis of the key EU and international priorities from the strategic policy documents

Keywords and phrases indicating the key EU and international priorities

Provalis Wordstat (could be done with other text mining software)



Identif y

Table 1-II: Summary of the methodologies used for answering sub-question Q2.1: How to identify the needs of society to EU citizens?

Methodology Knowledge source Process Outcome Software usage

Q2.1 How to identify the needs of society to EU citizens?

Experts’ exploratory methodology for the identification of the needs of society to EU citizens

The most important (from online sources) channels that citizens make to use their voice heard in EU.

Human content analysis

Narratives and keywords indicating the needs of society to EU citizens

none

Experts’ exploratory approach for the identification of the needs of society to EU citizens from Eurobarometer surveys and EC public consultations reports

Eurobarometer on-line surveys and EC consultation reports


Identified new (emerging) challenges for EU society – in the form of keywords and phrases

none

Experts’ selection of keywords for the identification of the needs of society to EU citizens

Tactic knowledge: experts’ understanding of (1) strategic policy documents, (2) Decision 2013/743/EU (3) WPs


Keywords and phrases indicating the needs of society to EU citizens

none

Computer content analysis for the identification of the needs of society to EU citizens from Eurobarometer reports

Eurobarometer reports

Software content analysis of the

social needs to EU citizens from

Eurobarometer reports

Keywords and phrases indicating the needs of society to EU citizens

Provalis Wordstat

(also by other text mining software)

Social media analysis for the identification of the needs of society to EU citizens

Top-Down Approach

Social media (Twitter and Facebook) and other relevant online sources (Wikipedia and YouTube)

Software data analysis of social

media (Twitter and Facebook) and

others for analysis of online sources from Wikipedia and YouTube

Identified needs of society to EU citizens from the Social media (Twitter, Facebook) and other online sources (Wikipedia and YouTube)

Facebook & Twitter: the R

Project for Statistical

Computing; Wikipedia: the

statistics Wikipedia

Page; YouTube: YouTube’s

search engine

Social media analysis for the identification of the needs of society to EU citizens

Bottom-up Approach

social media (Twitter and Facebook) and another relevant online source (Wikipedia)

Analysis of social media secondary

literature, analysis of trending topics

of Twitter, and most viewed

Wikipedia pages

Identified needs of society to EU citizens from the Social media Secondary Literature, Twitter and Wikipedia

the R Project for Statistical Computing;

Most viewed Wikipedia

Pages search



Identif y

Table 1-III: Summary of the methodologies used for answering sub-question Q3.1: How to identify the subsequent technological or scientific advances?

Methodology Knowledge source Process Outcome Software

usage

Q3.1 How to identify the subsequent technological or scientific advances?

Experts’ exploratory approach using human content analysis for the assessment of relevance of H2020

Experts identified the sources (from online sources) of information on technological and scientific advances


Experts’ opinion on the latest technological and scientific advances – in the form of keywords and phrases

none

Bibliometric methodology for the identification of subsequent technological and scientific advantages

Academic literature: Science journal (Science's Breakthrough of the Year) and MIT Technology Review (Breakthrough Technologies)

Software content analysis of the academic publications

Tracking of the dissemination and exploitation of new ideas in the academic publications, that are in direct relation with the Horizon 2020

Google Scholar & Science and MIT Technology Review websites

Social media analysis for the identification of subsequent technological and scientific advantages

Top-Down Approach

Social media (Twitter and Facebook) and other relevant online sources (Wikipedia and YouTube)

Software data analysis of social media (Twitter and Facebook) and others for analysis of online sources from Wikipedia and YouTube

Identified needs of society to EU citizens from the Social Media (Twitter, Facebook) and other online sources (Wikipedia and YouTube)

Facebook & Twitter: the R Project for Statistical Computing; Wikipedia: the statistics Wikipedia Page; YouTube: YouTube’s search engine

Social media analysis for the identification of the subsequent technological and scientific advantages

Bottom-up Approach

Social media (Twitter and Facebook) and another relevant online source (Wikipedia)

Analysis of social media secondary literature, analysis of trending topics of Twitter, and most viewed Wikipedia pages

Identified needs of society to EU citizens from the Social media Secondary Literature, Twitter and Wikipedia

the R Project for Statistical Computing;

Most viewed Wikipedia Pages search

Patent analysis for the identification of subsequent technological and scientific advantages

Patent specifications

Analysis of key patenting trends and analysis of inventor involvement in FP projects; usage of secondary data sources

Only a first crude analysis was performed that yielded y list of key words resembling key patenting trends based on technology classes with high patent activity

So, far none; later: PATSTAT and R/STATA



Identif y

Table 1-IV: Summary of the methodologies used for answering sub-question Q1.2: How to asses if the

Horizon 2020 is relevant to the identified EU and international priorities?; Q2.2: How to asses if the Horizon 2020 is relevant to the identified needs of society to EU citizens?; and Q3.2: How to asses if the

Horizon 2020 is relevant to the identified subsequent technological or scientific advances?

Methodological Knowledge source Process Outcome Software

usage Note

Experts’ exploratory approach using human content analysis for the assessment of relevance of H2020

Council Decision 2013/743/EU, the 2014-2015 Work Programme and the 2016-2017 Work Programme


Matching of keywords identified through the exploratory methods (Q1.1, Q2.1, and Q1.2) in the text of the legal documents

None

This approach has been applied for Q1.2, Q2.2, and Q3.2

Experts’ exploratory approach using computer content analysis for the assessment of relevance of H2020 (Q1.2, Q2.2 and Q3.2)

Part 11 of the 2014 -2015 WP & the 2016–2017 WP ‘Smart, green and integrated transport’ (Societal Challenges)

Software content analysis (keywords, phrases and topics extraction)

Keywords and phrases indicating the main themes in the part 11 of the 2014 -2015 & the 2016–2017 work programmes ‘Smart, green and integrated transport’ (Societal Challenges)

Provalis Wordstat (also by other text mining software)

This approach has been applied for part 11 of the 2014 -2015 WP & the 2016–2017 WP ‘Smart, green and integrated transport’ (Societal Challenges)

Text mining approach for the assessment of relevance of H2020 to the identified EU and international priorities

Key policy documents of EU and international bodies and the Council Decision 2013/743/EU

Software content analysis (keywords, phrases and topics extraction)

Matching of keywords, phrases and topics in key policy documents and Council Decision 2013/743/EU

Provalis Wordstat (also by other text mining software)

This approach has been tested for the Q1.2 on the Decision level

Social media analysis for the assessment of relevance of H2020 (Q1.2, Q2.2, Q3.2)

Selection of Twitter accounts and project’s websites related to H2020

Combination data analysis social media software and human content analysis

Observations whether the contributions extracted from citizens’ voices were or not present into H2020

the R Project for Statistical Computing for the analysis of #H2020 hashtag

This approach has been applied for Q2.2 and Q3.2



Identif y

Appendix 2. Summary of used methodologies and assessment of their advantages and disadvantages

The proposed approach includes the following methodologies for providing answers to questions Q1.1, Q2.1 and Q3.1:

1. Experts’ exploratory methodology: from source documents – key policy strategic documents of prominent EU and

international organisations and institutions; Eurobarometer documents – for the purpose of Q1.1 and Q2.1;

from online sources – for the purpose of Q2.1 and Q3.1; from tacit knowledge – for the purpose of Q2.1.

2. Computer content analysis: from source documents - key policy strategic documents of prominent EU and

international organisations and institutions; Eurobarometer documents – for the purpose of Q1.1 and Q2.1.

3. Social media analysis: from social media – e.g. Facebook, Twitter, and others - for the identification of the

needs to EU citizens (Q2.1) and subsequent technological or scientific advances (Q3.1)

4. Bibliometric analysis – from scholar.google.com and Science journal (Science's Breakthrough of the Year) and MIT Technology Review (Breakthrough Technologies) for the identification of subsequent technological and scientific advantages (Q3.1).

5. Patent analysis - for the identification of subsequent technological & scientific advantages (Q3.1).

The following table presents the summary of the usage of the methods for providing answers to questions Q1.1, Q2.1 and Q3.1.



Identif y

Table 2-I: Summary of used method for providing answers to questions Q1.1, Q2.1 and Q3.1

Methodology Q1.1 Q2.1 Q3.1

Experts’ exploratory methodology – from source documents

(Key documents of EU &

international bodies)

(Eurobarometer

surveys and reports)

Experts’ exploratory methodology – from online sources

(selected online sources)

(selected online

sources)

Experts’ exploratory methodology – from tacit knowledge

(from expert tacit knowledge)

Computer content analysis – from source documents

(Key documents of EU &

international bodies)

(Eurobarometer

surveys and reports)

Social media analysis – from selected social media

(Twitter, Facebook, Wikipedia, Youtube)

(Twitter, Facebook, Wikipedia, Youtube)

Bibliometric analysis

(from

scholar.google.com & Science journal

(Science's Breakthrough of the Year) and MIT Technology Review

(Breakthrough Technologies))

Patent analysis

The proposed approach includes the following methodologies for providing answers to questions Q1.2, Q2.2 and Q3.2:

1. Expert exploratory methodology:

directly from legal documents of H2020 – Council Decision 2013/743/EU, the 2014-2015 Work Programme and the 2016-2017 Work Programme – for the purpose of Q1.2, Q2.2 and Q3.2.

from keywords, phrases and topics extracted by text mining – for the purpose of Q1.2, Q2.2, and Q3.2, only for part 11 of the 2014 -2015 WP and for part 11 of the 2016–2017 WP ‘Smart, green and integrated transport’ (Societal Challenges)

2. Computer content analysis:

using extracted keywords, phrases and topics from source documents -key strategic policy documents of EU and international organisations & Council Decision 2013/743/EU – for the purpose of Q1.2.

3. Social media analysis:

from social media – e.g. Facebook, Twitter, and others - for the assessment of relevance of H2020 – for the purpose of Q2.2 and Q3.2.

The following table presents the summary of the usage of the methods for providing answers to questions Q1.2, Q2.2 and Q3.2.



Identif y

Table 2-II: Summary of used method for providing answers to questions Q1.2, Q2.2 and Q3.2

Methodology Q1.2 Q2.2 Q3.2

Experts’ exploratory methodology – from source documents

(comparing keywords resulting from Q1.1 with

texts of Decision 2013/743/EU and WPs)





Experts’ exploratory methodology – from keywords, phrases and topics extracted by text mining


keywords extracted from Part 11 of WPs)


keywords extracted from Part 11 of WPs)

(comparing keywords resulting from Q3.1 with keywords extracted from

Part 11 of WPs)

Computer content analysis

(automatic matching of the keywords, topics and phrases from key EU and international

policy documents and Decision 2013/743/EU)

Social media analysis – from selected social media

(comparing keywords resulting from Q2.1

(bottom-up approach) with H2020 official

twitter accounts and funded projects

websites)

(comparing keywords resulting from Q3.1

(bottom-up approach) and with H2020 official twitter accounts and

funded projects websites)

The advantages and disadvantages have been identified for the above mentioned methodologies for providing answers to questions Q1.1, Q2.1 and Q3.1, and are provided in the following tables.



Identif y

Table 2-III: Expert exploratory methodology – from source documents – for the purpose of Q2.1

Advantages

Q2.1:

- Analysis of Eurobarometer surveys.

- EC public consultations reports and related advice reports of EC Expert Groups (available on-line) allows to gain the most actual knowledge about current observations, concerns and expectations of EU citizens.

- Eurobarometer is focused specifically on the emerging trends.

- It provides both multidisciplinary expert opinions and the citizens voices which are essential to capture wider societal concerns and emerging trends.

- Eurobarometer surveys, EC public consultations reports and related advice reports of EC Expert Groups are reliable source of information as they are carried out according to the highest level of the methodological principles, including even distribution of EU countries, as well as occupation groups, gender, age, stakeholders’ representatives etc.

- The method is relatively fast and cheap and uses publicly accessible data sources.

Disadvantages

Q2.1:

- Validation of results is necessary (checking if emerging challenges are recognised properly and if selected keywords are adequate to these challenges).

- While the verification of identified topics and keywords is necessary, some experts did not provide their feedback. That brings the risk that obtained results are not fully adequate.

- Single-source analysis does not guarantee complete results.

- On-line content analysis is linked to the risk of temporary 'searching peaks' (or ‘temporary fears’).

Strategies for overcoming disadvantages

Q2.1:

- Results have to be validated by checking if/how often selected keywords appear in other on-line sources in similar context (i.e. if they are related to the same or similar topics).

- To obtain valid and exact results all expertise areas must be covered. All thematic experts must be given clear instructions, including the information that they have to participate in the task and provide their feedback in the timeline.

- Multiple on-line sources must be taken into account and analysed in parallel.

- Samples for on-line content analysis should be extracted at least 3 times and at least three months apart to avoid taking into account temporary 'searching peaks'.

Recommendation Method is useful in the cases when a holistic approach and in depth knowledge of thematic fields is necessary for conducting analysis with understanding of the



Identif y

context



Identif y

Table 2-IV: Expert exploratory methodology – from online sources – for the purpose of Q2.1 and Q3.1

Advantages Q2.1 and Q3.1:

- Allows recognising the context as well as avoiding trivial mistakes.

- Results which can be comparable across different methodologies on the basis of a common structure of H2020.

- A simple excel template can suit for a small group of experts, hence there is no need of specific custom software.

- Guided cooperation and common overview of the results exploits the potential of collective intelligence in large groups of experts.

- The group may include representatives from stakeholders, thus allowing including the position of industrial clusters and/or Research & Technology Organisation organisation.

Disadvantages Q2.1 and Q3.1:

- Involves human actions, which are, by definition, influenced by subjective opinions and human bias.

- It involves human work-force to read and understand reference source documents, which might be costly and time-consuming.

- The use of the excel template for the collection and selection phase is slow and involves manual works; aggregation also is performed manually, thus slowing down the input processing.

- Keywords inputs might be incomplete, since they rely on the expertise of a finite and limited number of experts time.

- The methodology results might depend on the selection of the experts, thus posing questions on the representativeness and repeatability of the results by changing the expert group members.


Q2 and Q3:

- Involve a statistically representative sample of expert, ensuring a complete coverage of the expertise portfolio, as well as a weighted distribution of the know-how in the experts group.

- This task can be automated by means of more refined input interfaces (e.g. a web-interface based on html).

- Can be automatised by means of text processing tools (word comparison & categorisation per synonyms).

Recommendation Method is useful in cases when tacit knowledge of experts is needed in order to produce collective intelligence, with the goal of detecting key topics in a framework programme (e.g. Horizon 2020), for the purpose of the assessment of relevance.



Identif y

Table 2-V: Expert exploratory methodology – from tacit knowledge – for the purpose of Q2.1 and Q3.1

Advantages Q2.1 and Q3.1:

- Allows recognising the context as well as avoiding trivial mistakes.

- Results which can be comparable across different methodologies on the basis of a common structure of H2020.

- Asimple excel template can suit for a small group of experts, hence there is no need of specific custom software.

- Guided cooperation and common overview of the results exploits the potential of collective intelligence in large groups of experts.

- The group may include representatives from stakeholders, thus allowing including the position of industrial clusters and/or Research & Technology Organisation organisation.

Disadvantages Q2.1 and Q3.1:

- Involves human actions, which are, by definition, influenced by subjective opinions and human bias.

- it involves human work-force to read and understand reference source documents, which might be costly and time-consuming.

- The use of the excel template for the collection and selection phase is slow and involves manual works.

- Aggregation also is performed manually, thus slowing down the input processing.

- Keywords inputs might be incomplete, since they rely on the expertise of a finite and limited number of experts time.

- The methodology results might depend on the selection of the experts, thus posing questions on the representativeness and repeatability of the results by changing the expert group members.


Q2 and Q3:

- Involve a statistically representative sample of expert, ensuring a complete coverage of the expertise portfolio, as well as a weighted distribution of the know-how in the experts group.

- This task can be automated by means of more refined input interfaces (e.g. a web-interface based on html).

- Can be automatised by means of text processing tools (word comparison & categorisation per synonyms).

Recommendation Method is useful in cases when tacit knowledge of experts is needed in order to produce collective intelligence, with the goal of detecting key topics in a framework programme (e.g. Horizon 2020), for the purpose of the assessment of relevance.



Identif y

Table 2-VI: Computer content analysis – from source documents – for the purpose of Q1.1 and Q2.1

Advantages - Can be automated.

- Relatively rapidly done.

- Outcome of the software is objective.

- Can handle large amounts of documents.

- Allows quantification.

- Straightforward to replicate.

- Expert knowledge and the automatic procedure are complementary.

- The dictionary created can be created that can be used subsequently.

- Can produce unexpected results that might be easily excluded by human expert (i.e. not necessarily obvious).

Disadvantages - Interpretation by the expert could be subjective.

- Context is not recognised automatically.

- Expert interpretation is needed for the results.

- If dictionary is not adequate there are problems with synonyms and words used in a non-significant manner.

- Wrong selection of keywords would lead to wrong results.


- The text mining method can be performed with the several human tasks used in order to assure the validity of the results: (1) human text mining expert who prepares the documents; (2) human text mining expert who selects the algorithms and other software features (e.g. minimum number of words in phrases); (3) human context expert who selects the documents to be extracted; (4) human context expert who has the context knowledge for the interpretation of the results.

- The objectivity of the method can be increased if at least two control groups of experts participate in the described tasks, especially in the task related to the interpretation of the results.

- Optimise the size and time-span when using the experts’ exploratory approach method, to suit the intended task (question investigation).

Recommendation Method is useful in cases when it is necessary to use computer aided text mining, as a quick tool which produces results, with the goal of detecting: a) key priorities of society from key EU and international policy documents, and b) key needs of society to EU citizens and from the Eurobarometer selected surveys.



Identif y

Table 2-VII: Social media analysis – from selected social media – for the purpose of Q2.1 and Q3.1

Advantages - It allows both top-down and bottom up approach

- Allows to conduct a direct analysis of the expressions made by citizens (primary data) through the bottom up approach

- Identify new NSEUCs (e.g. stop evictions, same-sex marriage) that have not been identified through other methodologies (complementarily to others) (from bottom up approach)

- Allows to assess the relevance of each of the predefined needs according to what citizens express in social media and other online sources (top down approach)

- Easy, fast and free to use by non-experts

- Different languages can be taken into account

- Using different social media allows to compare and to embrace greater diversity of publics

Disadvantages - Social media is just the reflection of the opinions of the sub-group of citizens that participates in the social media, not covering the whole population

- The usage of synonyms that are hard to recognize without the context

- Uneven coverage of countries, age and education

- Reliability of content; Limitation of historical data from Twitter and Facebook due to API Limitations of both SM.

- Manual annotation of YouTube searches information

- Barriers in identifying STECIAs as expressed by citizens from the bottom up approach


- Although data is limited to social media users, it contains more representative citizens’ voices than other sources.

- Two ways for mitigating the weakness in the usage of synonyms that are hard to recognize: a) To use R-programme to extract associated words, b) To qualitatively analyse conversation on Facebook page (through comments) or to explore interactions under particular hashtags in order to understand the meaning in context.

- Related to the countries this information is available on Facebook Data Collection Top down approach, with the limitation of only those geocalizated pages could be identified. Also countries information is available in the list of 15 European Countries Trending Topics.

- Content reliability is ensured with a specific design of validity checks conducted by external researchers.

- The R-programme allows to define a Monitoring process accounting for collecting data throughout time.

- Technical limitation of YouTube data collection could be overcome with last advances of R-program.

- Exploring new forms of obtaining YouTube data collection in an automatic manner.

- In order to mitigate the limitation of identifying STECIAs from bottom up approach, it would be useful to conduct a monitoring process that allows collecting more data in different occasions during a particular period of time in order to be able to contrast it chronologically.

-

Recommendation Method is useful in cases when using social media analysis is suitable methodology for including the voice of citizens on the analysis of needs of society to EU citizens from top down and bottom up approach. For the case of STECIAs, the analysis needs to include gathering information from a particular



Identif y

period of time.

Table 2-VIII: Bibliometric analysis – for the purpose of Q3.1

Advantages - The Google Scholar analysis may enable to discover the wider societal perspective that is available online.

- It provides quantitative results expressed as the number of hits of certain key words or topics.

- Flexible and straightforward.

- The usage of the relevant lists of yearly scientific and technological achivements (Science journal (Science's Breakthrough of the Year) and MIT Technology Review (Breakthrough Technologies)) is a good benchmark of the globally recognized important discoveris that reflect the trend in the area of subsequent technological and scientific advances

Disadvantages - Limited to the public online sources and to online users.

- The online search engines use different algorithms and they may potentially bias the results.

- Temporary trends determined by media interest or paid content may also bias the outputs and they should be considered.


- Combining several bibliometric tools may be an important validation technique.

- The selection of the initial key words and phrases should be carefully discussed, and only fully justified syntaxes will be considered, as they actually determine the outputs.

Recommendation The method is useful in cases when tracking is needed of the dissemination and exploitation of new ideas in the academic publications that are in direct relation with a framework programme (e.g. Horizon 2020).

The usage of the Science journal (Science's Breakthrough of the Year) and MIT Technology Review (Breakthrough Technologies) was useful for tracking the most recent subsequent technological and scientific advances, that also attract public attention.



Identif y

Table 2-IX: Patent analysis – for the purpose of Q3.1

Advantages - Patent analysis may discover certain technological breakthroughs where there are high hopes on commercialisation via patenting

- The results are quantitative

Disadvantages - Relevance is assessed only indirectly, as relevance of FP for patenting/in patenting fields

- The disadvantages of patents as proxies for innovation apply: e.g., the fact that many technologies and innovations are not patentable and/or are not patented; that patent value is highly skewed (i.e., most patents are not worth a lot); etc.

- There are many ways to conduct a patent analysis, i.e. there is a body of methods that can be applied to patent data. In this assignment, only a selection can be tested.

- Properly executing a patent analysis is time-consuming and needs to be given a lot of thoughts; moreover, it is difficult to perform a patent analysis only bottom-up on the whole set of patents without identifying an interesting sub-set of patents (e.g., via key word searches, etc.) first


- The patent analysis executed so far was only very simplistic and a first run for discussion

- Improving results may be possible if technology field specificity is stringer accounted for and/or the analysis is focused on specific sub-sets of patents such as highly-cited patents and/or patent families instead of single patents. This will be tested in the next steps.

Recommendation - The method may be useful when it is attempted to analyse whether H2020 and FP research is relevant for patenting and patenting-related commercial aims.

The advantages and disadvantages have been identified for the above mentioned methodologies for providing answers to questions Q1.2, Q2.2 and Q3.2, and are provided in the following tables.



Identif y

Table 2-X: Expert exploratory methodology – from source documents – for the purpose of Q1.2, Q2.2 and

Q3.2

Advantages Q1.2, Q2.2. and Q3.2.

- Human content analysis, allowing for understanding of the context from which presence of keywords, phrases and topics is being identified, especially when it comes to longer phrases and their implicit presence in the legal texts (Council Decision 2013/743/EU and WPs).

- The outcome is based on well allocated experts’ thematic expertise to the related sections of the Horizon 2020 and its consequent Work Programmes.

- The objectivity of the outcome is checked through the ‘Leader’ and ‘Buddy’ cooperation.

- The methodology allows combining use of several methods.

- During the implementation of this methodology there is no need to use a special software to process the information input.

- This methodology is relatively inexpensive and can achieve quick data collection technique.

Disadvantages Q1.2, Q2.2. and Q3.2.

- Since the methodology is based on human content analysis, the output may be biased by subjectivity.

- The outcome depends on the diversity of the thematic experts as to how well it covers all sections of the H2020.

- Using longer specific phrases may result in very limited explicit findings of their presence in the legal texts (Council Decision 2013/743/EU and WPs).


Q1.2, Q2.2. and Q3.2.

- The objectivity of the method can be increased if at least two control groups of experts (‘Leader’ and ‘Buddy’) participate in the task.

- Selection and allocation of experts to ensure all parts of H2020 are covered can minimise the subjectivity.

- Preparation of key words should avoid creation of long phrases as they are found less useful in application of this methodology.

Recommendation Method is useful in the cases when a holistic approach and in depth knowledge of thematic fields is necessary for conducting analysis with understanding of the context.



Identif y

Table 2-XI: Expert exploratory methodology – from keywords, phrases and topics extracted by text

mining – for the purpose of Q1.2, Q2.2 and Q3.2

Advantages Q1.2, Q2.2, Q3.2:

- The selection of the document(s) and WPs for text mining is arbitrary, thus allowing choosing the most representative for the scope of the analysis; The methodology is capable to scan in a very short time (e.g. seconds-minutes) a large amount of text that would require hours or even days of work for being read by human experts.

- The methodology requires only a trained operator to execute the software, thus limiting the amount of human interaction required and the costs of its execution.

- The methodology is per-se objective, and the objectivity of the results produced (i.e. keywords, phrases and topics lists) can be easily documented.

- The methodology is implemented in the Provalis QDAMiner 4 and Wordstat 7 softwares, thus it is available on the market and does not require being developed/implemented in software.

- The methodology can be carried out with different documents and WPs, thus allowing comparing results in different contexts.

Disadvantages Q1.2, Q2.2, Q3.2:

- The selection of the document(s) and WPs for text mining is arbitrary, thus influencing the objectivity of the results.

- The methodology is does not require is, by definition, not interactive, thus not allowing and/or leaving room for human interaction.

- The methodology is implemented in the Provalis QDAMiner 4 and Wordstat 7 softwares, thus it is limited by the option that the software allows and does not leave room for modifications.

- The methodology does not provide qualitative information, thus not giving indications on the quality of the results.

- The evaluation of the results is performed by expert assessment, thus it involves human work-force, which might be expensive, time-consuming and affected by human bias.


Q1.2, Q2.2, Q3.2:

- Conduct a preliminary consultation in order to get a wide expert opinion on which documents and WPs must be taken as reference; Set a number of intermediate check-points where expert(s) can check the preliminary results, thus allowing human interaction.

- Implement custom software, which allows for more degree of freedom, thus implementing custom analyses that may be of specific interest for the application to the H2020.

- Results from Step 4 (topics) can be preliminary analysed since, compared to keywords and phrases, they can provide more qualitative aspects.

- No specific mitigation measures are envisaged, since the human assessment is highly needed at the end of the methodology.

Recommendation Method is useful in the cases when synthetic and concise information for evaluating the development of the WPs across a framework period is needed.



Identif y

However, it should be taken into account that it has proven to be valuable at the keyword and phrases level, while topic mining did not prove to be useful.



Identif y

Table 2-XII: Computer content analysis – from keywords, phrases and topics extracted by text mining –

for the purpose of Q1.2

Advantages Q1.2:

- The selection of the document(s) and WPs for text mining is arbitrary, thus allowing choosing the most representative for the scope of the analysis.

- The methodology is capable to scan in a very short time (e.g. seconds-minutes) a large amount of text that would require hours or even days of work for being read by human experts.

- The methodology requires only a trained operator to execute the software, thus limiting the amount of human interaction required and the costs of its execution.

- The methodology is per-se objective, and the objectivity of the results produced (i.e. keywords, phrases and topics lists) can be easily documented.

- The methodology is implemented in the Provalis QDAMiner 4 and Wordstat 7 softwares, thus it is available on the market and does not require being developed/implemented in software.

- The methodology can be carried out with different documents and WPs, thus allowing comparing results in different contexts.

Disadvantages Q1.2:

- The selection of the document(s) and WPs for text mining is arbitrary, thus influencing the objectivity of the results.

- The methodology is does not require is, by definition, not interactive, thus not allowing and/or leaving room for human interaction.

- The methodology is implemented in the Provalis QDAMiner 4 and Wordstat 7 softwares, thus it is limited by the option that the software allows and does not leave room for modifications; The methodology does not provide qualitative information, thus not giving indications on the quality of the results.

- The evaluation of the results is performed by expert assessment, thus it involves human work-force, which might be expensive, time-consuming and affected by human bias.


Q1.2:

- Conduct a preliminary consultation in order to get a wide expert opinion on which documents and WPs must be taken as reference.

- Set a number of intermediate check-points where expert(s) can check the preliminary results, thus allowing human interaction; Implement custom software, which allows for more degree of freedom, thus implementing custom analyses that may be of specific interest for the application to the H2020.

- Results from Step 4 (topics) can be preliminary analysed since, compared to keywords and phrases, they can provide more qualitative aspects.

- No specific mitigation measures are envisaged, since the human assessment is highly needed at the end of the methodology.



Identif y

Recommendation Method is useful in cases when it is necessary to conduct computer aided text mining, as a quick tool which produces results, with the goal of assessing relevance of H2020, in both supervised and unsupervised manner.

Table 2-XIII: Social media analysis – from selected social media – for the purpose of Q1.2, Q2.2 and Q3.2

Advantages - The methodology allows the analysis of Twitter accounts related to H2020, exploring the contents highlighted. The resulting information is useful to check whether citizens’ voices are present in the Twitter accounts selected.

- The methodology enables to monitor which priorities of the programme are present in H2020 project’s Twitter accounts and websites selected in order to check if they respond to what citizens express or some NSCEUs/STECIAs are missing.

Disadvantages - The use of the R-programme implies to collect only tweets published in the last 7/9 days; therefore, it has been used only for the analysis of hashtag #H2020. The data of tweets available in each twitter account selected has been downloaded manually through the conversion in pdf file in order to analyse if keywords selected were present or not.

- The selection of keywords analysed is limited to the results obtained in the period covered from bottom up approach.


- The collection of data is limited to a short period. To define a monitoring process to provide a monthly or yearly overview with more covered topics on twitter accounts selected.

Amount of Data available. The selection of twitter accounts available. Selection of relevant twitter accounts available related to H2020 in order to analyse them for the assessment of relevance. The selection will depend on the aim of the specific need analysis.

Recommendation Method is useful in the cases when it is necessary to assess whether the identified NSEUCs and technological and scientific advances are present. The analysis allows observing whether the contributions extracted from citizens’ voices are included or not in H2020 social media accounts and projects websites funded.

Applying relevanc

196 | P a g e EXPERT GROUP ON EVALUATION

Identif y

Appendix 3. Integrated keywords

II-1-1

Pillar: II. Industrial Leadership CRISTINA PLESCAN Part: II.1. Leadership in Enabling and Industrial Technologies Section: II.1.1. Information and Communication Technologies (ICT)

Code 0 Key-term is not covered

Code 1 Key-term is covered Decision WP-14-15 WP-16-17 Q1-covered 11 11 11 Total Q1 13 13 13 Q2-covered 38 38 38 Total Q2 38 38 38 Q3-covered 24 24 24 Total Q3 24 24 24

Decision WP-14-15 WP-16-17 Q1-covered 85% 85% 85% Q2-covered 100% 100% 100% Q3-covered 100% 100% 100%

Applying relevanc


Identif y

Question Source Keywords DeQ1 Expert boost investment Q1 Expert bridge digital divides Q1 Expert cybercrime Q1 Expert digital single market Q1 Expert digital technologies Q1 Expert enhance enabling technology Q1 Expert growth and jobs Q1 Expert high-speed internet Q1 Expert increase competitiveness Q1 Expert innovation 2.0 and entrepreneurship Q1 Expert smart grids using ICTs Q1 Expert strengthen innovation chain Q1 Expert improve research and innovation Q2 Expert 5G

Q2 Expert adaptive design Q2 Expert age appropriate design Q2 Expert cloud computing Q2 Expert data anonymization Q2 Expert data quality Q2 Expert data storage Q2 Expert datification

Applying relevanc


Identif y

Q2 Expert quantum Q2 Expert robotics Q2 Expert security Q2 Expert wireless Q2 Text Mining SOCIAL_MEDIA Question Source Keywords Q2 Text Mining VIRTUAL_REALITY Q2 Text Mining FACIAL_RECOGNITION Q2 Text Mining ONLINE_SHOPPING Q2 Text Mining PERSONAL_ROBOT_ASSISTANT Q2 Text Mining UBIQUITOUS_COMMUNICATION. Q2 Social Media iPad Q2 Social Media Soundcloud Q2 Social Media Amazon Q2 Social Media Twitter Q2 Social Media Periscope Q2 Social Media iPhone Q2 Social Media Snapchat Q2 Social Media Data protection Q2 Social Media Apple Q2 Social Media Android Q2 Social Media Internet Q2 Social Media Facebook Q2 Social Media Software Q2 Social Media Wikipedia Q2 Social Media Youtube Q2 Social Media open source-javascript runtime environment

Applying relevanc


Identif y

Q2 Social Media social networks Q2 Social Media online shopping Q2 Social Media VPNs (Virtual Private Network) Q2 Social Media RAID (redundant array of independent disks)

Question Source Keywords Q3 Expert Big Data Q3 Expert Speech technology Q3 Expert Advanced machine learning Q3 Expert Quantum computing Q3 Expert eHealth Q3 Expert Software alternative to pesticides Q3 Expert ICT for safer environment Q3 Patent autonomous vehicles Q3 Patent mobile communication Q3 Patent smartphone Q3 Patent software-controlled hardware Q3 Patent software patents Q3 Scientific publications Ultraprivate Smartphones Q3 Scientific publications Neuromorphic Chips Q3 Scientific publications Microscale 3-D Printing Q3 Scientific publications Mobile Collaboration Q3 Scientific publications Oculus Rift Q3 Scientific publications 3-D Images Q3 Scientific publications Alternative online payments Q3 Scientific publications Conversational Interfaces Q3 Scientific publications Robots That Teach Each Other

Applying relevanc


Identif y

Q3 Scientific publications Internet devices powered by Wi-Fi and other telecommunications signals

Q3 Scientific publications Robots that cooperate Q3 Scientific publications Artificial Intelligence

II-1-2

Pillar: II. Industrial Leadership CRISTINE Part: II.1. Leadership in Enabling and Industrial Technologies

Section: II.1.2. Nanotechnologies, Advanced Materials, Advanced Manufacturing and Processing, and Biot(NMPB)

Code 0 Key-term is not covered Code 1 Key-term is covered

Decision WP-14-15 WP-16-17 Q1-covered 5 5 5 Total Q1 5 5 5 Q2-covered 48 53 53 Total Q2 53 53 53 Q3-covered 15 15 15 Total Q3 15 15 15


Applying relevanc


Identif y

Question Source Keywords Decision Q1 Expert Improve research and innovation 1 Q1 Expert strengthen innovation chain 1 Q1 Expert increase competitiveness 1 Q1 Expert enhance enabling technology 1 Q1 Expert stimulate investment in new technologies 1 Q2 Expert nature-based solutions 0 Q2 Expert eco-innovation 0 Q2 Expert smart insulation 1 Q2 Expert vacuum insulation 1 Q2 Expert aerogel insulation 1 Q2 Expert biological building materials 1 Q2 Expert phase change materials 1 Q2 Expert nanotechnologies 1 Q2 Expert graphene 1 Q2 Expert molecular 1 Q2 Expert microsensors 1 Q2 Expert nanosafety 1 Q2 Expert nanopharmaceutical 1 Q2 Expert efficiency 1

Applying relevanc


Identif y

Q2 Expert electronics 1 Q2 Expert photonics 1 Q2 Expert nanodevices 1 Q2 Expert nanomaterials 1 Q2 Expert nanosystems 1 Q2 Expert miniaturisation 0 Q2 Expert nanorisk 1 Question Source Keywords Decision Q2 Expert coating 1 Q2 Expert laser 1 Q2 Expert composites 1 Q2 Expert metallurgy 1 Q2 Expert hybrid 1 Q2 Expert carbine 1 Q2 Expert biomaterials 1 Q2 Expert biotechnology 1 Q2 Expert bioengineering 1 Q2 Expert biodegradable 1 Q2 Expert biomanufactiring 1 Q2 Expert KET 1 Q2 Expert GMO 0 Q2 Expert pharmaceutical 1 Q2 Expert cell 1 Q2 Expert bioprocessing 1 Q2 Expert sustainability 1 Q2 Expert genomics 1 Q2 Expert molecular 1

Applying relevanc


Identif y

Q2 Expert biosystems 1 Q2 Expert biocatalysis 1 Q2 Expert bioinformatics 1 Q2 Expert metagenomics 1 Q2 Expert LCA 1 Q2 Expert SME 1 Q2 Expert composites 1

Question Source Keywords DecisioQ2 Expert lightweight Q2 Expert 4.0 Q2 Expert circularity Q2 Expert robotics Q2 Social media STEM_CELL_RESEARCH Q2 Social media STEM_CELL_THERAPY Q3 Expert Organic semiconductor Q3 Expert Black phosphorus Q3 Expert 2D Materials Q3 Expert Nanosensors and the Internet of Nanothings Q3 Expert Phenolic carbon Q3 Expert Tissue engineering Q3 Expert Biofibre-based pyjamas Q3 Patent Medical technology Q3 Patent autonomous vehicles Q3 Patent mobile communication Q3 Patent smartphone Q3 Patent software-controlled hardware

Applying relevanc


Identif y

Q3 Patent software patents Q3 Patent standard essential Q3 Patent technology convergence.

II-1-3 Pillar: II. Industrial Leadership david

Part: II.1. Leadership in Enabling and Industrial Technologies

Section: II.1.3. Space Code 0 Key-term is not covered Code 1 Key-term is covered


Decision WP-14-15 WP-16-17 Q1-covered 100% 100% 100%

Applying relevanc


Identif y

Q2-covered 100% 100% 100% Q3-covered 100% 100% 100%

Question Source Keywords Decision WP-14-15 Q1 Expert research and innovation 1 Q1 Expert increase competitiveness 1 Q1 Expert enabling technology 1 Q2 Expert navigation 1 Q2 Expert geoinformation 1 Q2 Expert climate 1 Q2 Expert satellite 1 Q2 Expert Copernicus 1 Q2 Expert Galileo 1 Q2 Expert surveillance 1 Q2 Expert tracking 1 Q2 Expert weather 1 Q2 Expert safety 1 Q2 Expert telecommunications 1 Q2 Expert environment 1 Q3 Scientific words Rosseta mission 1 Q3 Scientific publications Agile Robots 1

Applying relevanc


Identif y

Q3 Scientific publications Cube Sat 1

II-2 Pillar: II. Industrial Leadership CRISTINA PLESCAN Section: II.2. Access to risk finance Code 0 Key-term is not covered Code 1 Key-term is covered Decision WP-14-15 WP-16-17 Q1-covered 16 18 18 Total Q1 18 18 18 Q2-covered 10 10 10 Total Q2 12 12 12 Q3-covered 0 0 0 Total Q3 0 0 0 Decision WP-14-15 WP-16-17 Q1-covered 89% 100% 100%

Applying relevanc


Identif y

Q2-covered 83% 83% 83% Q3-covered #DIV/0! #DIV/0! #DIV/0! Question Source Keywords Decision Q1 Expert boost investment 1 Q1 Expert increase competitiveness 1 Q1 Expert fund absorption 1 Q1 Expert effective financial instruments 1 Q1 Expert hig-risk loans 1 Q1 Expert innovative start-ups 1 Q1 Text Mining TRADE_AND_INVESTMENT 0 Q1 Text Mining FINANCIAL_SECTOR 1 Q1 Text Mining PUBLIC_FINANCES 1 Q1 Text Mining CAPITAL_MARKETS 1 Q1 Text Mining FINANCIAL_INSTRUMENTS 1 Q1 Text Mining FINANCIAL_SERVICES 1 Q1 Text Mining FINANCIAL_SUPPORT 1 Q1 Text Mining INTERNATIONAL_FINANCIAL_INSTITUTIONS 1 Q1 Text Mining FREE_TRADE 0 Q1 Text Mining IMPROVE_THE_BUSINESS_ENVIRONMENT 1

Applying relevanc


Identif y

Q1 Text Mining ACCESS_TO_FINANCE 1 Q1 Text Mining FUNDING_MECHANISMS 1 Q2 Expert SME 1 Q2 Expert finance 1 Q2 Expert micro-enterprise 0 Q2 Expert small enterprise 1 Q2 Expert crisis 0 Q2 Expert banking 1 Question Source Keywords Decision Q2 Expert tax 0 Q2 Expert competitiveness 1 Q2 Expert agreements 1 Q2 Expert stakeholders 1 Q2 Expert transparency 0 Q2 Text Mining BOOST_ECONOMIC_GROWTH 1 Q2 Text Mining COMPETITION_AUTHORITY 1 Q2 Text Mining ECONOMIC_EFFECTIVENESS 1 Q3 NOT APPLICABLE

Applying relevanc


Identif y

II-3 Pillar: II. Industrial Leadership CRISTINA PLESCAN Section: II.3. Innovation in SMEs Code 0 Key-term is not covered Code 1 Key-term is covered Decision WP-14-15 WP-16-17 Q1-covered 7 7 7 Total Q1 7 7 7 Q2-covered 7 7 7 Total Q2 7 7 7 Q3-covered 10 10 10 Total Q3 10 10 10

Applying relevanc


Identif y


Question Source Keywords Decision Q1 Expert improve research and innovation 1Q1 Expert strengthen innovation chain 1Q1 Expert boost investment 1Q1 Expert increase competitiveness 1Q1 Expert entrepreneurship 1Q1 Expert innovative start-ups 1Q1 Text Mining IMPROVE_THE_BUSINESS_ENVIRONMENT 1Q2 Expert new enterprises 1Q2 Expert low unemployment 1Q2 Expert digital skills 1Q2 Expert digital economy 1Q2 Expert industry 4.0 1Q2 Expert entrepreneurship 1Q2 Expert knowledge transfer 1

Applying relevanc


Identif y

Q3 Social Media alternative currencies (bitcoin & lebocoin) 1Q3 Patent Computer technology 1Q3 Patent Electrical machinery 1Q3 Patent Digital communication 1Q3 Patent Medical technologies 1Q3 Patent autonomous vehicles 1Q3 Patent mobile communication 1Q3 Patent smartphone 1Q3 Patent software-controlled hardware 1Q3 Patent software patents 1

III-1

Section: III.1 Health, Demographic Change and Wellbeing Toivo

Katerin

Code 0 Key-term is not covered

Code 1 Key-term is covered Decision WP-14-15 WP-16-17 Q1-covered 8 7 7 Total Q1 11 11 11 Q2-covered 18 18 17 Total Q2 28 28 28 Q3-covered 14 13 13 Total Q3 25 25 25

Applying relevanc


Identif y


Question Source Keywords DecisioQ1 Text Mining PREVENTION_AND_TREATMENT 1 Q1 Text Mining NONCOMMUNICABLE_DISEASES 0 Q1 Text Mining PUBLIC_HEALTH 1 Q1 Text Mining QUALITY_OF_LIFE 1 Q1 Text Mining ANTIMICROBIAL_RESISTANCE 1 Q1 Text Mining HEALTH_SYSTEMS 1 Q1 Text Mining INFECTIOUS_DISEASES 1 Q1 Text Mining HEALTH_TARGETS 0 Q1 Text Mining HEALTH_CARE 1 Q1 Text Mining MENTAL_HEALTH 1 Q1 Text Mining NEGLECTED_TROPICAL_DISEASES 0 Q2 Expert health service 1

Applying relevanc


Identif y

Q2 Expert health literacy 1 Q2 Expert poverty 1 Q2 Expert ageing 1 Q2 Expert sustainable health 1 Q2 Expert climate change 1 Q2 Expert wellbeing 1 Q2 Expert cancer 1 Q2 Expert cardiovascular 1 Q2 Expert prevention 1 Q2 Expert migration 1 Q2 Expert care 1 Q2 Expert illness 1 Q2 Expert vaccines 1

Question Source Keywords DecisioQ2 Expert participation 1 Q2 Expert reproductive health 0 Q2 Social Media Nervous system 0 Q2 Social Media Asperger-Syndrome 0 Q2 Social Media zika virus 0 Q2 Social Media Lyme disease 0 Q2 Social Media tuberculosis 1 Q2 Social Media multiple sclerosis 0 Q2 Social Media syphilis 0 Q2 Social Media fibromyalgie 0 Q2 Social Media mental illness 1 Q2 Social Media meningitis 0

Applying relevanc


Identif y

Q2 Social Media drugs 1 Q2 Social Media human papiloma virus 0 Q3 Expert Optogenetics 0 Q3 Expert Systems Metabolic Engineering 0 Q3 Expert Molecular methods 0 Q3 Expert vaccines 1 Q3 Expert communicable diseases 1 Q3 Social Media Health and safety 1 Q2 Social Media nutrition 1 Q2 Social Media HIV 1 Q3 Social Media diabetes 1 Q3 Scientific publications Brain Mapping 1 Q3 Scientific publications Genome Editings 1

Question Source Keywords DecisioQ3 Scientific publications Simple blood tests for cancer 1 Q3 Scientific publications Growing human brain cells for diagnostics 1 Q3 Scientific publications Internet of DNA 0 Q3 Scientific publications Immune Engineering 1 Q3 Scientific publications DNA App Store 0 Q3 Scientific publications Ageing 1 Q3 Scientific publications Chips that mimic the brain 0 Q3 Scientific publications Manipulating memories 0 Q3 Scientific publications DNA sequencing 1 Q3 Scientific publications A vaccine against Ebola 0 Q3 Scientific publications Bioengineered yeasts capable of producing the baine 0 Q3 Scientific publications Psychology research 1

Applying relevanc


Identif y

Q3 Scientific publications Proteins by design 0 Q3 Scientific publications Metalenses 0

III-2

Section: III.2 Food Security, Sustainable Agriculture and Forestry, Marine, Maritime and Inland WaterBioeconomy


Decision WP-14-15 WP-16-17 Q1-covered 34 34 34 Total Q1 34 34 34 Q2-covered 41 41 41 Total Q2 44 44 44

Applying relevanc


Identif y

Q3-covered 11 11 11 Total Q3 11 11 11


Question Source Keywords DQ1 Expert food safety Q1 Expert hunger Q1 Expert rural infrastructure Q1 Expert plant and livestock gene banks Q1 Expert agricultural in developing countries Q1 Expert oceans and international law Q1 Expert ocean acidification Q1 Expert agricultural productivity

Applying relevanc


Identif y

Q1 Expert small-scale food producers (including women, indigenous peoples, family farmers, pastoralists and fishers)

Q1 Expert sustainable consumption Q1 Expert sustainable food production Q1 Expert resilient agricultural practices Q1 Expert land and soil quality Q1 Expert regulate harvesting Q1 Expert overfishing, illegal fishing Q1 Expert plans to restore fish stocks Q1 Expert sustainable use of marine resources Q1 Expert limit extreme food price volatility Q1 Expert maintain genetic diversity of seeds Q1 Expert maintain cultivated plants Q1 Expert maintain farmed and domesticated animals Q1 Expert maintain wild species Q1 Text Mining NATURAL_RESOURCES Q1 Text Mining GREEN_GROWTH

Question Source Keywords Q1 Text Mining RESOURCE_EFFICIENT Q1 Text Mining SUSTAINABLE_DEVELOPMENT_GOALS Q1 Text Mining INFRASTRUCTURE_INVESTMENT Q1 Text Mining NATURAL_DISASTERS Q1 Text Mining FOOD_SECURITY Q1 Text Mining FOOD_NUTRITION Q1 Text Mining RENEWABLE_ENERGY

Applying relevanc


Identif y

Q1 Text Mining RURAL_DEVELOPMENT Q1 Text Mining CLIMATE_AND_ENERGY Q1 Text Mining CLIMATE_ACTION Q2 Expert sustainability Q2 Expert Trade Q2 Expert agriculture Q2 Expert biodiversity Q2 Expert Food Q2 Expert Security Q2 Expert sustainable Q2 Expert Forestry Q2 Expert Inland Q2 Expert Wáter Q2 Expert bioeconomy Q2 Expert Rural Q2 Expert Hunger Q2 Expert Land Q2 Expert development Q2 Expert adaptation Question Source Keywords Q2 Expert fisheries Q2 Expert sea Q2 Expert production Q2 Expert consumption Q2 Expert impact Q2 Expert biological

Applying relevanc


Identif y

Q2 Expert ecological Q2 Expert biotechnology Q2 Expert healthy Q2 Expert consumers Q2 Expert acidification Q2 Expert degradation Q2 Expert ocean Q2 Expert biomass Q2 Expert fresh Q2 Expert drinking Q2 Expert waste Q2 Expert ecosystem Q2 Expert aquatic Q2 Expert overfishing Q2 Expert circular economy Q2 Expert desalination Q2 Expert poverty Q2 Expert industrialisation Q2 Expert GMO Q2 Text Mining FOOD_THEY_BUY

Question Source Keywords Q2 Text Mining EVERYDAY_PRODUCTS Q2 Text Mining WORRY_ABOUT_THE_QUALITY Q3 Expert sustainable intensification Q3 Expert ocean acidification Q3 Expert rural development

Applying relevanc


Identif y

Q3 Expert high nature value farming Q3 Expert agroecology Q3 Social Media Protecting biodiversity Q3 Scientific publications Agricultural Drones Q3 Scientific publications Reverse-osmosis desalination plant Q3 Scientific publications Supercharged Photosynthesis Q3 Scientific publications Precise Gene Editing in Plants Q3 Scientific publications Evolutionary Biology

III-3 Section: III.3 Secure, Clean and Efficient Energy Irena


Applying relevanc


Identif y



Question Source Keywords Decision WP-14-15Q1 Expert decarbonising the economy 1 Q1 Expert reduce GHG 1 Q1 Expert increase renewable sources use 1 Q1 Expert increase energy efficiency 1

Applying relevanc


Identif y

Q1 Expert smart grids using ICTs 1 Q1 Expert facilitate energy investments 1 Q1 Expert clean energy 1 Q1 Expert diversify the energy sources 1 Q1 Expert reduce energy dependency 0 Q1 Expert reform energy policy 1 Q1 Expert European Energy Union 0 Q1 Text Mining NATURAL_RESOURCES 1 Q1 Text Mining LOW_CARBON 1 Q1 Text Mining RESOURCE_EFFICIENT 1 Q1 Text Mining INFRASTRUCTURE_INVESTMENT 1 Q1 Text Mining NATURAL_DISASTERS 1 Q1 Text Mining LOW_CARBON_ECONOMY 1 Q1 Text Mining RENEWABLE_ENERGY 1 Q1 Text Mining CLIMATE_AND_ENERGY 1 Q1 Text Mining GREENHOUSE_GAS_EMISSIONS 1 Q1 Text Mining GREEN_GROWTH 1 Q1 Text Mining CLIMATE_ACTION 1 Q1 Text Mining REDUCE_EMISSIONS 1

Question Source Keywords Decision Q2 Expert adaptive design Q2 Expert aerogel insulation Q2 Expert age appropriate design Q2 Expert artificial intelligence

Applying relevanc


Identif y

Q2 Expert batteries Q2 Expert biodiversity Q2 Expert bioenergy Q2 Expert biofuel Q2 Expert biological architecture Q2 Expert biological building materials Q2 Expert blue infrastructure Q2 Expert building energy Q2 Expert building integrated renewables Q2 Expert building performance Q2 Expert building renovation Q2 Expert buildings recyclability Q2 Expert carbon Q2 Expert carbondioxide Q2 Expert circular economy Q2 Expert city farming Q2 Expert climate change Q2 Expert climate policies Q2 Expert CO2 Q2 Expert competitiveness Q2 Expert consumers Q2 Expert consumption Question Source Keywords Decision Q2 Expert cradle to cradle Q2 Expert decarbonisation Q2 Expert deforestation Q2 Expert district energy efficiency

Applying relevanc


Identif y

Q2 Expert eco design Q2 Expert eco-districts Q2 Expert eco-innovation Q2 Expert ecological sites Q2 Expert electricity Q2 Expert emissions Q2 Expert end-user Q2 Expert energy efficiency Q2 Expert energy flexibility Q2 Expert forest Q2 Expert fossil Q2 Expert fusion Q2 Expert geothermal Q2 Expert green housing Q2 Expert green infrastructure Q2 Expert health Q2 Expert health impact Q2 Expert Sustainable neighbourhood Q2 Expert heating Q2 Expert high performance ventilation Q2 Expert home apps

Question Source Keywords Decision Q2 Expert households Q2 Expert hybrid ventilation Q2 Expert hydrocarbons Q2 Expert indoor pollution

Applying relevanc


Identif y

Q2 Expert insulation Q2 Expert LCA Q2 Expert lifestyle quality Q2 Expert local Q2 Expert market Q2 Expert minerals Q2 Expert mitigation Q2 Expert multi generation living Q2 Expert nature-based solutions Q2 Expert nuclear Q2 Expert Nearly zero-energy buildings Q2 Expert on-site renewables Q2 Expert overconsumption Q2 Expert passive house Q2 Expert phase change materials Q2 Expert pollution Q2 Expert Poor indoor environment Q2 Expert pretreatment Q2 Expert prosument Q2 Expert recycling Q2 Expert refurbishment

Question Source Keywords DecisionQ2 Expert renewable Q2 Expert renewable energy Q2 Expert renewables Q2 Expert renovation

Applying relevanc


Identif y

Q2 Expert resilience Q2 Expert resource use inefficiency Q2 Expert resources Q2 Expert risk Q2 Expert robotics Q2 Expert secure Q2 Expert smart grid Q2 Expert smart insulation Q2 Expert solar Q2 Expert source separation Q2 Expert storage Q2 Expert sustainability Q2 Expert sustainable building Q2 Expert sustainable development Q2 Expert sustainable lifestyle Q2 Expert sustainable living Q2 Expert transport Q2 Expert vertical gardens Q2 Expert wind Q2 Expert energy Q2 Expert smart appliances

Question Source Keywords DecisionQ2 Text Mining SOLAR_PANELS Q2 Text Mining AIR_POLLUTION Q2 Text Mining BIG_POLLUTERS Q2 Text Mining WIND_FARMS

Applying relevanc


Identif y

Q2 Text Mining AGRICULTURAL_POLLUTION Q2 Text Mining ENERGY_SAVING Q2 Text Mining HOME_APPLIANCES Q2 Text Mining SOIL_DEGRADATION Q2 Text Mining TYPES_OF_WASTE Q2 Text Mining ENVIRONMENTAL_DECISIONS Q2 Text Mining GARDEN_WASTE Q2 Text Mining CIRCULAR_ECONOMY Q2 Social Media IAEA Q2 Social Media flixbus (social public transportation initiatives) Q3 Expert pollution Q3 Expert emissions Q3 Expert low-carbon Q3 Expert smart buildings Q3 Expert smart cities Q3 Expert sustainable Q3 Expert renewable Q3 Expert electricity Q3 Expert sustainability AND energy Q3 Expert local AND energy Q3 Expert pollution AND energy

Question Source Keywords DecisioQ3 Expert ecological architecture Q3 Expert green housing Q3 Social Media Hybrid airships Q3 Scientific publications Smart Wind and Solar Power

Applying relevanc


Identif y

Q3 Scientific publications High-efficiency solar panels

III-4 Section: III.4 Smart, Green and Integrated Transport Michele

Applying relevanc


Identif y




Question Source Keywords DecQ1 Expert decarbonising the economy Q1 Expert reduce greenhouse gas emissions

Applying relevanc


Identif y

Q1 Expert increase renewable sources use Q1 Expert modernise transport sector Q1 Expert develop green technologies Q1 Expert smart grids using ICTs Q1 Expert increase competitiveness Q1 Text Mining NATURAL_RESOURCES Q1 Text Mining GREEN_GROWTH Q1 Text Mining RESOURCE_EFFICIENT Q1 Text Mining INFRASTRUCTURE_INVESTMENT Q1 Text Mining LOW_CARBON_ECONOMY Q1 Text Mining RENEWABLE_ENERGY Q1 Text Mining AVIATION_SECURITY Q1 Text Mining CLIMATE_AND_ENERGY Q1 Text Mining GREENHOUSE_GAS_EMISSIONS Q1 Text Mining LOW_CARBON Q1 Text Mining SMART_GROWTH Q1 Text Mining CLIMATE_ACTION Q1 Text Mining INTERNATIONAL_CIVIL_AVIATION Q1 Text Mining REDUCE_EMISSIONS Q2 Expert transport Q2 Expert electric cars Q2 Expert smart Q2 Expert travel

Question Source Keywords DeciQ2 Expert oil Q2 Expert decarbonisation

Applying relevanc


Identif y

Q2 Expert infrastructure Q2 Expert mobility Q2 Expert trade Q2 Expert network Q2 Expert greenhouse Q2 Expert emissions Q2 Expert climate Q2 Expert green transport Q2 Expert electric vehicles Q2 Expert e-mobility Q2 Expert low carbon Q2 Expert environment Q2 Expert carbon footprint Q2 Expert emissions Q2 Expert GHGs Q2 Expert clean air Q2 Expert CO2 Q2 Expert decarbonisation Q2 Expert green transport Q2 Expert green Q2 Expert integrated Q2 Expert transport Q2 Expert electric Q2 Expert vehicle Question Source Keywords DeciQ2 Expert road Q2 Expert emissions

Applying relevanc


Identif y

Q2 Expert greenhouse Q2 Expert aviation Q2 Expert mobility Q2 Expert congestion Q2 Expert efficiency Q2 Expert electrical Q2 Expert logistics Q2 Expert noise Q2 Expert traffic Q2 Expert pollution Q2 Expert climate Q2 Expert urban Q2 Expert regional Q2 Expert car Q2 Expert battery Q2 Social Media Mediterranean Corridor Q2 Social Media Mobility (transport) Q2 Social Media Public Transport Q2 Social Media routes

Question Source Keywords DeQ3 Expert low carbon transport

Applying relevanc


Identif y

Q3 Expert Urban mobility Q3 Expert Electric vehicles Q3 Expert air AND transport Q3 Expert pollution AND transport Q3 Expert climate AND transport Q3 Scientific publications Agile Robots Q3 Scientific publications Car-to-Car Communication Q3 Scientific publications Reusable Rockets Q3 Scientific publications Autopilot cars

III-5

Applying relevanc


Identif y

Section: III.5 Climate Action, Environment, Resource Efficiency and Raw Materials Code 0 Key-term is not covered Code 1 Key-term is covered



Question Source Keywords Decision

Applying relevanc


Identif y

Q1 Expert water scarcity 1Q1 Expert sanitation and hygiene 0Q1 Expert desertification 0Q1 Expert trafficking of protected species 1Q1 Expert mountain ecosystems 1Q1 Expert biodiversity and ecosystems 1Q1 Expert coastal and marine areas 1Q1 Expert decarbonising the economy 1Q1 Expert desalination 0Q1 Expert green technologies 1Q1 Expert terrestrial and inland freshwater ecosystems 1Q1 Expert climate change 1Q1 Expert global warming 1Q1 Expert water quality 1Q1 Expert energy efficiency 1Q1 Expert recycling 1Q1 Expert water-use efficiency 1Q1 Expert limit temperature increase 1Q1 Expert low-carbon economy 1Q1 Expert hazardous chemicals and materials 1Q1 Expert biodiversity loss 1Q1 Expert Excellent science 1Q1 Expert food waste at the retail and consumer levels 0Q1 Expert greenhouse gas emissions 1Q1 Expert pollution 1

Applying relevanc


Identif y

Question Source Keywords Decision Q1 Expert tackle resource scarcity 1Q1 Expert wastewater treatment 1Q1 Expert water efficiency 1Q1 Expert water harvesting 1Q1 Expert water recycling and reuse technologies 1Q1 Text Mining CLIMATE CHANGE 1Q1 Text Mining CLIMATE ACTION 1Q1 Text Mining CLIMATE AND ENERGY 1Q1 Text Mining CLIMATE FINANCE 1Q1 Text Mining ENERGY EFFICIENCY 1Q1 Text Mining GHG EMISSIONS 1Q1 Text Mining GLOBAL EMISSIONS 1Q1 Text Mining GREEN GROWTH 1Q1 Text Mining GREENHOUSE GAS EMISSIONS 1Q1 Text Mining INDUSTRIAL POLICY 1Q1 Text Mining INFRASTRUCTURE INVESTMENT 1Q1 Text Mining LOW CARBON 1Q1 Text Mining NATURAL DISASTERS 1Q1 Text Mining NATURAL RESOURCES 1Q1 Text Mining REDUCE EMISSIONS 1Q1 Text Mining RENEWABLE ENERGY 1Q1 Text Mining RESOURCE EFFICIENCY 1Q1 Text Mining SUSTAINABLE DEVELOPMENT 1Q1 Text Mining LOW CARBON ECONOMY 1Q1 Text Mining RAW MATERIALS 1

Applying relevanc


Identif y

Question Source Keywords Decision Q2 Expert aerogel insulation 0Q2 Expert biodiversity 1Q2 Expert biological architecture 1Q2 Expert buildings recyclability 0Q2 Expert circularity 1Q2 Expert city farming 0Q2 Expert climate action 1Q2 Expert climate change 1Q2 Expert climate policies 1Q2 Expert deforestation 1Q2 Expert eco-innovation 1Q2 Expert ecosystems 1Q2 Expert energy efficiency 1Q2 Expert environment quality 1Q2 Expert equilibrium of natural and built environment 1Q2 Expert food waste 0Q2 Expert footprint 0Q2 Expert green infrastructure 1Q2 Expert greenhouse 1Q2 Expert greening cities 0Q2 Expert high performance ventilation 1Q2 Expert home apps 0Q2 Expert hybrid ventilation 1Q2 Expert indoor environment 0

Applying relevanc


Identif y

Q2 Expert material loops 1

Question Source Keywords Decision Q2 Expert micropollutants 0Q2 Expert mineral resources 1Q2 Expert mitigation 1Q2 Expert nature-based solutions 0Q2 Expert overconsumption 1Q2 Expert pollution 1Q2 Expert raw material 1Q2 Expert recycling 1Q2 Expert renewable energy 1Q2 Expert resilience 1Q2 Expert resource 1Q2 Expert resource assessment 1Q2 Expert reusable 1Q2 Expert risks 1Q2 Expert rural 0Q2 Expert scarcity 1Q2 Expert smart appliances 0Q2 Expert smart insulation 0Q2 Expert sustainability 1Q2 Expert sustainable lifestyle 1Q2 Expert sustainable living 1Q2 Expert sustainable urban environment 1Q2 Expert teledetection 1Q2 Expert temperature increase 1

Applying relevanc


Identif y

Q2 Expert vertical gardens 0

Question Source Keywords Decision Q2 Expert vulnerability 1 Q2 Expert waste 1 Q2 Expert circular economy 1 Q2 Expert waste management 1 Q2 Text Mining RENEWABLE ENERGY 1 Q2 Text Mining SEPARATE WASTE 1 Q2 Text Mining PLASTIC WASTE 0 Q2 Text Mining ENVIRONMENTAL PROTECTION 1 Q2 Text Mining ENERGY EFFICIENT 1 Q2 Text Mining HOUSEHOLD WASTE 1 Q2 Text Mining REDUCE WASTE 1 Q2 Text Mining WATER POLLUTION 1 Q2 Text Mining EFFICIENT USE OF RESOURCES 1 Q2 Text Mining ENERGY CONSUMPTION 1 Q2 Text Mining GENERATING TOO MUCH WASTE 1 Q2 Text Mining NATURAL RESOURCES 1 Q2 Text Mining RAW MATERIALS 1 Q2 Text Mining WASTE FOR RECYCLING 1 Q2 Text Mining NOISE POLLUTION 0 Q2 Text Mining PLASTIC BAGS 1 Q2 Text Mining RECYCLED MATERIALS 1 Q2 Text Mining ELECTRONIC WASTE 1 Q2 Text Mining WATER CONSUMPTION 1

Applying relevanc


Identif y

Q2 Social Media Climate change 1 Q2 Social Media Environment 1

Question Source Keywords Decision Q2 Social Media Sustainability 1 Q2 Social Media Environmental Risk 1 Q2 Social Media Globalization 1 Q2 Social Media Wind energy 1 Q2 Social Media Solar energy 1 Q2 Social Media Recycling 1 Q2 Social Media Renewable energy 1

Applying relevanc


Identif y

Question Source Keywords DeQ3 Expert Smart city Q3 Expert Alternative Electricity Q3 Expert Artificial photosynthesis Q3 Expert circular economy Q3 Expert climate change adaptation Q3 Expert climate change mitigation Q3 Expert climate services Q3 Expert Data base on climate change Q3 Expert Energy storage Q3 Expert Fossil fuel Q3 Expert Fuel cell vehicles Q3 Expert Green light-emitting diodes Q3 Expert nuclear fusion

Q3 Expert Plasma pre-treatment detecting deformations and movements landslides

Q3 Expert raw material Q3 Expert Recyclable thermoset plastics Q3 Expert solar energy Q3 Expert sustainable lifestyles Q3 Expert waste management Q3 Social Media Fighting pollution

Applying relevanc


Identif y

Q3 Social Media Fighting climate change Q3 Social Media Reducing and managing waste Q3 Social Media Environmental technologies

Q3 Scientific publications Nano-Architecture

Q3 Scientific publications Full waveform tomography

III-6

Section: III.6 Europe in a changing world - Inclusive, innovative and reflective societies Ramon,




Applying relevanc


Identif y

Question Source Keywords Decision Q1 Expert enhance performance of education systems 1 Q1 Expert increase development of new skills 1 Q1 Expert boost for new jobs 1 Q1 Expert poverty 1 Q1 Expert growth and jobs 1 Q1 Expert live in dignity 1 Q1 Expert decent jobs 1 Q1 Expert simplify regulations 1 Q1 Expert digital skills 1 Q1 Expert fight corruption 0 Q1 Expert well-being of societies 1 Q1 Expert youth employment 1 Q1 Expert anti-corruption capacity building 0 Q1 Expert provide access to justice 0 Q1 Expert inclusive institutions 1 Q1 Expert social protection policies 0

Applying relevanc


Identif y

Q1 Expert Debt on a sustainable path 0 Q1 Text Mining DEVELOPMENT STRATEGIES 1 Q1 Text Mining DIGITAL ECONOMY 1 Q1 Text Mining ECONOMIC AND DEVELOPMENT COOPERATION 1 Q1 Text Mining ECONOMIC CRISIS 1 Q1 Text Mining EDUCATION AND TRAINING 1 Q1 Text Mining EDUCATION SYSTEMS 1 Q1 Text Mining EMERGING ECONOMIES 1 Q1 Text Mining EMPLOYMENT RATE 1

Question Source Keywords Decision Q1 Text Mining EQUAL OPPORTUNITIES 1 Q1 Text Mining FINANCING FOR DEVELOPMENT 1 Q1 Text Mining FISCAL CONSOLIDATION 0 Q1 Text Mining FISCAL TRANSPARENCY 0 Q1 Text Mining FUNDAMENTAL RIGHTS 1 Q1 Text Mining GROWTH AND JOBS 1 Q1 Text Mining INFORMATION SHARING 0 Q1 Text Mining INTERNAL MARKET 0 Q1 Text Mining MONEY LAUNDERING 0 Q1 Text Mining NEIGHBOURHOOD POLICY 0 Q1 Text Mining PERSONS WITH DISABILITIES 1 Q1 Text Mining POVERTY ERADICATION 1 Q1 Text Mining QUALITY EDUCATION 1 Q1 Text Mining QUALITY OF LIFE 1 Q1 Text Mining REFUGEES AND OTHER DISPLACED PERSONS 0

Applying relevanc


Identif y

Q1 Text Mining RESPECT FOR HUMAN RIGHTS 1 Q1 Text Mining SINGLE CURRENCY 1 Q1 Text Mining SINGLE MARKET 1 Q1 Text Mining SOCIAL PROTECTION 1 Q1 Text Mining SPEED INTERNET 1 Q1 Text Mining SUSTAINABLE AND INCLUSIVE GROWTH 1

Question Source Keywords Decision Q2 Expert open and innovative government 1 Q2 Expert innovation talents and skills 1 Q2 Expert green innovation 1 Q2 Expert public sector innovation 1 Q2 Expert health innovation 1 Q2 Expert inclusive innovation 1 Q2 Expert knowledge creation and diffusion 0 Q2 Expert citizens empowerment 1 Q2 Expert public sector reform 0 Q2 Expert partnerships with citizens 1 Q2 Text Mining ECONOMIC GROWTH 1 Q2 Text Mining QUALITY OF LIFE 1 Q2 Text Mining ECONOMIC AND FINANCIAL CRISIS 1 Q2 Text Mining COMPETITION POLICY 1

Applying relevanc


Identif y

Q2 Text Mining LEVEL OF EDUCATION 1 Q2 Text Mining EMPLOYMENT OPPORTUNITIES 1 Q2 Text Mining FACING FINANCIAL DIFFICULTIES 1 Q2 Text Mining SOCIAL RELATIONSHIPS 1 Q2 Text Mining SOLIDARITY AND RESPONSIBILITY 1 Q2 Text Mining HOUSEHOLD INCOME 1 Q2 Text Mining MEDICAL EXPENSES 1 Q2 Text Mining SINGLE MARKET 1 Q2 Text Mining SOCIAL FACTORS 1 Q2 Text Mining FINANCIAL CRISIS 1 Q2 Text Mining FINANCIAL INCENTIVES 1 Q2 Text Mining SINGLE CURRENCY 1

Question Source Keywords Decision Q2 Text Mining ECONOMIC ANALYSES 1 Q2 Text Mining ECONOMIC CRISIS 1 Q2 Text Mining EMPLOYMENT OPPORTUNITIES 1 Q2 Social Media Dialogue & politics 1 Q2 Social Media Justice 1 Q2 Social Media Migration 1 Q2 Social Media Politics 1 Q2 Social Media Public Policies 1 Q2 Social Media Refugees 0 Q2 Social Media Social inclusion 1 Q2 Social Media Social Innovation 0 Q2 Social Media Sustainable Tourism 1 Q2 Social Media Work 1

Applying relevanc


Identif y

Q2 Social Media Electoral elections 0 Q2 Social Media Solidarity 1 Q2 Social Media Discrimination 0 Q2 Social Media Jobs 1 Q2 Social Media Islam 0 Q2 Social Media Leadership 1 Q2 Social Media Public Opinion 1 Q2 Social Media Political issues 1 Q2 Social Media peace 1 Q2 Social Media human rights 1

Question Source Keywords Decision Q3 Expert social inclusion 1 Q3 Expert income inequality 0 Q3 Expert migration management 1 Q3 Expert poverty 0 Q3 Expert migration 1 Q3 Expert natural heritage 1 Q3 Expert demography 1 Q3 Social Media Competition 1 Q3 Social Media Sectorial competitiveness 1 Q3 Social Media Rights of persons with disabilities 1 Q3 Social Media Dignity 0 Q3 Social Media Education 1

Applying relevanc


Identif y

Q3 Social Media Employment 1 Q3 Social Media Justice 1

III-7

Section: III.7 Secure societies – Protecting freedom and security of Europe and its citizens Ramon


Decision WP-14-15 WP-16-17 Q1-covered 37 35 25 Total Q1 71 49 49 Q2-covered 44 39 29 Total Q2 60 57 57 Q3-covered 4 4 4

Applying relevanc


Identif y

Total Q3 6 6 6 Decision WP-14-15 WP-16-17 Q1-covered 52% 71% 51% Q2-covered 73% 68% 51% Q3-covered 67% 67% 67%

Question Source Keywords Decision W1

Q1 Expert personal data protection 0 Q1 Expert fight corruption 0 Q1 Expert justice 1 Q1 Expert fundamental rights 1 Q1 Expert mutual trust 1 Q1 Expert fight abuse 0 Q1 Expert reduce exploitation 0 Q1 Expert well-being of societies 1 Q1 Expert climate-related extreme events 1

Applying relevanc


Identif y

Q1 Expert economic, social and environmental shocks and disasters 1 Q1 Expert secure cyberspace 1 Q1 Expert anti-corruption capacity building 0 Q1 Expert law enforcement cooperation 1 Q1 Expert promote peaceful and inclusive societies 1 Q1 Expert access to justice 1 Q1 Expert inclusive institutions 1 Q1 Expert nationality discrimination 0 Q1 Expert gender discrimination 0 Q1 Expert racial discrimination 0 Q1 Expert ethnic discrimination 0 Q1 Expert religion discrimination 0 Q1 Expert belief discrimination 0 Q1 Expert discrimination for disability 0 Q1 Expert age discrimination 0 Q1 Expert sexual orientation discrimination 0


Q1 Expert discrimination for people belonging to a minority 0 Q1 Expert organised crime 1 Q1 Expert human trafficking 1 Q1 Expert smuggling 0 Q1 Expert cybercrime 1 Q1 Expert cross-border crime 0 Q1 Expert terrorism 1 Q1 Expert radicalisation 1

Applying relevanc


Identif y

Q1 Expert manage migration 1 Q1 Expert legal migration 1 Q1 Expert anticipate events early 1 Q1 Expert modern slavery 0 Q1 Expert child labour 0 Q1 Expert child soldiers 0 Q1 Expert ensure equal opportunity 1 Q1 Expert stronger security and defence 0 Q1 Expert cooperation in defence procurement 1 Q1 Expert reinforce the work of Frontex 1 Q1 Expert rapid border interventions 0 Q1 Expert reduce illicit financial and arms flows 0 Q1 Expert combat all forms of organized crime 1 Q1 Expert substance abuse, narcotic drug abuse and alcohol 0

Q1 Text Mining VIOLENT_EXTREMISM 1

Q1 Text Mining PEACE_AND_SECURITY 1

Q1 Text Mining DATA_PROTECTION 0

Question Source Keywords Decision

Q1 Text Mining INTERNATIONAL_LAW 1

Q1 Text Mining SOCIAL_AND_TERRITORIAL_COHESION 1

Q1 Text ENERGY_SECURITY 1

Applying relevanc


Identif y

Mining

Q1 Text Mining GLOBAL_GOVERNANCE 1

Q1 Text Mining NUCLEAR_SAFETY 0

Q1 Text Mining TERRORIST_FINANCING 0

Q1 Text Mining TERRORIST_GROUPS 1

Q1 Text Mining VULNERABLE_SITUATIONS 1

Q1 Text Mining BORDER_SECURITY 0

Q1 Text Mining CHEMICAL_WEAPONS 0

Q1 Text Mining COMMON_ASYLUM 0

Q1 Text Mining COUNTERING_TERRORISM_AND_VIOLENT_EXTREMISM 1

Q1 Text Mining CYBER_SECURITY 1

Q1 Text Mining EARLY_WARNING 1

Q1 Text Mining EXIT_FROM_THE_CRISIS 1

Q1 Text Mining FINANCIAL_STABILITY 1

Q1 Text FREE_MOVEMENT 0

Applying relevanc


Identif y

Mining

Q1 Text Mining HAZARDOUS_CHEMICALS 0

Q1 Text Mining MINSK_AGREEMENTS 0

Q1 Text Mining NEIGHBOURHOOD_POLICY 1

Q1 Text Mining TAX_EVASION 0


Q2 Expert aid 1 Q2 Expert anti-islam 0 Q2 Expert asylum 0 Q2 Expert bioweapon 0 Q2 Expert borders 1 Q2 Expert bullying 0 Q2 Expert citizenship 1 Q2 Expert cooperation 1 Q2 Expert crime 1 Q2 Expert crisis 1 Q2 Expert cybercrime 1 Q2 Expert defence 1

Applying relevanc


Identif y

Q2 Expert energy 1 Q2 Expert environment 1 Q2 Expert equality 1 Q2 Expert extremism 1 Q2 Expert fairness 1 Q2 Expert freedom 1 Q2 Expert human rights 1 Q2 Expert immigration 0 Q2 Expert inclusion 1 Q2 Expert inequality 1 Q2 Expert jobs 1 Q2 Expert migration 1 Q2 Expert mobility 1


Q2 Expert peace 1 Q2 Expert police 1 Q2 Expert policy 1 Q2 Expert poverty 1 Q2 Expert privacy 1 Q2 Expert protection 1 Q2 Expert radical 1 Q2 Expert radicalization 1 Q2 Expert refugees 1 Q2 Expert rights 1 Q2 Expert risks 1

Applying relevanc


Identif y

Q2 Expert Schengen area 0 Q2 Expert security 1 Q2 Expert social exclusion 1 Q2 Expert surveillance 1 Q2 Expert terrorism 1 Q2 Expert threat 1 Q2 Expert transparency 1 Q2 Expert violence 0 Q2 Expert xenophobia 0 Q2 Expert flexicurity 0

Q2 Text Mining COPING_WITH_THE_CRISIS 0

Q2 Text Mining SOCIAL_EXCLUSION 1

Q2 Text Mining DATA_SECURITY 0

Q2 Text Mining INTRODUCING_HEAVIER_FINES_FOR_OFFENDERS 0

Question Source Keywords Decision Q2 Text Mining LOAN_SHARKS 0Q2 Text Mining DAILY_LIFE_TO_PROTECT 1Q2 Text Mining DEPENDENCE_ON_TECHNOLOGY 1Q2 Text Mining FINANCIAL_SUPERVISION 0Q2 Text Mining PRIVACY_AND_DATA 1Q2 Social Media Terrorism 1Q2 Social Media Justice 1

Applying relevanc


Identif y

Q2 Social Media Prisons 0Q2 Social Media Violence 0Q2 Social Media Security 1Q3 Expert surveillance 1Q3 Expert extremism 0Q3 Expert Military equipment 1Q3 Social Media Crime and terrorism 1Q3 Social Media Asylum and protection of removal 0Q3 Scientific publications Ultraprivate Smartphones 1

IV Section: IV Spreading Excellence and Widening Participation


Decision WP-14-15 WP-16-17 Q1-covered 6 4 4

Applying relevanc


Identif y

Total Q1 6 6 6 Q2-covered 12 9 9 Total Q2 12 12 12 Q3-covered 0 0 0 Total Q3 0 0 0

Decision WP-14-15 WP-16-17 Q1-covered 100% 67% 67% Q2-covered 100% 75% 75% Q3-covered #DIV/0! #DIV/0! #DIV/0!

Question Source Keywords Decision Q1 Expert digital technologies and online services as a horizontal policy Q1 Expert improve research and innovation Q1 Expert strengthen innovation chain Q1 Expert enhance performance of education systems Q1 Expert complete tertiary education Q1 Expert boost for new jobs

Applying relevanc


Identif y

Q2 Expert excellence Q2 Expert mobility Q2 Expert education Q2 Expert talent Q2 Expert job Q2 Expert twinning Q2 Expert teaming Q2 Expert collaboration Q2 Expert skills Q2 Expert capacity Q2 Social Media Citizens involvement Q2 Social Media participation

Q3 Not applicable

V

Section: V Science with and for Society Ramon Code 0 Key-term is not covered Code 1 Key-term is covered

Applying relevanc


Identif y


Decision WP-14-15 WP-16-17 Q1-covered 100% 83% 100% Q2-covered 95% 90% 86% Q3-covered #DIV/0! #DIV/0! #DIV/0!

Question Source Keywords DecisioQ1 Expert boost for new jobs Q1 Expert enhance performance of education systems Q1 Expert sustainable development and lifestyles Q1 Expert reinforce higher education

Applying relevanc


Identif y

Q1 Expert strengthen innovation chain Q1 Expert improve research and innovation Q2 Expert awareness Q2 Expert citizen Q2 Expert education Q2 Expert employment Q2 Expert engagement Q2 Expert ethics Q2 Expert gender Q2 Expert integrity Q2 Expert investment

Question Source Keywords DecisioQ2 Expert knowledge Q2 Expert layman Q2 Expert preparedness

Applying relevanc


Identif y

Q2 Expert public Q2 Expert schools Q2 Expert social Q2 Expert society Q2 Expert transdisciplinary Q2 Social Media Citizens involvement Q2 Social Media participation Q2 Social Media science Q2 Social Media social sciences Q3 Not applicable

SECTION OF H2020 VI.1 Cross-cutting issues. 1.International cooperation Thematic section VI.1 International cooperation Leader: FLECHA Ramon

Applying relevanc


Identif y

Buddies: PEJIC-BACH Mirjana, KONDRATENKO Irena, VALTA Katerina

Question Source Keywords

Q1 Expert improve research and innovation

Q1 Expert strengthen innovation chain

Q1 Expert increase competitiveness

Q1 Expert simplify regulations

Q1 Expert free trade Agreement with the USA

Q1 Expert strengthen capacity for early warning, risk reduction and management of national and global health risks in developing countries

Q1 Expert encourage trade liberalization efforts Q1 Expert keep markets open Q1 Expert fight protectionism

Q1 Expert encourage climate-resilient sustainable development by promoting international cooperation

Q1 Expert ease access to markets Q1 Expert trust in transatlantic relations Q1 Expert better cooperation with third countries

Q1 Expert assist developing countries in attaining long-term debt sustainability through coordinated policies

Q1 Expert duty-free and quota-free market access on a lasting basis for all least developed countries

Q1 Expert development, transfer, dissemination and diffusion of environmentally sound technologies to developing countries

Question Source Keywords

Applying relevanc


Identif y

Q1 Expert

enhance international support for implementing effective and targeted capacity-building in developing countries to support national plans to implement all the Sustainable Development Goals

Q1 Expert

recruitment, development, training and retention of the health workforce in developing countries, least developed countries and small island developing States

Q1 Expert transboundary cooperation in water resources management Q1 Expert international support to developing countries

Q1 Expert international cooperation on and access to science, technology and innovation

Q1 Expert international cooperation and capacity-building support to developing countries in water- and sanitation-related activities and programmes

Q1 Text Mining INTERNATIONAL_COMMUNITY Q1 Text Mining INTERNATIONAL_LAW Q1 Text Mining INTERNATIONAL_COOPERATION Q1 Text Mining AFRICAN_COUNTRIES Q1 Text Mining CANDIDATE_COUNTRIES Q2 Expert open markets Q2 Expert international trade Q2 Expert production-consumption systems Q2 Expert public-private partnerships (PPPs) Q2 Expert international relations Q2 Expert NGOs Q2 Expert EU enlargement Q3 NOT APPLICABLE

Applying relevanc


Identif y

VI-2 SECTION OF H2020 VI.2 Social sciences and humanities (SSH) Thematic section VI. 2 Social sciences and humanities Leader: FECHA Ramon Buddies: PEJIC-BACH Mirjana, KONDRATENKO Irena, VALTA Katerina

Question Source Keywords Q1 Expert improve research and innovation Q1 Expert strengthen innovation chain Q1 Expert poverty Q1 Expert personal data protection Q1 Expert justice Q1 Expert fundamental rights Q1 Expert mutual trust Q1 Expert fight abuse Q1 Expert reduce exploitation Q1 Expert reduce trafficking Q1 Expert fight all forms of violence Q1 Expert fight torture of children Q1 Expert extreme poverty Q1 Expert hunger

Q1 Expert

access to safe, nutritious and sufficient food by all people including infants

Q1 Expert nationality discrimination

Applying relevanc


Identif y

Q1 Expert sex discrimination Q1 Expert racial discrimination Q1 Expert ethnic discrimination Q1 Expert religion discrimination

Question Source Keywords Q1 Expert belief discrimination Q1 Expert discrimination for disability Q1 Expert age discrimination Q1 Expert sexual orientation discrimination Q1 Expert discrimination for people belonging to a minority Q1 Expert adopt social protection policies Q1 Expert access to essential health-care services Q1 Text Mining CREATE_JOBS Q1 Text Mining STRUCTURAL_REFORMS Q1 Text Mining INCLUSIVE_ECONOMIC_GROWTH Q1 Text Mining JOB_CREATION Q1 Text Mining LABOUR_MARKET Q1 Text Mining LABOUR_MOBILITY Q1 Text Mining MARKET_ACCESS Q1 Text Mining SOCIAL_DIALOGUE Q1 Text Mining SOCIAL_MARKET_ECONOMY Q2 Expert employment Q2 Expert unemployment Q2 Expert inclusion Q2 Expert wellbeing Q2 Expert poverty

Applying relevanc


Identif y

Q2 Expert sharing Q2 Expert cultural integration Q2 Expert education Q2 Expert social inclusion Q3 NOT APPLICABLE

VI-3 SECTION OF H2020 a Thematic section VI.1. Gender Leader: FLECHA Ramon Buddies: PEJIC-BACH Mirjana

Question Source Keywords DecisioQ1 Expert promote women empowerment Q1 Expert gender equality Q1 Expert promote and protect women's rights

Q1 Expert recognize the role of women as key actors to ensure peace and security

Q1 Expert capacity-building such as education and training of women in Science, Technology, Engineering and Mathematics (STEM) fields

Q1 Expert sex discrimination Q1 Expert give women equal rights to economic resources Q1 Expert give women equal rights to ownership Q1 Expert give women equal rights control over land and property

Q1 Expert give women equal rights to financial services, inheritance and natural resources

Q1 Expert eliminate violence against women and girls

Applying relevanc


Identif y

Q1 Expert women's equal opportunities for leadership

Q1 Expert support policy for the promotion of gender equality and the empowerment of all women and girls

Q1 Expert protect women migrants Q1 Text Mining WOMEN_AND_GIRLS Q1 Text Mining GENDER_EQUALITY Q1 Text Mining ROLE_OF_WOMEN Q1 Text Mining ADOLESCENT_GIRLS

Question Source Keywords DecisioQ1 Text Mining EMPOWERMENT_OF_WOMEN Q1 Text Mining MATERNAL_MORTALITY Q1 Text Mining SEXUAL_AND_GENDER_BASED_VIOLENCE Q1 Text Mining WOMEN_IN_STEM_FIELDS Q2 Expert Gender Q2 Expert societal Q2 Expert vulnerability Q2 Expert education Q2 Expert awareness Q2 Social Media Women Q2 Social Media Same-sex marriage

Q3 NOT APPLICABLE

Applying relevance-assessing methodologies to …...Applying relevance-assessing methodologies to...

Documents

Transcript of Applying relevance-assessing methodologies to …...Applying relevance-assessing methodologies to...